Comments on: How to Install Sqoop on Amazon Elastic Map Reduce (EMR)

By: SN

SN — Thu, 31 Dec 2015 19:03:46 +0000

I tried these steps to install SQOOP but I get the following error message. Can someone please tell me why?

Exception in thread “main” java.lang.RuntimeException: java.io.IOException: Cannot run program “/mnt/var/lib/hadoop/steps/s-2EZU086RJ5ABY/./install-sqoop-SN.sh” (in directory “.”): error=2, No such file or directory
at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:143)
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: Cannot run program “/mnt/var/lib/hadoop/steps/s-2EZU086RJ5ABY/./install-sqoop-SN.sh” (in directory “.”): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:96)
… 6 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
… 7 more

By: Sqoop via an ssh tunnel on Amazon EMR « Getting there is 99.5% the fun

Sqoop via an ssh tunnel on Amazon EMR « Getting there is 99.5% the fun — Fri, 29 Mar 2013 02:07:00 +0000

[…] turns out other people have solved this problem, but here’s our solution for posterity. Note that it’s […]

By: Justin

Justin — Thu, 22 Nov 2012 02:30:27 +0000

This worked for me. One obstacles I had was that I wrote my .sh file on Windows so the “End Of Line” format was incorrect.

You can use Notepad++ to fix this, go to Edit > EOL Conversion > UNIX Format

Thanks Kyle.

By: Rupinder

Rupinder — Wed, 21 Nov 2012 14:11:16 +0000

I just checked and successfully installed sqoop as a bootstrap action in EMR.

By: Rupinder

Rupinder — Wed, 21 Nov 2012 13:26:45 +0000

Thanks. This was helpful.
Is it possible to configure the script to install Sqoop as a bootstrap action in EMR Console ?

By: ranjith

ranjith — Fri, 10 Aug 2012 21:37:20 +0000

The last sqoop import and export statement
./sqoop-1.4.1-incubating__hadoop-0.20/bin/sqoop export –connect jdbc:mysql:/// –table –export-dir –fields-terminated-by , –input-null-non-string ‘\\N’ –username –password

Is this a part of the emr job flow ? Can we add this as a step to the job flow that which runs the export of data from hive to mysql on the emr instance? Or do I need to run this from terminal?

By: Siyin

Siyin — Tue, 17 Jul 2012 16:01:08 +0000

Hi,

I followed your steps exactly and got the following error running sqoop:

Exception in thread “main” java.lang.NoClassDefFoundError: com/cloudera/sqoop/Sqoop
Caused by: java.lang.ClassNotFoundException: com.cloudera.sqoop.Sqoop
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.cloudera.sqoop.Sqoop. Program will exit.

Any ideas on why it’s looking for Cloudera’s Sqoop library and how do I get past this? Thanks.

By: ryan

ryan — Wed, 27 Jun 2012 22:15:53 +0000

Got any examples of how to take data from Hive table and make a CSV file on HDFS?

Also, have you tried the ‘–num-mappers’ param to sqoop (http://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html#_syntax_3)? Wondering if/how much that speeds things up. I’m going to be trying an export from hive into a mysql table and doing an update on a multi-col key (via –update-key field1,field2,field3) where my original source is a dynamodb table. Using EMR to aggregate metrics and push the aggregation numbers into a mysql table.

By: meble

meble — Wed, 20 Jun 2012 15:42:36 +0000

helpfull information’s about how to install sqoop thx!:)

By: Brendan Benson

Brendan Benson — Sat, 09 Jun 2012 23:40:26 +0000

This is a huge help. It sucks that this has to be such a hacky process through AWS. Thanks for the example.

One typo: in the script runner command, it should be –args, not –arg.