Comments on: How to Install Sqoop on Amazon Elastic Map Reduce (EMR) http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/ Thu, 02 Mar 2017 22:34:44 +0000 hourly 1 http://wordpress.org/?v=4.2.2 By: SN http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-225304 Thu, 31 Dec 2015 19:03:46 +0000 http://blog.kylemulka.com/?p=545#comment-225304 I tried these steps to install SQOOP but I get the following error message. Can someone please tell me why?

Exception in thread “main” java.lang.RuntimeException: java.io.IOException: Cannot run program “/mnt/var/lib/hadoop/steps/s-2EZU086RJ5ABY/./install-sqoop-SN.sh” (in directory “.”): error=2, No such file or directory
at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:143)
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: Cannot run program “/mnt/var/lib/hadoop/steps/s-2EZU086RJ5ABY/./install-sqoop-SN.sh” (in directory “.”): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:96)
… 6 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
… 7 more

]]>
By: Sqoop via an ssh tunnel on Amazon EMR « Getting there is 99.5% the fun http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93390 Fri, 29 Mar 2013 02:07:00 +0000 http://blog.kylemulka.com/?p=545#comment-93390 […] turns out other people have solved this problem, but here’s our solution for posterity. Note that it’s […]

]]>
By: Justin http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93336 Thu, 22 Nov 2012 02:30:27 +0000 http://blog.kylemulka.com/?p=545#comment-93336 This worked for me. One obstacles I had was that I wrote my .sh file on Windows so the “End Of Line” format was incorrect.

You can use Notepad++ to fix this, go to Edit > EOL Conversion > UNIX Format

Thanks Kyle.

]]>
By: Rupinder http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93335 Wed, 21 Nov 2012 14:11:16 +0000 http://blog.kylemulka.com/?p=545#comment-93335 I just checked and successfully installed sqoop as a bootstrap action in EMR.

]]>
By: Rupinder http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93334 Wed, 21 Nov 2012 13:26:45 +0000 http://blog.kylemulka.com/?p=545#comment-93334 Thanks. This was helpful.
Is it possible to configure the script to install Sqoop as a bootstrap action in EMR Console ?

]]>
By: ranjith http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93325 Fri, 10 Aug 2012 21:37:20 +0000 http://blog.kylemulka.com/?p=545#comment-93325 The last sqoop import and export statement
./sqoop-1.4.1-incubating__hadoop-0.20/bin/sqoop export –connect jdbc:mysql:/// –table –export-dir –fields-terminated-by , –input-null-non-string ‘\\N’ –username –password

Is this a part of the emr job flow ? Can we add this as a step to the job flow that which runs the export of data from hive to mysql on the emr instance? Or do I need to run this from terminal?

]]>
By: Siyin http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93322 Tue, 17 Jul 2012 16:01:08 +0000 http://blog.kylemulka.com/?p=545#comment-93322 Hi,

I followed your steps exactly and got the following error running sqoop:

Exception in thread “main” java.lang.NoClassDefFoundError: com/cloudera/sqoop/Sqoop
Caused by: java.lang.ClassNotFoundException: com.cloudera.sqoop.Sqoop
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.cloudera.sqoop.Sqoop. Program will exit.

Any ideas on why it’s looking for Cloudera’s Sqoop library and how do I get past this? Thanks.

]]>
By: ryan http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93311 Wed, 27 Jun 2012 22:15:53 +0000 http://blog.kylemulka.com/?p=545#comment-93311 Got any examples of how to take data from Hive table and make a CSV file on HDFS?

Also, have you tried the ‘–num-mappers’ param to sqoop (http://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html#_syntax_3)? Wondering if/how much that speeds things up. I’m going to be trying an export from hive into a mysql table and doing an update on a multi-col key (via –update-key field1,field2,field3) where my original source is a dynamodb table. Using EMR to aggregate metrics and push the aggregation numbers into a mysql table.

]]>
By: meble http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93309 Wed, 20 Jun 2012 15:42:36 +0000 http://blog.kylemulka.com/?p=545#comment-93309 helpfull information’s about how to install sqoop thx!:)

]]>
By: Brendan Benson http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/comment-page-1/#comment-93306 Sat, 09 Jun 2012 23:40:26 +0000 http://blog.kylemulka.com/?p=545#comment-93306 This is a huge help. It sucks that this has to be such a hacky process through AWS. Thanks for the example.

One typo: in the script runner command, it should be –args, not –arg.

]]>