python - AWS EMR Spark "No Module named pyspark" -



python - AWS EMR Spark "No Module named pyspark" -

i created spark cluster, ssh master, , launch shell:

master=yarn-client ./spark/bin/pyspark

when following:

x = sc.textfile("s3://location/files.*") xt = x.map(lambda x: handlejson(x)) table= sqlctx.inferschema(xt)

i next error:

error python worker: /usr/bin/python: no module named pyspark pythonpath was: /mnt1/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/11/spark-assembly-1.1.0-hadoop2.4.0.jar java.io.eofexception java.io.datainputstream.readint(datainputstream.java:392) org.apache.spark.api.python.pythonworkerfactory.startdaemon(pythonworkerfactory.scala:151) org.apache.spark.api.python.pythonworkerfactory.createthroughdaemon(pythonworkerfactory.scala:78) org.apache.spark.api.python.pythonworkerfactory.create(pythonworkerfactory.scala:54) org.apache.spark.sparkenv.createpythonworker(sparkenv.scala:97) org.apache.spark.api.python.pythonrdd.compute(pythonrdd.scala:66) org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:262) org.apache.spark.rdd.rdd.iterator(rdd.scala:229) org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:62) org.apache.spark.scheduler.task.run(task.scala:54) org.apache.spark.executor.executor$taskrunner.run(executor.scala:177) java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) java.lang.thread.run(thread.java:745)

i checked pythonpath

>>> os.environ['pythonpath'] '/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/home/hadoop/spark/python/:/‌​home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar'

and looked within jar pyspark, , it's there:

jar -tf /home/hadoop/spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar | grep pyspark pyspark/ pyspark/shuffle.py pyspark/resultiterable.py pyspark/files.py pyspark/accumulators.py pyspark/sql.py pyspark/java_gateway.py pyspark/join.py pyspark/serializers.py pyspark/shell.py pyspark/rddsampler.py pyspark/rdd.py ....

has run before? thanks!

you'll want reference these spark issues:

https://issues.apache.org/jira/browse/spark-3008 https://issues.apache.org/jira/browse/spark-1520

the solution (assuming rather not rebuild jar):

unzip -d foo spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar cd foo # if don't have openjdk 1.6: # yum install -y java-1.6.0-openjdk-devel.x86_64 /usr/lib/jvm/openjdk-1.6.0/bin/jar cvmf meta-inf/manifest ../spark/lib/spark-assembly-1.1.0-hadoop2.4.0.jar . # don't neglect dot @ end of command

python amazon-web-services apache-spark amazon-emr

Comments

Post a Comment

Popular posts from this blog

Delphi change the assembly code of a running process -

Delphi change the assembly code of a running process - i'm trying alter address 00741fa5 has push test.009e721c . alter push test.009e71c8 . procedure callback; asm force $9e71c8 end; procedure tform2.btn1click(sender: tobject); var ppid :dword; pprocess : cardinal; begin getwindowthreadprocessid(findwindow(nil,pchar('test')), @ppid); pprocess := openprocess(process_all_access, false, ppid); injectasm(pprocess, $741fa5, 10, integer(@callback), 0); end; here injectasm function (with error checking) function injectasm(process: longword; injectaddress, injectsize, codeaddress, codesize : integer): pointer; var csize, rsize : integer; replaced : array of byte; jmp : array [0..4] of byte; jmpaddress : integer; nopv : byte; : integer; nbr: ulong_ptr; begin //injectsize must equal or greater 5, because need space our //(far) jmp wwxxyyzz instruction //if there's no space, inject in place of few instructions if injectsize ...
Read more

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) -

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) - hello have problem hibernate , jackson. have 2 pojos @entity @table(name = "users") public class user implements serializable { static final long serialversionuid = 4235637833262722256l; @id @generatedvalue private int id; @column(name = "creation_date") private date creationdate; @column(name = "first_name") private string firstname; @column(name = "last_name") private string lastname; @version @column(name = "version") private int version; @column(name = "e_mail") private string email; @column(name = "enabled") private boolean enabled; @column(name = "login") private string login; @column(name = "password") private string password; @jsonmanagedreference("role") @onetomany(mappedby = "user", cascade = cascadetype.all,fetch = fetchtyp...
Read more

C++ 11 "class" keyword -

C++ 11 "class" keyword - i've started using c++ 11 little bit more , have few questions on special uses of class keyword. know it's used declare class, there's 2 instances have seen don't understand: method<class t>(); and class class_name *var; why precede typename keyword class in first example, , keyword pointer in sec example? that known elaborated type specifier , necessary when class name "shadowed" or "hidden" , need explicit. class t { }; // love of god don't t t; t t2; if compiler smart, give these warnings: main.cpp:15:5: error: must utilize 'class' tag refer type 't' in scope t t2; ^ class main.cpp:14:7: note: class 't' hidden non-type declaration of 't' here t t; ^ if class not defined, deed forwards declaration. for example: template <typename t> void method() { using type = typename t::value_type; } method...
Read more