yarn - Why DistributedCache is caching all files directly into the root of the tmp storage dir in Hadoop 2 -



yarn - Why DistributedCache is caching all files directly into the root of the tmp storage dir in Hadoop 2 -

i'm migrating hadoop 1.0.4 codes hadoop 2.3 platform, , met weird behavior alter of distributedcache:

in hadoop 1, if want cache file @ /user/foo/file/bar/name.avro, distributedcache re-create file local cache folder, , create same sub directories accordingly. file stored @ /[root_of_tmp_cache_dir]/user/foo/file/bar/name.avro.

now same codes in hadoop 2 set file straight in root folder without creating sub directories. cached file stored at: /[root_of_tmp_cache_dir]/name.avro.

this cause name conflicts if caching multiple files when file names part-r-00000.avro.

of course, applying link , renaming cached file unique name can 1 way solve problem; more generally, creating unique names seem trivial in many cases, when needs guarantee unique names across different mappers/reducers. i'm wondering if there other ways alter behavior, such creating folder within tmp dir or maybe tune mapreduce configuration parameter?

one way tried seek create uri "path#path", linking ourselves, seem next exception:

14-10-2014 16:05:41 pdt admm_train info - caused by: java.lang.illegalargumentexception: resource name must relative 14-10-2014 16:05:41 pdt admm_train info - @ org.apache.hadoop.mapreduce.v2.util.mrapps.parsedistributedcacheartifacts(mrapps.java:489) 14-10-2014 16:05:41 pdt admm_train info - @ org.apache.hadoop.mapreduce.v2.util.mrapps.setupdistributedcache(mrapps.java:430) 14-10-2014 16:05:41 pdt admm_train info - @ org.apache.hadoop.mapred.yarnrunner.createapplicationsubmissioncontext(yarnrunner.java:455) 14-10-2014 16:05:41 pdt admm_train info - @ org.apache.hadoop.mapred.yarnrunner.submitjob(yarnrunner.java:283) 14-10-2014 16:05:41 pdt admm_train info - @ org.apache.hadoop.mapreduce.jobsubmitter.submitjobinternal(jobsubmitter.java:432)

this bug due alter of internal behavior of distributedcache in hadoop 1 , 2.

in hadoop 1, to-be-cached file stored @ local tmp directory retaining previous path structure. example, if caching hdfs:///foo/bar/file1 in hadoop1, stored @ /[some tmp path]/foo/bar/file1.

in hadoop 2, distributecache peel out path structure, store hdfs:///foo/bar/file1, straight store @ /[some tmp path]/file1.

also if using symblink names, hadoop 2 rename file link name while hadoop 1 not. results in compatibility conflicts while switching hadoop 1 hadoop 2.

an easy solution utilize symlink name, , access file via name stored differently, can still access them in same fashion.

yarn hadoop2 distributed-cache

Comments

Post a Comment

Popular posts from this blog

Delphi change the assembly code of a running process -

Delphi change the assembly code of a running process - i'm trying alter address 00741fa5 has push test.009e721c . alter push test.009e71c8 . procedure callback; asm force $9e71c8 end; procedure tform2.btn1click(sender: tobject); var ppid :dword; pprocess : cardinal; begin getwindowthreadprocessid(findwindow(nil,pchar('test')), @ppid); pprocess := openprocess(process_all_access, false, ppid); injectasm(pprocess, $741fa5, 10, integer(@callback), 0); end; here injectasm function (with error checking) function injectasm(process: longword; injectaddress, injectsize, codeaddress, codesize : integer): pointer; var csize, rsize : integer; replaced : array of byte; jmp : array [0..4] of byte; jmpaddress : integer; nopv : byte; : integer; nbr: ulong_ptr; begin //injectsize must equal or greater 5, because need space our //(far) jmp wwxxyyzz instruction //if there's no space, inject in place of few instructions if injectsize ...
Read more

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) -

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) - hello have problem hibernate , jackson. have 2 pojos @entity @table(name = "users") public class user implements serializable { static final long serialversionuid = 4235637833262722256l; @id @generatedvalue private int id; @column(name = "creation_date") private date creationdate; @column(name = "first_name") private string firstname; @column(name = "last_name") private string lastname; @version @column(name = "version") private int version; @column(name = "e_mail") private string email; @column(name = "enabled") private boolean enabled; @column(name = "login") private string login; @column(name = "password") private string password; @jsonmanagedreference("role") @onetomany(mappedby = "user", cascade = cascadetype.all,fetch = fetchtyp...
Read more

C++ 11 "class" keyword -

C++ 11 "class" keyword - i've started using c++ 11 little bit more , have few questions on special uses of class keyword. know it's used declare class, there's 2 instances have seen don't understand: method<class t>(); and class class_name *var; why precede typename keyword class in first example, , keyword pointer in sec example? that known elaborated type specifier , necessary when class name "shadowed" or "hidden" , need explicit. class t { }; // love of god don't t t; t t2; if compiler smart, give these warnings: main.cpp:15:5: error: must utilize 'class' tag refer type 't' in scope t t2; ^ class main.cpp:14:7: note: class 't' hidden non-type declaration of 't' here t t; ^ if class not defined, deed forwards declaration. for example: template <typename t> void method() { using type = typename t::value_type; } method...
Read more