用新版本的job打成jar包在终端运行mapreduce程序遇到的问题 -找不到Map class?
2011-10-13 14:16
579 查看
hadoop@ubuntu:~/hadoop-0.20.2/bin$ ./hadoop jar ~/finger.jar Finger kaoqin output
出现的错误:
11/10/14 13:52:07 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/10/14 13:52:07 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/10/14 13:52:07 INFO input.FileInputFormat: Total input paths to process : 5
11/10/14 13:52:07 INFO mapred.JobClient: Running job: job_201110141041_0014
11/10/14 13:52:08 INFO mapred.JobClient: map 0% reduce 0%
11/10/14 13:52:19 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:19 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:25 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:25 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:31 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:31 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:40 INFO mapred.JobClient: Job complete: job_201110141041_0014
11/10/14 13:52:40 INFO mapred.JobClient: Counters: 3
11/10/14 13:52:40 INFO mapred.JobClient: Job Counters
11/10/14 13:52:40 INFO mapred.JobClient: Launched map tasks=8
11/10/14 13:52:40 INFO mapred.JobClient: Data-local map tasks=8
11/10/14 13:52:40 INFO mapred.JobClient: Failed map tasks=1
从出错的信息看,一开始我以为是要把map和reduce类一起放在主类当中,然后在定制job的时候再设置用到的map和reduce类,后来尝试了一下,重新打包运行发现还是不可以。
后来才发现,No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).与hadoop版本有关,新版本中job好像用不起来。
解决办法:用 job.setJarByClass(Finger.class);因为 包含map 的 jar 文件是要分发到datanode上面的, 我一开始 也是遇到这个问题, 后来就想通了。测试也好用了~~
出现的错误:
11/10/14 13:52:07 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/10/14 13:52:07 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/10/14 13:52:07 INFO input.FileInputFormat: Total input paths to process : 5
11/10/14 13:52:07 INFO mapred.JobClient: Running job: job_201110141041_0014
11/10/14 13:52:08 INFO mapred.JobClient: map 0% reduce 0%
11/10/14 13:52:19 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:19 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:25 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:25 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:31 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:31 INFO mapred.JobClient: Task Id : attempt_201110141041_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Finger$FingerMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: Finger$FingerMapper1
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
11/10/14 13:52:40 INFO mapred.JobClient: Job complete: job_201110141041_0014
11/10/14 13:52:40 INFO mapred.JobClient: Counters: 3
11/10/14 13:52:40 INFO mapred.JobClient: Job Counters
11/10/14 13:52:40 INFO mapred.JobClient: Launched map tasks=8
11/10/14 13:52:40 INFO mapred.JobClient: Data-local map tasks=8
11/10/14 13:52:40 INFO mapred.JobClient: Failed map tasks=1
从出错的信息看,一开始我以为是要把map和reduce类一起放在主类当中,然后在定制job的时候再设置用到的map和reduce类,后来尝试了一下,重新打包运行发现还是不可以。
后来才发现,No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).与hadoop版本有关,新版本中job好像用不起来。
解决办法:用 job.setJarByClass(Finger.class);因为 包含map 的 jar 文件是要分发到datanode上面的, 我一开始 也是遇到这个问题, 后来就想通了。测试也好用了~~
相关文章推荐
- 在eclipse上运行MapReduce的wordcount程序所遇到的问题
- maven项目找不到classNotFound问题,jar包运行时没有在lib目录中
- 在 Eclipse 中运行 MapReduce 程序会遇到的问题
- AndroidStudio2.2遇到的运行时提示***.class,***.jar删不掉的问题
- server 2003 安装 VS 2003 遇到的问题 :安装frontpage 2000 web 客户端 和 安装程序运行时文件 失败
- [置顶] 使用java命令运行class文件提示“错误:找不到或无法加载主类“的问题分析
- eclipse 导出可运行jar包 cmd命令执行java **.jar时找不到第三方jar包问题
- 编写MapReduce程序访问HBase 遇到的问题与解决方法
- 在eclipse中JAVA从打包成jar程序到双击运行的整个过程(没有主清单属性问题的解决)
- 开发中遇到的关于classnotfound和resource文件找不到和getresource的路径问题
- 在Ubuntu9.10下安装DirectFB-1.0.1以及运行DFB程序时遇到的问题及解决方法
- 安装SQL200的遇到的问题:以前的某个程序安装已在安装计算机上穿件挂起的文件操作。运行安装程序之前必须重新启动计算机
- windows环境下Eclipse开发MapReduce程序遇到的四个问题及解决办法
- eclipse运行maven install的时候,提示找不到JDK中的rt.jar包下的程序
- 编译 Apache Karaf 2.2.9遇到找不到jar包问题的解决方案
- 关于ios 程序加载百度地图lib,出现链接错误:找不到符号 (null): _OBJC_CLASS_$_BMKMapManager的解决办法
- 单点登录(四)-----遇到问题-----cas server 源码部署tomcat运行报错ClassNotFoundException: org.jasig.cas.CasEnvironmentCo
- jboss服务器端,project中运行的jar包找不到“java.lang.ClassNotFoundException: com.google.gson.Gson”
- 我老遇到代码没检验出有错但一运行就会说有个问题导致程序停止工作
- 解决Hadoop运行jar包时MapReduce任务启动前OutOfMemoryError:Java heap space问题