Win7下提交MapReduceJob到集群环境
2014-12-09 14:06
351 查看
1、下载hadoop-2.5.2.tar.gz,解压到D:\hadoop目录下
2、下载hadoop-common-2.2.0-bin-master.zip,将其中的hadoop.dll和winutils.exe复制到D:\hadoop\hadoop-2.5.2\bin目录下
3、配置环境变量HADOOP_HOME=D:\hadoop\hadoop-2.5.2,在PATH中添加%HADOOP_HOME%\bin
4、复制org.apache.hadoop.mapred.YARNRunner.java,修改如下三个地方
1) 替换Line390:391
//vargs.add(MRApps.crossPlatformifyMREnv(jobConf, Environment.JAVA_HOME) + "/bin/java");
vargs.add("$JAVA_HOME/bin/java");
2) 在Line456插入
replaceEnvironment(environment);
3) 在Line638插入
private void replaceEnvironment(Map<String, String> environment) {
String tmpClassPath = (String)environment.get("CLASSPATH");
tmpClassPath = tmpClassPath.replaceAll(";", ":");
tmpClassPath = tmpClassPath.replaceAll("%PWD%", "\\$PWD");
tmpClassPath = tmpClassPath.replaceAll("%HADOOP", "\\$HADOOP");
tmpClassPath = tmpClassPath.replaceAll("\\\\", "/").replaceAll("%", "");
environment.put("CLASSPATH", tmpClassPath);
}
5、用新的YARNRunner.class替换hadoop-mapreduce-client-jobclient-2.5.2.jar中的同名文件
6、在Eclipse中创建一个JavaProject,新增一个用户库hadoop_2.5.2,将hadoop-2.5.2中的jar添加至这个用户库,在Project中引用这个用户库
7、编写wordcount程序,并提交到Hadoop集群,即可得到正确结果,不会报如下错误
Job job_1418033228434_0012 failed with state FAILED due to: Application application_1418033228434_0012 failed 2
times due to AM Container for appattempt_1418033228434_0012_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:
/bin/bash: line 0: fg: no job control
[align=left]
[/align]
ExitCodeException exitCode=1:
/bin/bash: line 0: fg: no job control
[align=left]
[/align]
[align=left] at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )[/align]
[align=left] at org.apache.hadoop.util.Shell.run( Shell.java:455 )[/align]
[align=left] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)[/align]
at java.util.concurrent.FutureTask.run( FutureTask.java:262 )
[align=left] at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)[/align]
[align=left] at java.lang.Thread.run( Thread.java:744 )[/align]
[align=left]
[/align]
[align=left]
[/align]
[align=left]Container exited with a non-zero exit code 1[/align]
[align=left].Failing this attempt.. Failing the application.[/align]
2014-12-09 13:42:05,383 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) )
- Counters: 0
8、如果报下面的错误,请注意可能是你修改了hadoop-mapreduce-client-core-2.5.2.jar中的mapred-default.xml,比如在其中添加了mapreduce.application.classpath,请将修改还原。
Job job_1418033228434_0013 failed with state FAILED due to: Application application_1418033228434_0013
failed 2 times due to AM Container for appattempt_1418033228434_0013_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:
[align=left]ExitCodeException exitCode=1:[/align]
[align=left] at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )[/align]
[align=left] at org.apache.hadoop.util.Shell.run( Shell.java:455 )[/align]
[align=left] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)[/align]
[align=left] at java.util.concurrent.FutureTask.run( FutureTask.java:262 )[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)[/align]
[align=left] at java.lang.Thread.run( Thread.java:744 )[/align]
[align=left]
[/align]
[align=left]
[/align]
[align=left]Container exited with a non-zero exit code 1[/align]
[align=left].Failing this attempt.. Failing the application.[/align]
2014-12-09 13:45:02,949 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) )
- Counters: 0
2、下载hadoop-common-2.2.0-bin-master.zip,将其中的hadoop.dll和winutils.exe复制到D:\hadoop\hadoop-2.5.2\bin目录下
3、配置环境变量HADOOP_HOME=D:\hadoop\hadoop-2.5.2,在PATH中添加%HADOOP_HOME%\bin
4、复制org.apache.hadoop.mapred.YARNRunner.java,修改如下三个地方
1) 替换Line390:391
//vargs.add(MRApps.crossPlatformifyMREnv(jobConf, Environment.JAVA_HOME) + "/bin/java");
vargs.add("$JAVA_HOME/bin/java");
2) 在Line456插入
replaceEnvironment(environment);
3) 在Line638插入
private void replaceEnvironment(Map<String, String> environment) {
String tmpClassPath = (String)environment.get("CLASSPATH");
tmpClassPath = tmpClassPath.replaceAll(";", ":");
tmpClassPath = tmpClassPath.replaceAll("%PWD%", "\\$PWD");
tmpClassPath = tmpClassPath.replaceAll("%HADOOP", "\\$HADOOP");
tmpClassPath = tmpClassPath.replaceAll("\\\\", "/").replaceAll("%", "");
environment.put("CLASSPATH", tmpClassPath);
}
5、用新的YARNRunner.class替换hadoop-mapreduce-client-jobclient-2.5.2.jar中的同名文件
6、在Eclipse中创建一个JavaProject,新增一个用户库hadoop_2.5.2,将hadoop-2.5.2中的jar添加至这个用户库,在Project中引用这个用户库
7、编写wordcount程序,并提交到Hadoop集群,即可得到正确结果,不会报如下错误
Job job_1418033228434_0012 failed with state FAILED due to: Application application_1418033228434_0012 failed 2
times due to AM Container for appattempt_1418033228434_0012_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:
/bin/bash: line 0: fg: no job control
[align=left]
[/align]
ExitCodeException exitCode=1:
/bin/bash: line 0: fg: no job control
[align=left]
[/align]
[align=left] at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )[/align]
[align=left] at org.apache.hadoop.util.Shell.run( Shell.java:455 )[/align]
[align=left] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)[/align]
at java.util.concurrent.FutureTask.run( FutureTask.java:262 )
[align=left] at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)[/align]
[align=left] at java.lang.Thread.run( Thread.java:744 )[/align]
[align=left]
[/align]
[align=left]
[/align]
[align=left]Container exited with a non-zero exit code 1[/align]
[align=left].Failing this attempt.. Failing the application.[/align]
2014-12-09 13:42:05,383 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) )
- Counters: 0
8、如果报下面的错误,请注意可能是你修改了hadoop-mapreduce-client-core-2.5.2.jar中的mapred-default.xml,比如在其中添加了mapreduce.application.classpath,请将修改还原。
Job job_1418033228434_0013 failed with state FAILED due to: Application application_1418033228434_0013
failed 2 times due to AM Container for appattempt_1418033228434_0013_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1:
[align=left]ExitCodeException exitCode=1:[/align]
[align=left] at org.apache.hadoop.util.Shell.runCommand( Shell.java:538 )[/align]
[align=left] at org.apache.hadoop.util.Shell.run( Shell.java:455 )[/align]
[align=left] at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( Shell.java:702)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:300)[/align]
[align=left] at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:81)[/align]
[align=left] at java.util.concurrent.FutureTask.run( FutureTask.java:262 )[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)[/align]
[align=left] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615)[/align]
[align=left] at java.lang.Thread.run( Thread.java:744 )[/align]
[align=left]
[/align]
[align=left]
[/align]
[align=left]Container exited with a non-zero exit code 1[/align]
[align=left].Failing this attempt.. Failing the application.[/align]
2014-12-09 13:45:02,949 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380) )
- Counters: 0
相关文章推荐
- Win7下无法提交MapReduce Job到集群环境(转)
- Win7下eclipse提交Job到hadoop集群
- windows下idea中搭建hadoop开发环境,向远程hadoop集群提交mapreduce任务
- 在windows环境下,向Hadoop集群提交Job
- [Hadoop] - Win7下提交job到集群上去
- MapReduce程序的3种集群提交运行模式详解---基于Windows与Linux两种开发环境
- Windows下的eclipse(with Hadoop Plug-in)向Linux集群提交MapReduce任务
- 攻城狮在路上(陆)-- 提交运行MapReduce程序到hadoop集群运行
- 基于HBase Hadoop 分布式集群环境下的MapReduce程序开发
- HADOOP 分布式集群环境下第一个mapReduce程序—WordCount
- 笔记:深入解析MapReduce架构设计与实现原理 第5章 Job提交与初始化
- Hadoop集群链接_Eclipse开发环境:成功运行mapreduce所遇问题
- MapReduce V1:Job提交流程之JobClient端分析
- MapReduce执行过程源码分析(一)——Job任务的提交
- MapReduce(六): 向JobTracker提交任务
- Could not locate executable null 解决办法【在windows环境下开发hadoop2.4.1或者hadoop2.5.2的mapreduce,执行job遇到的一些错误】
- 基于HBase Hadoop 分布式集群环境下的MapReduce程序开发
- Hadoop源码分析23:MapReduce的Job提交过程
- 获取原生mapreduce程序、sqoop、hive提交的jobId
- MapReduce: Job提交过程