mapreduce job fail when submitted from windows machine
2014-11-04 14:24
363 查看
参照http://stackoverflow.com/questions/24075669/mapreduce-job-fail-when-submitted-from-windows-machine
遇到的问题如下:
可以在java里设置
public Configuration config() {
Configuration conf = new YarnConfiguration();
conf.set("fs.defaultFS", HDFS);
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.address", “c0004650.itcs.hp.com:8032”);
conf.set("yarn.resourcemanager.scheduler.address", “c0004650.itcs.hp.com:8030”);
conf.set("mapreduce.app-submission.cross-platform", "true");
// put all of the third party jars like this, but these jars should be
// put on hadoop clusters.
// conf.set("mapreduce.application.classpath",
// "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,/opt/mount/learn/mahout-1.0-lib/*");
return conf;
}
也可以
(mapred-core.xml)
By setting the value to
error was gone
遇到的问题如下:
2014-06-06 08:32:37,684 [main] INFO Job.monitorAndPrintJob - Job job_1399458460502_0053 running in uber mode : false 2014-06-06 08:32:37,704 [main] INFO Job.monitorAndPrintJob - map 0% reduce 0% 2014-06-06 08:32:37,717 [main] INFO Job.monitorAndPrintJob - Job job_1399458460502_0053 failed with state FAILED due to: Application application_1399458460502_0053 failed 2 times due to AM Container for appattempt_1399458460502_0053_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) .Failing this attempt.. Failing the application. java.io.IOException: Job failed!
可以在java里设置
public Configuration config() {
Configuration conf = new YarnConfiguration();
conf.set("fs.defaultFS", HDFS);
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.address", “c0004650.itcs.hp.com:8032”);
conf.set("yarn.resourcemanager.scheduler.address", “c0004650.itcs.hp.com:8030”);
conf.set("mapreduce.app-submission.cross-platform", "true");
// put all of the third party jars like this, but these jars should be
// put on hadoop clusters.
// conf.set("mapreduce.application.classpath",
// "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,/opt/mount/learn/mahout-1.0-lib/*");
return conf;
}
也可以
(mapred-core.xml)
<property> <description>If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa. </description> <name>mapreduce.app-submission.cross-platform</name> <value>false</value> </property>
By setting the value to
truethe
error was gone
相关文章推荐
- SharePoint 2007 Prompt for Credentials When Accessing FQDN Sites From a Windows Vista or Windows 7 Computer
- Fail to push agent from Primary site to client machine
- When you copy large files to or from earlier operating systems, the copy operation may be slower than expected on some Windows Vista-based computers
- Could not locate executable null 解决办法【在windows环境下开发hadoop2.4.1或者hadoop2.5.2的mapreduce,执行job遇到的一些错误】
- install Composer fail from xampp in windows
- When Not to Create a Thread-come from WINDOWS VIA C C++
- Database startup hangs on Windows when machine has more than 64 cores
- copy file from romote linux machine to windows local machine
- "Access denied" when opening files or folders created in Windows 7 from Windows XP
- How to use python remove the '^M' when copy words from Windows to Linux
- Remote Desktop from Windows to Linux
- a job stcik from smth bbs today
- Why Can't I Access My DAQ Functions From a Windows 2000 User Account?
- Hint for migrating NetBeans from Windows to Linux.
- The Problem Came Up when we upgrade from KingDee K3 to SAP R3(FI module)
- What does Windows do when it starts up?
- Some notes when transferring from Web 1.1 to Web 2.0
- 新域名 @live.com 和 @windowslive.com 即将上线 (from cnbeta)
- Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
- Booting Linux from Windows' Boot Manager