java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "
2016-09-02 17:22
2915 查看
运行clouder yarn-client模式的spark抛异常:
原因是提交命令没有添加--master参数。
如下命令运行正常:
</pre><p></p><p></p><pre name="code" class="java">16/09/02 17:16:32 WARN net.ScriptBasedMapping: Exception running /etc/hadoop/conf.cloudera.yarn/topology.py 10.55.45.251 java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "/root"): error=13, 权限不够 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at org.apache.hadoop.util.Shell.runCommand(Shell.java:508) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738) at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251) at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188) at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119) at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101) at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81) at org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:38) at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:208) at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:187) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:187) at org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:166) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.scheduler.TaskSetManager.<init>(TaskSetManager.scala:165) at org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:200) at org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:164) at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1052) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:921) at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:861) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1607) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.io.IOException: error=13, 权限不够 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:186) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) ... 26 more
原因是提交命令没有添加--master参数。
如下命令运行正常:
sudo -u spark spark-submit \ --class com.raysdata.etl.GPSLogClean \ --master yarn-cluster \ --executor-memory 1G \ --total-executor-cores 10 \ /tmp/raysdata-1.0-SNAPSHOT.jar \ /user/optadmin/GPS/Position/Correct/2016/log \ /user/optadmin/spark/gps/output
相关文章推荐
- Exception in thread "main" java.io.IOException: Cannot run program "XX": CreateProcess error
- Exception in thread "main" java.io.IOException: Cannot run program "chmod": CreateProcess error=2
- Exception in thread &quot;main&quot; java.io.IOException: Cannot run program &quot;XX&quot;: CreateP
- Exception in thread "main" java.io.IOException: Cannot run program "XX": CreateProcess error
- 转换webp格式图片,java.io.IOException: Cannot run program "dwebp": error=2, No such file or directory
- 出错1Exception in thread "main" java.io.IOException: Cannot run program "C:\winmine.e xe": CreateProce
- hadoop java.io.IOException: Cannot run program "nice"
- linux下java.io.IOException: Cannot run program "/opt/jdk/jre/bin/java": error=13, Permission denied
- Execute failed: java.io.IOException: Cannot run program "sdk-linux/build-tools/22.0.0/aapt": error=2
- java.io.IOException: Cannot run program "ls": CreateProcess error=2,错误解决方法
- tensorflow serving 安装报错:java.io.IOException: Cannot run program "patch"
- java.io.IOException: Cannot run program "chmod": CreateProcess error=2
- Gradle时Execute failed: java.io.IOException: Cannot run program "sdk-linux/build-tools/22.0.0/aapt"
- java.io.IOException: Cannot run program "bash": error=12, Cannot allocate memory
- java.io.IOException: Cannot run program "/opt/java/sdk/build-tools/23.0.1/aapt": error=2, No such fi
- eclipse环境问题之——Cannot run program "C:\Java\jdk1.6.0_02\bin\javaw.exe" (in directory "D:\project\SDK"): CreateProcess error=87
- java.io.IOException - Cannot run program "git": java.io.IOException: error=2, No such file or direct
- java.io.IOException: Cannot run program "bash"
- Execute failed: java.io.IOException: Cannot run program "sdk-linux/build-tools/22.0.0/aapt": error=2
- java.io.IOException: Cannot run program "C:\Program": CreateProcess error=2,