sparkPi在yarn模式下运行
2014-07-29 18:19
105 查看
编写运行脚本
[hadoop@localhost bin]$ cat ./run_sparkPi_on_yarn.sh
export YARN_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
export SPARK_JAR=/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
spark-class org.apache.spark.deploy.yarn.Client --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar
--class org.apache.spark.examples.SparkPi --args yarn-standalone
为其增加执行权限
[hadoop@localhost bin]$ chmod u+x run_sparkPi_on_yarn.sh
控制台log如下：
[hadoop@localhost bin]$ ./run_sparkPi_on_yarn.sh
14/07/29 18:03:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/29 18:03:07 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:08 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 1
14/07/29 18:03:08 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
queueApplicationCount = 3, queueChildQueueCount = 0
14/07/29 18:03:08 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 8192
14/07/29 18:03:08 INFO yarn.Client: Preparing Local resources
14/07/29 18:03:09 INFO yarn.Client: Uploading file:/home/hadoop/spark-0.9.1-bin-hadoop2/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-examples_2.10-assembly-0.9.1.jar
14/07/29 18:03:10 INFO yarn.Client: Uploading file:/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
14/07/29 18:03:11 INFO yarn.Client: Setting up the launch environment
14/07/29 18:03:11 INFO yarn.Client: Setting up container launch context
14/07/29 18:03:11 INFO yarn.Client: Command for starting the Spark ApplicationMaster: $JAVA_HOME/bin/java -server -Xmx512m -Djava.io.tmpdir=$PWD/tmp org.apache.spark.deploy.yarn.ApplicationMaster --class org.apache.spark.examples.SparkPi --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --args 'yarn-standalone' --worker-memory 1024 --worker-cores 1 --num-workers 2 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
14/07/29 18:03:11 INFO yarn.Client: Submitting application to ASM
14/07/29 18:03:11 INFO impl.YarnClientImpl: Submitted application application_1406524413406_0004 to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:12 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:13 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:14 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:15 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:16 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:17 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:18 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:19 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:20 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:21 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:22 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:23 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:24 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:25 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:26 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:27 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:28 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:29 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:30 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:31 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: FINISHED
distributedFinalState: SUCCEEDED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/A
appUser: hadoop
这个过程中我们可以通过http://localhost:8088/cluster来监视任务的执行过程
这里已经结束了 通过点击applicationid我们可以进入更详细的画面：
通过下方的logs我们可以进入log页面查看log 程序的输出也会来这里给出
这个log是stderr 就是程序运行的log都会通过stderr输入
与之对应的是stdout是程序的执行结果：
这里给出了pi的大概值
同样的这些log可以在hadoop的logs目录中找到
我机器的路径是在
[hadoop@localhost logs]$ cd userlogs/
[hadoop@localhost userlogs]$ ls
application_1403063579463_0001 application_1403080077168_0006 application_1404437711537_0005
application_1403063579463_0002 application_1403080077168_0007 application_1404437711537_0006
application_1403074938465_0001 application_1403080077168_0009 application_1405671242399_0001
application_1403074938465_0002 application_1403080077168_0010 application_1406016955739_0001
application_1403074938465_0003 application_1403080077168_0011 application_1406016955739_0002
application_1403074938465_0004 application_1403080077168_0012 application_1406016955739_0003
application_1403074938465_0005 application_1403135013165_0001 application_1406524413406_0001
application_1403080077168_0001 application_1403136067016_0001 application_1406524413406_0002
application_1403080077168_0002 application_1403136067016_0002 application_1406524413406_0003
application_1403080077168_0003 application_1403136067016_0003 application_1406524413406_0004
application_1403080077168_0004 application_1403136067016_0004
application_1403080077168_0005 application_1404175010388_0001
[hadoop@localhost userlogs]$ pwd
/home/hadoop/hadoop/logs/userlogs
我们的applicationid为application_1406524413406_0004 所以我们进入这个目录：
[hadoop@localhost application_1406524413406_0004]$ ll
total 12
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000001
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000002
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000003
[hadoop@localhost application_1406524413406_0004]$ pwd
/home/hadoop/hadoop/logs/userlogs/application_1406524413406_0004
然后就可以找到对应stdout和stderr了
[hadoop@localhost application_1406524413406_0004]$ cd container_1406524413406_0004_01_000001/
[hadoop@localhost container_1406524413406_0004_01_000001]$ ls
stderr stdout
[hadoop@localhost container_1406524413406_0004_01_000001]$
[hadoop@localhost bin]$ cat ./run_sparkPi_on_yarn.sh
export YARN_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
export SPARK_JAR=/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
spark-class org.apache.spark.deploy.yarn.Client --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar
--class org.apache.spark.examples.SparkPi --args yarn-standalone
为其增加执行权限
[hadoop@localhost bin]$ chmod u+x run_sparkPi_on_yarn.sh
[hadoop@localhost bin]$ ll total 792 -rw-rw-r--. 1 hadoop hadoop 2601 Mar 27 13:44 compute-classpath.cmd -rwxrwxr-x. 1 hadoop hadoop 3330 Mar 27 13:44 compute-classpath.sh -rw-rw-r--. 1 hadoop hadoop 5219 Jul 4 10:33 JavaSparkM-0.0.1-SNAPSHOT.jar -rw-rw-r--. 1 hadoop hadoop 10283 Jul 1 17:23 JavaSparkT.jar -rwxrwxr-x. 1 hadoop hadoop 2070 Mar 27 13:44 pyspark -rw-rw-r--. 1 hadoop hadoop 1827 Mar 27 13:44 pyspark2.cmd -rw-rw-r--. 1 hadoop hadoop 1000 Mar 27 13:44 pyspark.cmd -rwxrwxr-x. 1 hadoop hadoop 3055 Mar 27 13:44 run-example -rw-rw-r--. 1 hadoop hadoop 2046 Mar 27 13:44 run-example2.cmd -rw-rw-r--. 1 hadoop hadoop 1012 Mar 27 13:44 run-example.cmd -rwxrw-r--. 1 hadoop hadoop 181 May 30 10:35 run_graphx.sh -rwxrw-r--. 1 hadoop hadoop 2155 Jul 1 17:25 run-java-example -rwxrw-r--. 1 hadoop hadoop 410 Jul 4 10:42 run_java_wordcount_local.sh -rwxrw-r--. 1 hadoop hadoop 440 Jul 4 10:39 run_java_wordcount.sh -rwxrw-r--. 1 hadoop hadoop 352 May 22 11:05 run_sparkPi_on_yarn.sh -rwxrw-r--. 1 hadoop hadoop 430 May 22 11:06 run_spark_wordcount.sh -rw-rw-r--. 1 hadoop hadoop 698341 May 23 16:04 spam.data -rwxrwxr-x. 1 hadoop hadoop 5151 Mar 27 13:44 spark-class -rwxrwxr-x. 1 hadoop hadoop 3212 Mar 27 13:44 spark-class2.cmd -rw-rw-r--. 1 hadoop hadoop 1010 Mar 27 13:44 spark-class.cmd -rwxrwxr-x. 1 hadoop hadoop 3184 Mar 27 13:44 spark-shell -rwxrwxr-x. 1 hadoop hadoop 941 Mar 27 13:44 spark-shell.cmd -rw-rw-r--. 1 hadoop hadoop 5231 May 21 15:34 spark-wordcount-scala.jar [hadoop@localhost bin]$
控制台log如下：
[hadoop@localhost bin]$ ./run_sparkPi_on_yarn.sh
14/07/29 18:03:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/29 18:03:07 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:08 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 1
14/07/29 18:03:08 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
queueApplicationCount = 3, queueChildQueueCount = 0
14/07/29 18:03:08 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 8192
14/07/29 18:03:08 INFO yarn.Client: Preparing Local resources
14/07/29 18:03:09 INFO yarn.Client: Uploading file:/home/hadoop/spark-0.9.1-bin-hadoop2/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-examples_2.10-assembly-0.9.1.jar
14/07/29 18:03:10 INFO yarn.Client: Uploading file:/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
14/07/29 18:03:11 INFO yarn.Client: Setting up the launch environment
14/07/29 18:03:11 INFO yarn.Client: Setting up container launch context
14/07/29 18:03:11 INFO yarn.Client: Command for starting the Spark ApplicationMaster: $JAVA_HOME/bin/java -server -Xmx512m -Djava.io.tmpdir=$PWD/tmp org.apache.spark.deploy.yarn.ApplicationMaster --class org.apache.spark.examples.SparkPi --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --args 'yarn-standalone' --worker-memory 1024 --worker-cores 1 --num-workers 2 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
14/07/29 18:03:11 INFO yarn.Client: Submitting application to ASM
14/07/29 18:03:11 INFO impl.YarnClientImpl: Submitted application application_1406524413406_0004 to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:12 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:13 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:14 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:15 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:16 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:17 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:18 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:19 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:20 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:21 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:22 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:23 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:24 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:25 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:26 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:27 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:28 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:29 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:30 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:31 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: FINISHED
distributedFinalState: SUCCEEDED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/A
appUser: hadoop
这个过程中我们可以通过http://localhost:8088/cluster来监视任务的执行过程
这里已经结束了 通过点击applicationid我们可以进入更详细的画面：
通过下方的logs我们可以进入log页面查看log 程序的输出也会来这里给出
这个log是stderr 就是程序运行的log都会通过stderr输入
与之对应的是stdout是程序的执行结果：
这里给出了pi的大概值
Pi is roughly 3.13822
同样的这些log可以在hadoop的logs目录中找到
我机器的路径是在
[hadoop@localhost logs]$ cd userlogs/
[hadoop@localhost userlogs]$ ls
application_1403063579463_0001 application_1403080077168_0006 application_1404437711537_0005
application_1403063579463_0002 application_1403080077168_0007 application_1404437711537_0006
application_1403074938465_0001 application_1403080077168_0009 application_1405671242399_0001
application_1403074938465_0002 application_1403080077168_0010 application_1406016955739_0001
application_1403074938465_0003 application_1403080077168_0011 application_1406016955739_0002
application_1403074938465_0004 application_1403080077168_0012 application_1406016955739_0003
application_1403074938465_0005 application_1403135013165_0001 application_1406524413406_0001
application_1403080077168_0001 application_1403136067016_0001 application_1406524413406_0002
application_1403080077168_0002 application_1403136067016_0002 application_1406524413406_0003
application_1403080077168_0003 application_1403136067016_0003 application_1406524413406_0004
application_1403080077168_0004 application_1403136067016_0004
application_1403080077168_0005 application_1404175010388_0001
[hadoop@localhost userlogs]$ pwd
/home/hadoop/hadoop/logs/userlogs
我们的applicationid为application_1406524413406_0004 所以我们进入这个目录：
[hadoop@localhost application_1406524413406_0004]$ ll
total 12
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000001
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000002
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000003
[hadoop@localhost application_1406524413406_0004]$ pwd
/home/hadoop/hadoop/logs/userlogs/application_1406524413406_0004
然后就可以找到对应stdout和stderr了
[hadoop@localhost application_1406524413406_0004]$ cd container_1406524413406_0004_01_000001/
[hadoop@localhost container_1406524413406_0004_01_000001]$ ls
stderr stdout
[hadoop@localhost container_1406524413406_0004_01_000001]$
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Apache Spark源码走读之10 -- 在YARN上运行SparkPi
- Yarn模式下Apache HAWQ的运行 [作者：白洁]
- Spark on YARN client模式作业运行全过程分析
- Spark的运行模式(2)--Yarn-Cluster和Yarn-Client
- spark on yarn 两种运行模式（client 、cluster）对比
- 在standalone模式下运行yarn 0.9.0对HDFS上的数据进行计算
- spark（四）yarn上的运行模式
- 【Hadoop】YARN 原理、MR本地&YARN运行模式
- spark-sql master on yarn 模式运行 select count(*) 报错日志
- yarn模式运行spark作业所有属性详解
- Spark on YARN两种运行模式介绍
- Spark on YARN客户端模式作业运行全过程分析
- Spark on YARN集群模式作业运行全过程分析
- spark在yarn上面的运行模型：yarn-cluster和yarn-client两种运行模式:
- yarn模式运行spark
- Spark on YARN的两种运行模式
- Spark on YARN两种运行模式
- Spark on YARN客户端模式作业运行全过程分析
- Spark的运行模式---yarn模式
- hadoop初识之三：搭建hadoop环境（配置HDFS，Yarn及mapreduce 运行在yarn）上及三种运行模式（本地模式，伪分布式和分布式介）