Spark计算Pi---Spark学习笔记2

话接上回Spark学习笔记1-编译源代码，已经成功编译了spark on yarn

启动yarn

victor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-namenode-victor-ubuntu.out
localhost: starting datanode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-datanode-victor-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-secondarynamenode-victor-ubuntu.out
starting yarn daemons
starting resourcemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-resourcemanager-victor-ubuntu.out
localhost: starting nodemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-nodemanager-victor-ubuntu.out
victor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ jps
22896 SecondaryNameNode
23383 Jps
22317 NameNode
22578 DataNode
23078 ResourceManager
23342 NodeManager

运行脚本，准备运行的是spark的计算Pi的例子。先来看下准备条件：1.spark客户端目录，那几个文件夹放编译好的jar包。2. 运行的shell

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating$ pwd
/home/victor/software/incubator-spark-0.8.1-incubating

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ pwd
/home/victor/software/incubator-spark-0.8.1-incubating/spark_compiled_client

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ ll
total 28
drwxrwxr-x  5 victor victor 4096  3月 28 21:38 ./
drwxrwxr-x 24 victor victor 4096  3月 28 22:47 ../
drwxrwxr-x  3 victor victor 4096  3月 28 02:41 assembly/
drwxrwxr-x  2 victor victor 4096 12月 11 06:35 conf/
drwxrwxr-x  3 victor victor 4096  3月 28 02:42 examples/
-rwxr-xr-x  1 victor victor 4802 12月 11 06:35 spark-class*

#!/bin/sh

export YARN_CONF_DIR=/home/victor/software/hadoop-2.2.0/etc/hadoop
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar  \
./spark-class org.apache.spark.deploy.yarn.Client   \
--jar ./examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \
--class org.apache.spark.examples.JavaSparkPi \
--args yarn-standalone  \
--num-workers 2  \
--master-memory 400m  \
--worker-memory 512m  \
--worker-cores 1

我的机器配置比较差，这里设置worker和master都比较保守。。。还是比较卡运行的时候。

执行截图：








<原创，转载请注明出处http://blog.csdn.net/oopsoom/article/details/22419597>