spark的几点备忘

spark如果运行在分布式环境下。处理文件需要使用分布式文件系统。比如运行:

<!-- lang: shell -->

./run-example org.apache.spark.examples.SparkPageRank spark://192.168.0.103:7077 /home/xx/data/pagerank_data.txt 100
远程worker如果没有本地文件/home/xx/data/pagerank_data.txt就会报错。

连接master（比如:worker,spark_shell等）报错

<!-- lang: shell -->

13/12/03 14:27:58 ERROR NettyRemoteTransport(null): dropping message RegisterApplication(ApplicationDescription(PageRank)) for non-local recipient akka://sparkMaster@192.168.0.103:7077/user/Master at akka://sparkMaster@error.d:7077 local is akka://sparkMaster@error.d:7077
13/12/03 14:27:58 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://spark@192.168.0.103:58493/user/$a],Actor[akka://sparkMaster@192.168.0.103:7077/user/Master]) for non-local recipient akka://sparkMaster@192.168.0.103:7077/remote at akka://sparkMaster@error.d:7077 local is akka://sparkMaster@error.d:7077
需要使用域名连接，比如

<!-- lang: shell -->

./run-example org.apache.spark.examples.SparkPageRank spark://error.d:7077 /home/xx/data/pagerank_data.txt 100