Spark standalone模式结合虚拟机遇到的问题

在spark standalone模式启动结合虚拟机遇到的driver url获取问题

学习博客：http://bit1129.iteye.com/blog/2179543

在job启动后，输出如下问题：
17/12/22 10:01:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

定位问题：
查看worker ui中executor的stdout和stderr。stderr输出如下：
Caused by: org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply from192.168.56.1:61266
in 120 seconds. This timeout is controlled by spark.rpc.askTimeout
全部过程中并没有使用该ip，但是考虑到启动该job时并未设置任何参数，所以应该是环境变量设置问题。查看ip地址，发现192.168.56.1为虚拟机网卡ip。
然后查看worker的工作日志，也就是worker链接到master后提示的日志文件，通常在logs目录下，有如下信息：
17/12/21 17:00:55 INFO Worker: Asked to launch executor app-20171221165506-0000/5 for spark_main
17/12/21 17:00:55 INFO SecurityManager: Changing view acls to: user
17/12/21 17:00:55 INFO SecurityManager: Changing modify acls to: user
17/12/21 17:00:55 INFO SecurityManager: Changing view acls groups to: 
17/12/21 17:00:55 INFO SecurityManager: Changing modify acls groups to: 
17/12/21 17:00:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(user); groups with view permissions: Set(); users  with modify permissions: Set(user);
groups with modify permissions: Set()
17/12/21 17:00:55 INFO ExecutorRunner: Launch command: 
"/home/user/java/jdk1.8.0_151/bin/java" 
"-cp" "/home/user/spark/spark-2.2.1-bin-hadoop2.7/conf/:/home/user/spark/spark-2.2.1-bin-hadoop2.7/jars/*" "-Xmx1024M" 
"-Dspark.driver.port=61266" 
"org.apache.spark.executor.CoarseGrainedExecutorBackend" 
"--driver-url" "spark://CoarseGrainedScheduler@192.168.56.1:61266" 
"--executor-id" "5" 
"--hostname" "10.16.143.60" 
"--cores" "1" 
"--app-id" "app-20171221165506-0000" 
"--worker-url" "spark://Worker@10.16.143.60:40670"
通过launch command命令发现，该ip是当worker启动job时传递的driver ip，根据官方文档，可以手动设置driver ip。

解决办法：
通过启动时设置“spark.driver.host”解决该问题。