Spark运行模式:cluster与client
2017-08-31 11:26
369 查看
When run SparkSubmit --class [mainClass], SparkSubmit will call a childMainClass which is
1. client mode, childMainClass = mainClass
2. standalone cluster mde, childMainClass = org.apache.spark.deploy.Client
3. yarn cluster mode, childMainClass = org.apache.spark.deploy.yarn.Client
The childMainClass is a wrapper of mainClass. The childMainClass will be called in SparkSubmit, and if cluster mode, the childMainClass will talk to the the cluster and launch a process on one woker to run the mainClass.
ps. use "spark-submit -v" to print debug infos.
Yarn client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn JavaWordCount.jar
childMainclass: org.apache.spark.examples.JavaWordCount
Yarn cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn-cluster JavaWordCount.jar
childMainclass: org.apache.spark.deploy.yarn.Client
Standalone client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 JavaWordCount.jar
childMainclass: org.apache.spark.examples.JavaWordCount
Stanalone cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 --deploy-mode cluster JavaWordCount.jar
childMainclass: org.apache.spark.deploy.rest.RestSubmissionClient (if rest, else org.apache.spark.deploy.Client)
Taking standalone spark as example, here is the client mode workflow. The mainclass run in the driver application which could be reside out of the cluster.
相关文章推荐
- Spark Client和Cluster两种运行模式的工作流程
- Spark的运行模式(2)--Yarn-Cluster和Yarn-Client
- spark在yarn上面的运行模型:yarn-cluster和yarn-client两种运行模式:
- spark on yarn 两种运行模式(client 、cluster)对比
- Spark on yarn client 和cluster模式运行序列图
- Spark Client和Cluster两种运行模式的工作流程
- Spark运行模式:cluster与client
- Spark运行模式(local standalond,yarn-client,yarn-cluster,mesos-client,mesos-cluster)
- Spark on YARN cluster & client 模式作业运行全过程分析
- Spark on YARN client模式作业运行全过程分析
- Spark-2.1.2 Hadoop-2.7.5 spark-submit yarn client cluster两种模式提交
- 在standalone-cluster模式上运行spark应用程序(用sbt打包)
- Spark On Yarn的两种模式yarn-cluster和yarn-client深度剖析
- Spark on Yarn Client和Cluster模式详解
- 一 spark on yarn cluster模式提交作业,一直处于ACCEPTED状态,改了Client模式后就正常了
- 从源码角度看Spark on yarn client & cluster模式的本质区别
- Spark on yarn有分为两种模式yarn-cluster和yarn-client
- 从源码角度看Spark on yarn client & cluster模式的本质区别
- 27课 :SPARK 运行在yarn资源调度框架 client 、cluster方式 !!
- Spark集群运行模式