Spark3.0.0 on yarn (基于hadoop3.2 ) 分布式安装 - 避坑指南

1 centos安装scala

1.下载：
https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz

2.解压至【/usr/local】

tar -xvzf scala-2.11.12.tgz

3.配置环境变量

vim /etc/profile

#scala
export SCALA_HOME=/usr/local/scala-2.11.12
export PATH=$SCALA_HOME/bin:$PATH

配置完成

2 安装spark

dockerapache-01
dockerapache-02
dockerapache-03 #master

1.下载spark，
http://spark.apache.org/downloads.html
这里由于我的hadoop是3.1.3, 所以选择基于3.2的进行尝试

2. 上传至master节点上，解压至【/usr/local】

tar -xvzf spark-3.0.0-preview2-bin-hadoop3.2.tgz

3.配置conf/spark-env.sh

cp spark-env.sh.template spark-env.sh
vim spark-env.sh

SPARK_CONF_DIR=/usr/local/spark-3.0.0-preview2-bin-hadoop3.2/conf
HADOOP_CONF_DIR=/usr/local/hadoop-3.1.3/etc/hadoop
YARN_CONF_DIR=/usr/local/hadoop-3.1.3/etc/hadoop

SPARK_MASTER_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER-Dspark.deploy.zookeeper.url=DockerApache-01:2181,DockerApache-01:2181,DockerApache-01:2181-Dspark.deploy.zookeeper.dir=/spark"

4.配置 conf/slaves

cp slaves.template slaves
vim slaves

dockerapache-01
dockerapache-02
dockerapache-03

将master节点安装包拷贝至其他节点

scp -r /usr/local/spark-3.0.0-preview2-bin-hadoop3.2/ DockerApache-02:`pwd`
scp -r /usr/local/spark-3.0.0-preview2-bin-hadoop3.2/ DockerApache-01:`pwd`

启动spark

在master节点sbin目录下：

./start-all.sh

在三台机器上分别jps，会发现三台机器上都有worker进程，或者master进程

启动客户端（yarn）

./spark-shell --master yarn --deploy-mode client

问题

启动spark shell出错：

2020-04-27 10:23:49,894 ERROR cluster.YarnClientSchedulerBackend: YARN application has exited unexpectedly with state UNDEFINED! Check the YARN application logs for more details.
2020-04-27 10:23:49,895 ERROR cluster.YarnClientSchedulerBackend: Diagnostics message: Shutdown hook called before final status was reported.
2020-04-27 10:23:49,919 ERROR spark.SparkContext: Error initializing SparkContext.

2020-04-27 10:23:49,936 ERROR client.TransportClient: Failed to send RPC RPC 9166854326805066924 to /10.28.3.12:39182: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException

配置hadoop下yarn-site.xml，添加以下配置

<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>