spark2如何集成到cdh里

最近做性能测试需要spark2测试下和spark1.6性能有多大差别，官方文档里写着可以集成，但是自己怎么搞都不行，折磨了3天的时间，目前终于把spark2集成到集群里了

我安装的是最新版本的

下载spark2安装包

wget http://archive.cloudera.com/beta/spark2/parcels/latest/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel
[root@namenode01 parcel-repo]# wget http://archive.cloudera.com/beta/spark2/parcels/2.0.0.cloudera.beta2/manifest.json
[root@namenode01 parcel-repo]# chown cloudera-scm.cloudera-scm manifest.json SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel

[root@namenode01 parcel-repo]# echo "9501a3b45add128d9d3fedcccc4797518a87769b" > SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel.sha

[root@namenode01 csd]# pwd

/opt/cloudera/csd

wget http://archive.cloudera.com/beta/spark2/csd/SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar
[root@namenode01 csd]# ls

SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar

[root@namenode01 csd]# chown cloudera-scm:cloudera-scm SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar ;chmod 644 SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar 

几面上重启cm

之后把之前的旧的文件cp到spark2目录下，只需修改下spark2的配置文件即可。

不要纠结spark2是否在cm管理界面的添加服务里是否可以看到spark2服务。

[root@namenode01 bin]# cp  /etc/spark/conf/classpath.txt  /opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234 /etc/spark2/conf.dist/

[root@namenode01 bin]# cp  /etc/spark/conf/spark-env.sh  /opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234 /etc/spark2/conf.dist/

 vi /etc/spark2/conf/spark-env.sh
export SPARK_HOME=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
这个地方要改成spark2的

[root@datanode01 ~]# spark2-shell 

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Spark context Web UI available at http://10.1.8.101:4040
Spark context available as 'sc' (master = local[*], app id = local-1497409518846).

Spark session available as 'spark'.

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1

      /_/

         

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)

Type in expressions to have them evaluated.

Type :help for more information.

scala>