nodemanager报错 spark.network.yarn.YarnShuffleService not found

HDP2.4版本

安装启动后nodemanager报错，无法启动，查看日志发现，

2016-03-11 10:51:23,638 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(540)) - Error starting NodeManager
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)

        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)

        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)

        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)

        ... 10 more

Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

        ... 11 more

2016-03-11 10:51:23,640 INFO  nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NodeManager at mapp3

************************************************************/

分析解决：

网上搜索了下，缺少org.apache.spark.network.yarn.YarnShuffleService相关的jar包文件，没有 spark-yarn-shuffle.jar包

查看正常节点上的spark*.jar包，

[root@map1 ~]# find / -name spark*.jar

/usr/hdp/2.4.0.0-169/spark/lib/spark-1.6.0.2.4.0.0-169-yarn-shuffle.jar

/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar

异常节点上没有这个包

[root@map3 ~]# find / -name spark*.jar

[root@map3 ~]#

将jar拷贝到异常节点

[root@map1 ~]#scp /usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar 172.16.16.41:/usr/hdp/2.4.0.0-169/hadoop/lib/

启动nodemanager正常了，这个包应该是spark on yarn上需要使用。