大数据学习--问题集锦（hadoop篇）--集群搭建

虚拟机复制出3台centos7 配置如下：

ip	hostname
192.168.0.104	earth(既是master也是slave)
192.168.0.105	jupiter
192.168.0.106	mars

问题1：

在master中执行

. /usr/hadoop/sbin/start-all.sh

. /usr/local/spark/spark-2.0.0-bin-hadoop2.7/sbin/start-all.sh

. /usr/local/spark/spark-2.0.0-bin-hadoop2.7/sbin/start-history-server.sh

分别在三台主机中执行jps

结果如下：

master（earth）：

9107 NodeManager

9539 Worker

8772 SecondaryNameNode

8341 NameNode

8967 ResourceManager

8537 DataNode

9197 Master

9999 Jps

jupiter：

23905 Jps

23491 Worker

23368 NodeManager

mars：

23634 Worker

23860 Jps

23335 DataNode

23511 NodeManager

浏览http://earth:50070/dfshealth.html#tab-datanode

In operation交替出现

mars:50010 (192.168.0.106:50010)

earth:50010 (192.168.0.104:50010)

但是却无法出现jupiter,联想到jupiter上的Jps的确缺少DataNode，继续查看jupiter的日志。

vim /usr/hadoop/logs/hadoop-hadoop-datanode-jupiter.log

报错如下：

2017-04-24 23:33:04,735 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b) service to earth/192
.168.0.104:9000 beginning handshake with NN
2017-04-24 23:33:04,878 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b) service t
o earth/192.168.0.104:9000 successfully registered with NN
2017-04-24 23:33:04,987 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action : DNA_REGISTER from earth/192.168.0.104:9000 with active state
2017-04-24 23:33:05,054 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b) service to earth/192
.168.0.104:9000 beginning handshake with NN
2017-04-24 23:33:05,216 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b) service t
o earth/192.168.0.104:9000 successfully registered with NN
2017-04-24 23:33:05,488 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x355bd0c85f9c,  containing 1 storage report(s), of which we sent 0. The reports had 0 total
blocks and used 0 RPC(s). This took 0 msec to generate and 214 msecs for RPC and NN processing. Got back no commands.
2017-04-24 23:33:05,664 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b) service to earth/192
.168.0.104:9000 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Data node DatanodeRegistration(192.168.0.105:50010, datanodeUuid=13bdffc4-13a3-420b-ae63-4f326bcebc3b, inf
oPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-fd1f99e0-a4c4-4115-b8bb-f69828c99b51;nsid=415968335;c=0) is attempting to report storage ID 13bdffc4-13a3-420b-ae63-4f326bcebc3b. N
ode 192.168.0.106:50010 is expected to serve this storage.
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanode(DatanodeManager.java:495)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1788)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1321)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:171)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28756)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:203)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:463)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:688)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
at java.lang.Thread.run(Thread.java:745)
2017-04-24 23:33:05,799 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f3
26bcebc3b) service to earth/192.168.0.104:9000
2017-04-24 23:33:06,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1261824428-192.168.1.118-1490246588550 (Datanode Uuid 13bdffc4-13a3-420b-ae63-4f326bcebc3b)
2017-04-24 23:33:06,067 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1261824428-192.168.1.118-1490246588550
2017-04-24 23:33:08,071 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2017-04-24 23:33:08,074 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2017-04-24 23:33:08,080 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at jupiter/192.168.0.105
************************************************************/

网上查了下解决方案如下：

1、删除了hadoop/tmp/hdfs/data，hdfs/name 目录下的所有文件；

2、重新格式化namenode： bin/hdfs namenode -format；

3、重启集群OK；

Hadoop/spark集群初步搭建完成！

WebUI截图如下：

http://earth:50070/dfshealth.html#tab-datanode



http://earth:8080/



http://192.168.0.104:7777/?showIncomplete=true



在整个部署环境过程中需要注意的几点（会随着今后的使用不断补充）：

1、集群是部署在虚拟机的环境下，firewalld可以选择禁用；如果要启用，务必将需要用到的8080/50070/8088/7777等端口打开；

2、启动HistoryServer时目录需要事先在hdfs上创建好，否则history-server启动报错。

比如我在spark-defaults.conf中配置

spark.eventLog.dir    hdfs://earth:9000/user/hadoop/directory

那么在hdfs中就必须新建user/hadoop/directory （可通过 hdfs dfs -mkdir -p /user/hadoop/directory）