hadoop 入门学习系列七-----hadoop集群搭建

基础工作

1. 修改主机名和用户名

2. 配置静态ip

3. 配置ssh无密码连接

4. 安装JDK

5. 配置Hadoop

6. 修改配置文件

* hdfs
* 　　hadoop-env.sh
* 　　core-site.xml
* 　　hdfs-site.xml
*　　 slaves
* yarn
* 　　yarn-env.sh
*　　 yarn-site.xml
* 　　slaves
* mapredue
* 　　mapred-env.sh
*　　 mapred-site.xml

在伪分布式的基础上，只需要将slaves文件修改为

slave1

slave2

利用scp将安装文件夹分发到从节点的相同目录下

scp -r /opt/hadoop-2.6.5 hadoop@slave1:/opt

scp -r /opt/hadoop-2.6.5 hadoop@slave2:/opt

开始测试

格式化hdfs

bin/hdfs namenode -format

启动hdfs

sbin/start-dfs.sh

新建文件夹

bin/hdfs dfs -mkdir -p /user/hadoop01/tmp/conf

上传文件

bin/hdfs dfs -put etc/hadoop/*-site.xml /user/hadoop01/tmp/conf

查看文件

bin/hdfs dfs -text /user/hadoop01/tmp/conf/core-site.xml

启动yarn

sbin/start-yarn.sh

运行wordcount示例

bin/hdfs dfs -mkdir -p /user/hadoop01/mapreduce/wordcount/input

bin/hdfs dfs -put /opt/softwares/hadoop-2.6.5/wcinput/* /user/hadoop01/mapreduce/wordcount/input/

bin/hdfs dfs -ls /user/hadoop01/mapreduce/wordcount/input/

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount /user/hadoop01/mapreduce/wordcount/input/ /user/hadoop01/mapreduce/wordcount/output/

查看运行结果

bin/hdfs dfs -text /user/hadoop01/mapreduce/wordcount/output/part*

hdfs界面
http://10.10.0.141:50070/dfshealth.html#tab-overview
yarn界面
http://10.10.0.141:8088/cluster
sbin/mr-jobhistory-daemon.sh start historyserver