3，数据挖掘环境搭建-Hadoop集群搭建与测试

安装hadoop-2.6.5.tar.gz

hdp10 namenode/datanode

hdp11 datanode

hdp12 datanode

0 安装前准备

修改hosts，修改主机名，关闭防火墙，关闭selinux,安装ssh，配置免密码，安装JDK

1,添加hadoop用户名

useradd hadoop

passwd hadoop

vi /etc/sudoers

root       ALL=(ALL)   ALL
hadoop ALL=(ALL)  ALL

2,修改/home/hadoop/apps/hadoop-2.6.5/etc/hadoop下的配置文件

2.1，修改hadoop-env.sh

vi hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_144

2.2 修改core-site.xml

vi core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hdp10:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hdpdata</value>
</property>
</configuration>

2.3，修改hdfs-site.xml

vi hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hdpdata</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>hdp10:50070</value>
</property>
<property>
<name>dfs.http.address</name>
<value>hdp10:50090</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/datanode</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>/home/hadoop/namenode</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>

2.4 修改mapred-site.xml

vi mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

2.5，修改yarn-site.xml

vi yarn-site.xml

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hdp10</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

3，将hadoop文件添加到系统环境变量

vi /etc/profile

export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

4，修改slaves，设置datanode与nodemanager

vi /home/hadoop/apps/hadoop-2.6.5/etc/hadoop/slaves

hdp10

hdp11

hdp12

5，将/etc/profile和hadoop文件夹拷贝至hdp11,hdp12相应的位置

scp /etc/profile root@hdp11:/etc

scp /etc/profile root@hdp12:/etc

scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp11:/home/hadoop/apps

scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp12:/home/hadoop/apps

chown -R hadoop:hadoop /home/hadoop/apps/hadoop-2.6.5

6，去除HDFS WARN util.NativeCodeLoader

vi /home/hadoop/apps/hadoop-2.5.2/etc/hadoop/log4j.properties

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

7，格式化hadoop文件系统

hdfs namenode -format

8，启动hadoop

cd /home/hadoop/apps/hadoop-2.6.5/sbin

start-dfs.sh，启动HDFS

stop-dfs.sh，关闭HDFS

start-yarn.sh，启动YARN

stop-yarn.sh， 关闭YARN

9，查看启动状态

jps

10，范例

vi 1.txt

hello world hello count
hello class
hello rose
hello jerry
hello class
hello rose

hadoop fs -mkdir -p /wordcount/input

hadoop fs -put 1.txt /wordcount/input

cd /home/hadoop/apps/hadoop-2.6.5/share/hadoop/mapreduce

hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /wordcount/input /wordcount/output0