Centos 下 Hadoop2.6.4安装 多节点部署

一个初学者安装hadoop，为了下次安装顺利，在这记录一下。

完全安装好之后会有3台环境一样的机器，为了方标识把主节点的主机名改成HadoopMaster，两个从节点分别为HadoopSlave、HadoopSlave2。HadoopMaster与两个从节点之间的通信需要通过ssh(shell+sftp),所以我们需要ssh环境，以及主节点到从节点之间免密码登陆。同时需要修改/etc/hosts文件把各个节点的主机名和ip做映射,干掉127.0.0.1 与localhost的映射。最后两两可以互相ping通主机名

我的安装环境是 ：

Linux：CentOS Linux release 7.2.1511

JDK：1.8.0_11

Openssh：6.6.1p1-22.el7.x86_64

Hadoop:2.6.4

1.java 环境（此处略写）

贴一下环境变量：

export JAVA_HOME=/usr/local/java/jdk1.8.0_11

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools/jar

安装目录自己改好

tips:Centos可能会自己装了OpenJDK，使用rpm -qa |grep jdk 查看 使用rpm -e --nodeps + 包名 卸载

2.Hadoop环境

(1) SSH免密码登陆(在主节点上配置)

在主节点：

# ssh-keygen -t rsa -P ''

这个命令会生成一个id_rsa.pub文件在/root/.ssh/下，把这个文件的名字改成 authorized_keys 等主节点的环境配置好了以后，把这个文件拷贝到从节点下：

# scp /root/.ssh/id_rsa.pub
root@HadoopSlave:/root/.ssh/authorized_key 并且在HadoopSlave上修改它的权限为600.

在主节点上通过

#ssh HadoopSlave测试

(2)Hadoop目录

解压好安装包 ( hadoop-2.6.4.tar.gz ) 后,放到了/usr/lcoal/hadoop下，同时在/usr/local/hadoop 创建tmp、hdfs、hdfs/data 、hdfs/name 目录

(3)Hadoop配置文件修改

(1)etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://HadoopMaster:9000</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/usr/local/hadoop/tmp</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131702</value>

</property>

</configuration>

(2)etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/usr/local/hadoop/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/usr/local/hadoop/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value> HadoopMaster:9001</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

</configuration>

(3)/etc/hadoop/mapred-site.xml.template

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value> HadoopMaster:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value> HadoopMaster:19888</value>

</property>

</configuration>

(4)/etc/hadoop/yarn-site.xml

<?xml version="1.0"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>HadoopMaster:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>HadoopMaster:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>HadoopMaster:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>HadoopMaster:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>HadoopMaster:8088</value>

</property>

<property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>768</value>

</property>

</configuration>

(5)/etc/hadoop/hadoop-env.sh

修改这句为本机的JAVA_HOME就好(如果不知道可以使用# echo $JAVA_HOME 查看)

# The java implementation to use.

export JAVA_HOME=/usr/local/java/jdk1.8.0_11

(6)/etc/hadoop/yarn-env.sh

#echo "run java in $JAVA_HOME"

JAVA_HOME=/usr/local/java/jdk1.8.0_11

(7) /etc/hadoop/slaves

删除默认的localhost 增加自己的从节点 HadoopSlave1 HadoopSlave2(这是主机名，需要在hosts里面对应好)

(8) 复制环境

1.如果使用的是虚拟机环境，所以可以克隆自己的虚拟磁盘。

2.其他方法可以使用scp复制过去 如 scp -r /usr/lcoal/hadoop
root@HadoopSlave:/usr/local/

(9) 添加hadoop到自己的环境变量

修改/etc/profile 添加:

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.4

export PATH=$HADOOP_HOME/bin:$PATH

3.Hadoop启动

(1)格式化自己的文件系统

# hdfs namenode -format

(2)启动

# sbin/start-al.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [HadoopMaster]

HadoopMaster: starting namenode, logging to /usr/local/hadoop/hadoop-2.6.4/logs/hadoop-root-namenode-HadoopMaster.out

HadoopSlave1: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.4/logs/hadoop-root-datanode-HadoopSlave1.out

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.4/logs/yarn-root-resourcemanager-HadoopMaster.out

HadoopSlave1: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.4/logs/yarn-root-nodemanager-HadoopSlave1.out

之类的东西

3.Hadoop自带的测试程序

1.在HDFS上建立一个文件目录

# hadoop fs -mkdir /myinput

2.查看HDFS上的目录结构

# hadoop fs -ls /

drwxr-xr-x - root supergroup 0 2016-03-30 20:53 /myinput

3.复制一个文件到新建的目录下

# hadoop fs -copyFromLocal /usr/local/hadoop/hadoop-2.6.4/etc/hadoop/mapred-site.xml.template /myinput

4.查看这个文件

# hadoop fs -cat /myinput/*

5.运行wordcount 例程

# hadoop jar hadoop-mapreduce-examples-2.6.4.jar wordcount /myinput /output

6.查看结果

# hadoop fs -cat /output/*

输出：

"AS 1

"License"); 1

(the 1

--> 2

2.0 1

<!-- 2

</configuration> 1

</property> 3

<?xml 1

<?xml-stylesheet 1

<configuration> 1

<name>mapreduce.framework.name</name> 1

<name>mapreduce.jobhistory.address</name> 1

<name>mapreduce.jobhistory.webapp.address</name> 1

<property> 3

<value> 2

<value>yarn</value> 1

ANY 1

Apache 1

BASIS, 1

CONDITIONS 1

HadoopMaster:10020</value> 1

HadoopMaster:19888</value> 1

IS" 1

KIND, 1

LICENSE 1

License 3

License, 1

License. 2

Licensed 1

OF 1

OR 1

Put 1

See 2

Unless 1

Version 1

WARRANTIES 1

WITHOUT 1

You 1

a 1

accompanying 1

agreed 1

an 1

and 1

applicable 1

at 1

by 1

compliance 1

copy 1

distributed 2

either 1

except 1

express 1

file 1

file. 2

for 1

governing 1

href="configuration.xsl"?> 1

http://www.apache.org/licenses/LICENSE-2.0 1

implied. 1

in 3

is 1

language 1

law 1

limitations 1

may 2

not 1

obtain 1

of 1

on 1

or 2

overrides 1

permissions 1

property 1

required 1

site-specific 1

software 1

specific 1

the 7

this 2

to 1

type="text/xsl" 1

under 3

use 1

version="1.0"?> 1

with 1

writing, 1

you 1