Hadoop2.0单机环境安装部署

步骤1：将安装包hadoop-2.2.0.tar.gz存放到某一个目录下，并解压

步骤2：修改解压后的目录中的文件夹etc/hadoop下的xml配置文件(如果文件不存在，则自己创建)

hadoop-evn.sh修改

export JAVA_HOME=/usr/java/jdk1.6.0_12

slaves文件修改为(可改也不改，如果改，则要在/etc/hosts中相应加上，本文是加上了)

YARN0001

如果是本机，默认的是localhost，就不用修改

以下XML配置文件，需在标签<configuration>和</configuration>之间增加配置项

mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

core-site.xml(其中YARN s是在/etc/hosts中设置的host,如果未设置，则换为localhost)

<property>

<name>fs.default.name</name>

<value>hdfs://YARN001:8020</value>

</property>

yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>/usr/local/hadoop2/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/usr/local/hadoop2/data</value>

</property>

一步步来

bin/hadoop namenode -format

sbin/hadoop-daemon.sh start namenode

sbin/hadoop-daemon.sh start datanode

jps

yarn001:50070/

sbin/yarn-daemon.sh start yarn

sbin/start-yarn.sh 启动yarn

yarn001:8088/

sbin/stop-yarn.sh 关闭yarn

跑一个测试用例

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 100

使用start-yarn.sh和start-dfs.sh启动

[user@fakeDistnode hadoop-2.2.0]$ sbin/start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-resourcemanager-fakeDistnode.out

localhost: starting nodemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-nodemanager-fakeDistnode.out

[user@fakeDistnode hadoop-2.2.0]$ jps

3474 NameNode

3865 NodeManager

3761 ResourceManager

4059 Jps

3591 DataNode

[user@fakeDistnode hadoop-2.2.0]$ sbin/stop-yarn.sh

stopping yarn daemons

stopping resourcemanager

localhost: stopping nodemanager

no proxyserver to stop

[user@fakeDistnode hadoop-2.2.0]$ sbin/stop-dfs.sh

Stopping namenodes on [YARN001]

The authenticity of host 'yarn001 (127.0.0.1)' can't be established.

RSA key fingerprint is 86:8e:c7:42:6b:df:ea:76:e9:42:34:a6:b5:0f:96:96.

Are you sure you want to continue connecting (yes/no)? yes

YARN001: Warning: Permanently added 'yarn001' (RSA) to the list of known hosts.

YARN001: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is 86:8e:c7:42:6b:df:ea:76:e9:42:34:a6:b5:0f:96:96.

Are you sure you want to continue connecting (yes/no)? yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: no secondarynamenode to stop

[user@fakeDistnode hadoop-2.2.0]$ sbin/start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-resourcemanager-fakeDistnode.out

localhost: starting nodemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-nodemanager-fakeDistnode.out

[user@fakeDistnode hadoop-2.2.0]$ jps

3474 NameNode

3865 NodeManager

3761 ResourceManager

4059 Jps

3591 DataNode

[user@fakeDistnode hadoop-2.2.0]$ sbin/stop-yarn.sh

stopping yarn daemons

stopping resourcemanager

localhost: stopping nodemanager

no proxyserver to stop

[user@fakeDistnode hadoop-2.2.0]$ sbin/stop-dfs.sh

Stopping namenodes on [YARN001]

The authenticity of host 'yarn001 (127.0.0.1)' can't be established.

RSA key fingerprint is 86:8e:c7:42:6b:df:ea:76:e9:42:34:a6:b5:0f:96:96.

Are you sure you want to continue connecting (yes/no)? yes

YARN001: Warning: Permanently added 'yarn001' (RSA) to the list of known hosts.

YARN001: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is 86:8e:c7:42:6b:df:ea:76:e9:42:34:a6:b5:0f:96:96.

Are you sure you want to continue connecting (yes/no)? yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: no secondarynamenode to stop

如果某个节点无法启动可以类似启动，如果是namenode，把下面的datanode改为namenode即可

[user@fakeDistnode hadoop-2.2.0]$ sbin/hadoop-daemon.sh start datanode

starting datanode, logging to /usr/local/hadoop-2.2.0/logs/hadoop-user-datanode-fakeDistnode.out

[user@fakeDistnode hadoop-2.2.0]$ jps

3474 NameNode

3591 DataNode

3632 Jps

那么使用start-yarn.sh和start-dfs.sh后，什么时候某个节点可能启动呢，我遇到了namenode无法启动的情况：

[user@centos64 hadoop-2.2.0]$ sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [localhost]

localhost: ssh: localhost: Temporary failure in name resolution

centos64: starting datanode, logging to /usr/local/hadoop-2.2.0/logs/hadoop-user-datanode-centos64.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.2.0/logs/hadoop-user-secondarynamenode-

centos64.out

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-resourcemanager-centos64.out

centos64: starting nodemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-user-nodemanager-centos64.out

原因是core-site.xml 我把fs.default.name 写成了hdfs://localhost:8020

如果没有 ssh-keygen要交互输入机器的账号密码（http://blog.csdn.net/lzlchangqi/article/details/13397415 实现SSH无密码登陆）

[hadoop@localhost hadoop-2.6.0]$ sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

15/03/11 04:47:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/03/11 04:47:53 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:47:53 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:47:54 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:47:54 WARN conf.Configuration: bad conf file: element not <property>

Starting namenodes on [YARN001]

The authenticity of host 'yarn001 (127.0.0.1)' can't be established.

RSA key fingerprint is e5:5f:ff:0e:13:a8:8f:20:dc:93:8b:40:ff:3e:77:40.

Are you sure you want to continue connecting (yes/no)? yes

YARN001: Warning: Permanently added 'yarn001' (RSA) to the list of known hosts.

hadoop@yarn001's password:

YARN001: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-namenode-localhost.localdomain.out

hadoop@yarn001's password:

YARN001: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-datanode-localhost.localdomain.out

Starting secondary namenodes [0.0.0.0]

hadoop@0.0.0.0's password:

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out

15/03/11 04:50:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

(请参考：http://blog.csdn.net/iloveyin/article/details/28909771 或http://blog.itpub.net/20777547/viewspace-1147174从这下人家编译好的,当然自己也可以下载源代码后自己编译)

15/03/11 04:50:42 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:50:42 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:50:43 WARN conf.Configuration: bad conf file: element not <property>

15/03/11 04:50:43 WARN conf.Configuration: bad conf file: element not <property>

starting yarn daemons

resourcemanager running as process 25218. Stop it first.

hadoop@yarn001's password:

hadoop@yarn001's password: YARN001: Permission denied, please try again.

YARN001: starting nodemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out

[hadoop@localhost hadoop-2.2.0]$ jps

26222 NodeManager

25688 NameNode

26002 SecondaryNameNode

25796 DataNode

25218 ResourceManager

26254 Jps

[user@centos64 hadoop-2.2.0]$ bin/hdfs dfs -ls /

Found 2 items

drwx------ - user supergroup 0 2015-05-30 22:53 /tmp

drwxr-xr-x - user supergroup 0 2015-05-30 22:53 /user

[user@centos64 hadoop-2.2.0]$ bin/hdfs dfs -ls -R /

drwx------ - user supergroup 0 2015-05-30 23:14 /tmp

drwxr-xr-x - user supergroup 0 2015-05-30 22:53 /user

drwxr-xr-x - user supergroup 0 2015-05-30 22:55 /user/user

[user@centos64 hadoop-2.2.0]$ bin/hdfs dfs -mkdir /user/user/pigtest

[user@centos64 hadoop-2.2.0]$ bin/hdfs dfs -ls -R /

drwx------ - user supergroup 0 2015-05-30 23:14 /tmp

drwxr-xr-x - user supergroup 0 2015-05-30 22:53 /user

drwxr-xr-x - user supergroup 0 2015-05-30 23:16 /user/user

drwxr-xr-x - user supergroup 0 2015-05-30 23:16 /user/user/pigtest

[user@centos64 Desktop]$ hdfs dfs -put /home/user/Desktop/student.txt /user/user/pigtest

[user@centos64 Desktop]$ hdfs dfs -put /home/user/Desktop/teacher.txt /user/user/pigtest

[user@centos64 Desktop]$ hdfs dfs -ls /user/user/pigtest/

Found 2 items

-rw-r--r-- 1 user supergroup 124 2015-05-30 23:46 /user/user/pigtest/student.txt

-rw-r--r-- 1 user supergroup 38 2015-05-30 23:48 /user/user/pigtest/teacher.txt

[user@centos64 Desktop]$