Hadoop/spark安装实战（系列篇3） Hadoop 伪分布模式安装

Hadoop/spark安装实战（系列篇3） Hadoop 伪分布模式安装

Hadoop 的安装与配置 伪分布模式安装

1/解压缩hadoop

tar -zxvf hadoop-1.2.1-bin.tar.gz

2/ 将文件移到/usr/local

mv hadoop-1.2.1 /usr/local

3/设置环境变量HADOOP_HOME，修改文件“/etc/profile”

[root@localhost local]# vi /etc/profile

export HADOOP_HOME=/usr/local/hadoop-1.2.1

export PATH=.:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

生效

[root@localhost local]# source /etc/profile

检验

[root@localhost local]# echo $PATH

.:.:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/l

ocal/jdk1.7.0_79/bin:/root/bin:/usr/local/jdk1.7.0_79/bin:/usr/local/ha

doop-1.2.1/bin

[root@localhost local]#

4 修改hadoop 环境变量脚本文件hadoop-env.sh

增加

export JAVA_HOME=/usr/local/jdk1.7.0_79

5 修改hadoop 核心配置文件core-site.xml

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/tmp</value>

<description> hadoop.tmp.dir</description>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://spark0:9000</value>

<description>HDFS fs.default.name</description>

</property>

</configuration>

6 hdfs 配置文件hdfs-site.xml，存储副本数1份

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

<description>1</description>

</property>

</configuration>

7 MapReduce 配置文件mapred-site.xml

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>spark0:9001</value>

<description>JobTracker </description>

</property>

</configuration>

8格式化文件系统

在第一次使用之前需要进行格式化，只在第一次启动的时候格式化，不要每次启

动都格式化

[root@localhost bin]# hadoop namenode -format

15/09/12 06:31:11 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = localhost.localdomain/127.0.0.1

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 1.2.1

STARTUP_MSG: build =

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r

1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG: java = 1.7.0_79

************************************************************/

15/09/12 06:31:13 INFO util.GSet: Computing capacity for map BlocksMap

15/09/12 06:31:13 INFO util.GSet: VM type = 32-bit

15/09/12 06:31:13 INFO util.GSet: 2.0% max memory = 1013645312

15/09/12 06:31:13 INFO util.GSet: capacity = 2^22 = 4194304

entries

9 启动hadoop 程序

[root@localhost bin]# start-all.sh

[root@localhost bin]# start-all.sh

Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-namenode-localhost.localdomain.out

localhost: starting datanode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-datanode-localhost.localdomain.out

localhost: starting secondarynamenode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-secondarynamenode-

localhost.localdomain.out

starting jobtracker, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-jobtracker-localhost.localdomain.out

localhost: starting tasktracker, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-tasktracker-localhost.localdomain.out

检验

[root@localhost bin]# jps

[root@localhost bin]# jps

27375 Jps

26949 DataNode

27046 SecondaryNameNode

26851 NameNode

27240 TaskTracker

27124 JobTracker

[root@localhost bin]#

10 检验 在本地物理机浏览器输入

http://192.168.2.100:50070/dfshealth.jsp

NameNode 'spark0:9000'

Started: Sat Sep 12 06:34:38 PDT 2015

Version: 1.2.1, r1503152

Compiled: Mon Jul 22 15:23:09 PDT 2013 by mattf

Upgrades: There are no upgrades in progress.

Browse the filesystem

Namenode Logs

-----------------------------------------------------------------------

---------

Cluster Summary

7 files and directories, 1 blocks = 8 total. Heap Size is 31.32 MB /

966.69 MB (3%)

Configured Capacity : 5.65 GB DFS Used : 28.01 KB Non DFS

Used : 3.17 GB DFS Remaining : 2.47 GB DFS Used% : 0 % DFS

Remaining% : 43.82 % Live Nodes : 1 Dead Nodes : 0

Decommissioning Nodes : 0 Number of Under-Replicated Blocks : 0

-----------------------------------------------------------------------

---------

NameNode Storage:

Storage Directory Type State /home/hadoop/tmp/dfs/name IMAGE_AND_EDITS

Active

-----------------------------------------------------------------------

---------

This is Apache Hadoop release 1.2.1

至此 hadoop安装完成