您的位置：首页 > 运维架构 > Linux

Centos6.5 x64 安装hadoop2.2.0

2014-08-06 16:48 387 查看

规划:

cloud01、cloud02 Namenode HA

cloud03 YARN

cloud04\05\06 ZK节点、Datanode

机器名 IP地址进程

cloud01 192.168.1.201 jdk hadoop NameNode、DFSZKFailoverController

cloud02 192.168.1.202 jdk hadoop NameNode、DFSZKFailoverController

cloud03 192.168.1.203 jdk hadoop ResourceManager

cloud04 192.168.1.204 jdk hadoop zookeeper DataNode、NoderManager、JouranlNode、QuorumPeerMain

cloud05 192.168.1.205 jdk hadoop zookeeper DataNode、NoderManager、JouranlNode、QuorumPeerMain

cloud06 192.168.1.206 jdk hadoop zookeeper DataNode、NoderManager、JouranlNode、QuorumPeerMain

说明:

---管理YARN yarn,Yet Another Resource Negotiator ，是 Hadoop 自 0.23.0 版本后新的 map-reduce 框架

---JouranlNode 数据实时同步 YARN 管理多个NodeManager QuorumPeerMain是ZooKeeper的进程

hadoop2.0 namenode 有HA功能需要安装zookeeper (Master选举) ZK需要要奇数个节点

hadoop2.0 有两个NamdeNode组成，一个处于active，一个standby

安装步骤：

1、先在datanode 节点上安装jdk和zookeeper

1.1 下载jdk 解压到/usr/local目前下 /usr/local/jdk1.8.0_05

1.2 设置环境变量/etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_05

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

export PATH

jps验证

1.3 创建mycloud目录，解压zookeeper-3.4.5.tar.gz 至这个文件目录下/mycloud/zookeeper-3.4.6 （之后hadoop也放到这个目录下）

创建/mycloud/zookeeper-3.4.6/data 目录，在该目录下创建文件myid 填入相应的server.id号，一行一个id号

修改conf目录下的zoo.cfg cp zoo_sample.cfg zoo.cfg

修改 zoo.cfg dataDir=/mycloud/zookeeper-3.4.6/data

最后添加:

server.1=cloud04:2888:3888 通讯端口:选举端口

server.2=cloud05:2888:3888 通讯端口:选举端口

server.3=cloud06:2888:3888 通讯端口:选举端口

1.4 启动zk （ZK的三个节点都要启动）

[root@cloud04 bin]# ./zkServer.sh start

jps验证:

看到QuorumPeerMain进程，即启动成功。

查看ZK服务状态：

[root@hnode03 bin]# ./zkServer.sh status

JMX enabled by default

Using config: /mycloud/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower / Mode: leader （两种模式）

说明:leader是主节点，follower是从节约点

2、安装hadoop

下载 hadoop-2.2.0.tar.gz

tar -xzvf hadoop-2.2.0.tar.gz -C /mycloud/

5个配置文件 hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml

（生产中最好需要两台独立的机器来做NameNode的HA）

hadoop-env.sh 修改第27行 JAVA_HOME

core-site.xml 修改内容如下:

<configuration>



<property>

<name>fs.defaultFS</name>

<value>hdfs://ns</value>

</property>



<property>

<name>hadoop.tmp.dir</name>

<value>/mycloud/hadoop-2.2.0/temp</value>

</property>



<property>

<name>ha.zookeeper.quorum</name>

<value>cloud04:2181,cloud05:2181,cloud06:2181</value>

</property>

</configuration>

hdfs-site.xml 修改内容如下:





<configuration>

<property>

<name>dfs.nameservices</name>

<value>ns</value>

</property>



<property>

<name>dfs.ha.namenodes.ns</name>

<value>cloud01,cloud02</value>

</property>



<property>

<name>dfs.namenode.rpc-address.ns.cloud01</name>

<value>cloud01:8020</value>

</property>



<property>

<name>dfs.namenode.http-address.ns.cloud01</name>

<value>cloud01:50070</value>

</property>



<property>

<name>dfs.namenode.rpc-address.ns.cloud02</name>

<value>cloud02:8020</value>

</property>



<property>

<name>dfs.namenode.http-address.ns.cloud02</name>

<value>cloud02:50070</value>

</property>



<property>

<name>dfs.namenode.share.edits.dir</name>

<value>qjournal://cloud04:8485;cloud05:8485;cloud06:8485/ns</value>

</property>



<property>

<name>dfs.journalnode.edits.dir</name>

<value>/mycloud/hadoop-2.2.0/journal</value>

</property>



<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>



<property>

<name>dfs.client.failover.proxy.provider.ns</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfigureFailoverProxyProvider</value>

</property>



<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>



<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/root/.ssh/id_rsa</value>

</property>

</configuration>

vi mapred-site.xml

<configuration>

<-- 指定Mr框架为YARN方式 -->

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

vi yarn-site.xml

<configuration>

<-- 指定resourcemanager地址 -->

<property>

<name>yarn.resourcemanager.hostname</name>

<value>cloud03</value>

</property>

<-- 指定nodemanager启动时加载server的方式为mapreduce_shuffle -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

3、ssh配置免密码登录

#ssh-keygen -t rsa

#ls

id_rsa id_rsa.pub

#ssh-copy-id -i cloud02 （到目标机免登录，把公钥拷至目标机）操作时输入目标机的密码

cloud01-->cloud02 前配好的hadoop拷至02、03、04、05、06

#scp -r /mycloud/ cloud02:/

#scp -r /mycloud/ cloud03:/

#scp -r /mycloud/ cloud04:/

#scp -r /mycloud/ cloud05:/

#scp -r /mycloud/ cloud06:/

4、配置环境变量

在第一台机器上添加一个HADOOP_HOME环境变量

/etc/profile

export JAVA_HOME=/usr/local/jdk1.6.0_45

export HADOOP_HOME=/project/hadoop-2.2.0

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

source /etc/profile

将profile拷至其他节点，同步

scp /etc/profile cloud02:/etc

scp /etc/profile cloud03:/etc

scp /etc/profile cloud04:/etc

scp /etc/profile cloud05:/etc

scp /etc/profile cloud06:/etc

5、配置slaves节点（即运行JouralNode的节点，它的管理节点是:ResourceManager，也就是YARN）

vi /hadoop-2.2.0/etc/hadoop/slaves

cloud04

cloud05

cloud06

6、全部配置好之后，将hadoop-2.2.0拷贝到其他机器

scp -r /mycloud/hadoop-2.2.0 cloud02:/mycloud/

scp -r /mycloud/hadoop-2.2.0 cloud03:/mycloud/

scp -r /mycloud/hadoop-2.2.0 cloud04:/mycloud/

scp -r /mycloud/hadoop-2.2.0 cloud05:/mycloud/

scp -r /mycloud/hadoop-2.2.0 cloud06:/mycloud/

7、启动

7.1 先启动zk(每台机器)

cd /project/zookeeper-3.4.5/bin/

./zkServer.sh start

./zkServer.sh status

(一个leader，两个follower)

7.2 再启动journalnode（只需要在一台机器运行即可，其他机器将自动通过ssh运行。如果只运行自己的，是hadoop-deamon.sh，多台一起运行的是

hadoop-deamons.sh，运行的这些机器，也是安装了ZooKeeper的机器）

sbin/hadoop-deamons.sh start journalnode

验证:jps 查看Jouranlnode进程有没有运行

sbin/hadoop-deamons.sh start journalnode (hadoop-deamon.sh运行本机journalnode hadoop-deamons.sh 其他机器也一起运行)

[root@cloud01 hadoop-2.2.0]# sbin/hadoop-daemons.sh start journalnode

8、格式化HDFS文件系统:

在cloud01（即NameNode的其中一个节点）上执行命令

#bin/hdfs haadmin DFSHAadmin –transitionToActive ns 验证HA条件

hadoop namenode -format

格式化后在根据core-site.xml中的hadoop

两个Namenode，格式化一个后，拷到另一个也作为格式化

9、将格式后之后，生成的HDFS的元数据拷贝至第2个NameNode节点

scp -r tmp/ cloud02:/project/hadoop-2.2.0/ （拷贝的目录tmp是执行格式化命令后生成的）

10、格式化zk（在cloud01上执行即可）

hdfs zkfc -formatZK

cd /project/zookeeper-3.4.5/bin/

./zkCli.sh

ls /

[hadoop-ha,zookeeper] 看到多一个hadoop-ha目录，说明格式化成功

11、启动HDFS(在cloud01上执行即可)

sbin/start-dfs.sh

12、启动YARN(在cloud03上执行)

sbin/start-yarn.sh

jps 查看进程

可以查看到多一个ResourceManager进程，被管理的节点为NodeManager

测试环境与应用:

切换Namenode 的两个服务器需要相互免密码登录

强制kill cloud01的NameNode进程后，cloud02自动转为Active (cloud02到cloud01也必须是ssh免密码登录)

cloud02重新启动后，变为Standby

sbin/hadoop-daemon.sh start namenode

hadoop fs -put /root/install.log / 上传文件

hadoop fs -ls 查看文件

hadoop fs -get /install.log /home/111.log 下载文件

hadoop jar /project/hadoop-2.2.0/.../hadoop-mapreduce-examples-2.2.0.jar wordcount /file.txt /resultout.txt

hadoop -fs -cat /resultout/part-r-0000 查看mapreduce结果文件

错误：

sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /mycloud/hadoop-2.2.0/logs/hadoop-root-journalnode-cloud01.out

[Fatal Error] core-site.xml:20:4: 元素内容必须由格式正确的字符数据或标记组成。

Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/mycloud/hadoop-2.2.0/etc/hadoop/core-site.xml; lineNumber: 20; columnNumber: 4; 元素内容必须由格式正确的字符数据或标记组成。

at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2152)

at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2001)

at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1918)

at org.apache.hadoop.conf.Configuration.set(Configuration.java:817)

at org.apache.hadoop.conf.Configuration.set(Configuration.java:792)

at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1114)

at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)

at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:431)

原因: 将property 写成了porperty

<property>

<name>fs.defaultFs</name>

<value>hdfs://nnode</value>

</property>

S大写

<name>fs.defaultFs</name> <name>fs.defaultFS</name>

<--指定hdfs的nameservice为ns1--> 注释方法错误，少了叹号 

sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /mycloud/hadoop-2.2.0/logs/hadoop-root-journalnode-cloud01.out

[Fatal Error] hdfs-site.xml:20:33: 注释中不允许出现字符串 "--"。

Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/mycloud/hadoop-2.2.0/etc/hadoop/hdfs-site.xml; lineNumber: 20; columnNumber: 33; 注释中不允许出现字符串 "--"。

at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2152)

at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2001)

at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1918)

at org.apache.hadoop.conf.Configuration.set(Configuration.java:817)

at org.apache.hadoop.conf.Configuration.set(Configuration.java:792)

at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1114)

at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)

at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:431)



原因：注释的内容中间还有 --

第一次格式化:HA没有true 见365，原因在hdfs-site.xml中

<property>

<name>dfs.client.failover.proxy.provider.nnode</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfigureFailoverProxyProvider</value>

</property> 这里配置错了，少了Provider

[root@cloud01 hadoop-2.2.0]# hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

14/06/15 22:47:28 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = cloud01/192.168.214.171

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 2.2.0

STARTUP_MSG: classpath = /mycloud/hadoop-2.2.0/etc/hadoop:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/mycloud/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/mycloud/hadoop-2.2.0/contrib/capacity-scheduler/*.jar

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z

STARTUP_MSG: java = 1.8.0_05

************************************************************/

14/06/15 22:47:28 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /mycloud/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

14/06/15 22:47:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Formatting using clusterid: CID-64625d97-d90d-490c-9d8d-f09a0735676d

14/06/15 22:47:29 INFO namenode.HostFileManager: read includes:

HostSet(

)

14/06/15 22:47:29 INFO namenode.HostFileManager: read excludes:

HostSet(

)

14/06/15 22:47:29 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000

14/06/15 22:47:29 INFO util.GSet: Computing capacity for map BlocksMap

14/06/15 22:47:29 INFO util.GSet: VM type = 64-bit

14/06/15 22:47:29 INFO util.GSet: 2.0% max memory = 966.7 MB

14/06/15 22:47:29 INFO util.GSet: capacity = 2^21 = 2097152 entries

14/06/15 22:47:29 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false

14/06/15 22:47:29 INFO blockmanagement.BlockManager: defaultReplication = 3

14/06/15 22:47:29 INFO blockmanagement.BlockManager: maxReplication = 512

14/06/15 22:47:29 INFO blockmanagement.BlockManager: minReplication = 1

14/06/15 22:47:29 INFO blockmanagement.BlockManager: maxReplicationStreams = 2

14/06/15 22:47:29 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false

14/06/15 22:47:29 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000

14/06/15 22:47:29 INFO blockmanagement.BlockManager: encryptDataTransfer = false

14/06/15 22:47:29 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)

14/06/15 22:47:29 INFO namenode.FSNamesystem: supergroup = supergroup

14/06/15 22:47:29 INFO namenode.FSNamesystem: isPermissionEnabled = true

14/06/15 22:47:29 INFO namenode.FSNamesystem: Determined nameservice ID: nnode

14/06/15 22:47:29 INFO namenode.FSNamesystem: HA Enabled: false

14/06/15 22:47:29 INFO namenode.FSNamesystem: Append Enabled: true

14/06/15 22:47:29 INFO util.GSet: Computing capacity for map INodeMap

14/06/15 22:47:29 INFO util.GSet: VM type = 64-bit

14/06/15 22:47:29 INFO util.GSet: 1.0% max memory = 966.7 MB

14/06/15 22:47:29 INFO util.GSet: capacity = 2^20 = 1048576 entries

14/06/15 22:47:29 INFO namenode.NameNode: Caching file names occuring more than 10 times

14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0

14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000

14/06/15 22:47:29 INFO namenode.FSNamesystem: Retry cache on namenode is enabled

14/06/15 22:47:29 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis

14/06/15 22:47:29 INFO util.GSet: Computing capacity for map Namenode Retry Cache

14/06/15 22:47:29 INFO util.GSet: VM type = 64-bit

14/06/15 22:47:29 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB

14/06/15 22:47:29 INFO util.GSet: capacity = 2^15 = 32768 entries

14/06/15 22:47:29 INFO common.Storage: Storage directory /mycloud/hadoop-2.2.0/temp/dfs/name has been successfully formatted.

14/06/15 22:47:29 INFO namenode.FSImage: Saving image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

14/06/15 22:47:29 INFO namenode.FSImage: Image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 196 bytes saved in 0 seconds.

14/06/15 22:47:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

14/06/15 22:47:29 INFO util.ExitUtil: Exiting with status 0

14/06/15 22:47:30 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at cloud01/192.168.214.171

************************************************************/

[root@cloud01 hadoop-2.2.0]#

终于成功了

************************************************************/

14/06/17 23:40:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /mycloud/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

14/06/17 23:40:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Formatting using clusterid: CID-47fe8dae-be4f-4f25-a542-1b43a720752b

14/06/17 23:40:24 INFO namenode.HostFileManager: read includes:

HostSet(

)

14/06/17 23:40:24 INFO namenode.HostFileManager: read excludes:

HostSet(

)

14/06/17 23:40:24 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000

14/06/17 23:40:24 INFO util.GSet: Computing capacity for map BlocksMap

14/06/17 23:40:24 INFO util.GSet: VM type = 64-bit

14/06/17 23:40:24 INFO util.GSet: 2.0% max memory = 966.7 MB

14/06/17 23:40:24 INFO util.GSet: capacity = 2^21 = 2097152 entries

14/06/17 23:40:24 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false

14/06/17 23:40:24 INFO blockmanagement.BlockManager: defaultReplication = 3

14/06/17 23:40:24 INFO blockmanagement.BlockManager: maxReplication = 512

14/06/17 23:40:24 INFO blockmanagement.BlockManager: minReplication = 1

14/06/17 23:40:24 INFO blockmanagement.BlockManager: maxReplicationStreams = 2

14/06/17 23:40:24 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false

14/06/17 23:40:24 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000

14/06/17 23:40:24 INFO blockmanagement.BlockManager: encryptDataTransfer = false

14/06/17 23:40:24 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)

14/06/17 23:40:24 INFO namenode.FSNamesystem: supergroup = supergroup

14/06/17 23:40:24 INFO namenode.FSNamesystem: isPermissionEnabled = true

14/06/17 23:40:24 INFO namenode.FSNamesystem: Determined nameservice ID: ns

14/06/17 23:40:24 INFO namenode.FSNamesystem: HA Enabled: true

14/06/17 23:40:24 INFO namenode.FSNamesystem: Append Enabled: true

14/06/17 23:40:24 INFO util.GSet: Computing capacity for map INodeMap

14/06/17 23:40:24 INFO util.GSet: VM type = 64-bit

14/06/17 23:40:24 INFO util.GSet: 1.0% max memory = 966.7 MB

14/06/17 23:40:24 INFO util.GSet: capacity = 2^20 = 1048576 entries

14/06/17 23:40:24 INFO namenode.NameNode: Caching file names occuring more than 10 times

14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0

14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000

14/06/17 23:40:24 INFO namenode.FSNamesystem: Retry cache on namenode is enabled

14/06/17 23:40:24 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis

14/06/17 23:40:24 INFO util.GSet: Computing capacity for map Namenode Retry Cache

14/06/17 23:40:24 INFO util.GSet: VM type = 64-bit

14/06/17 23:40:24 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB

14/06/17 23:40:24 INFO util.GSet: capacity = 2^15 = 32768 entries

Re-format filesystem in Storage Directory /mycloud/hadoop-2.2.0/temp/dfs/name ? (Y or N) y

14/06/17 23:40:29 INFO common.Storage: Storage directory /mycloud/hadoop-2.2.0/temp/dfs/name has been successfully formatted.

14/06/17 23:40:29 INFO namenode.FSImage: Saving image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

14/06/17 23:40:29 INFO namenode.FSImage: Image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 196 bytes saved in 0 seconds.

14/06/17 23:40:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

14/06/17 23:40:29 INFO util.ExitUtil: Exiting with status 0

14/06/17 23:40:29 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at cloud01/192.168.214.171

************************************************************/

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签：

相关文章推荐

新的分享

章节导航