hadoop2.7.1安装部署错误，求大神指导
2015-12-02 20:04
429 查看
环境，三台虚拟机,centos6.5，jdk 1.7.0_71,hadoop版本hadoop2.7.1
三台机分别配置/etc/sysconfig/network 为
NETWORKING=yes
HOSTNAME=master
NETWORKING=yes
HOSTNAME=master2
NETWORKING=yes
HOSTNAME=master3
配置/etc/hosts
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master2
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master3
配置core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadooptmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
配置hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
配置mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
配置yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
hadoop-env.sh、yarn-env.sh，mapred-env.sh 的JAVA_HOME 配置为
export JAVA_HOME=/usr/java/jdk1.7.0_71
slaves配置为
master2
master3
使用命令sbin/start-dfs.sh;sbin/start-yarn.sh 启动
主节点jps查看不到任何信息
但是搜索进程 (ps -ef|grep java)可以看到
hadoop 1214 1 0 19:42 ? 00:00:12 /usr/java/jdk1.7.0_71/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1
-Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop-hadoop-namenode-master.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
hadoop 1394 1 0 19:42 ? 00:00:08 /usr/java/jdk1.7.0_71/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1
-Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop-hadoop-secondarynamenode-master.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
hadoop 1956 1033 0 20:12 pts/0 00:00:00 grep java
在master3上面jps命令 打印
1067 DataNode
1297 Jps
在master2上面jps命令 打印
1153 NodeManager
1065 DataNode
1295 Jps
查看主节点日志 其他文件均正常，除了yarn-hadoop-resourcemanager-master.log 有错误信息
vi yarn-hadoop-resourcemanager-master.log
2015-12-02 19:42:38,272 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2015-12-02 19:42:38,272 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2015-12-02 19:42:38,272 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
Caused by: java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:879)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
... 4 more
Caused by: java.io.FileNotFoundException: /tmp/Jetty_master_8088_cluster____i4ls4w/webapp/webapps/cluster/.keep (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:857)
... 5 more
2015-12-02 19:42:38,277 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at master/127.0.0.1
************************************************************/
求各位指导！！！
三台机分别配置/etc/sysconfig/network 为
NETWORKING=yes
HOSTNAME=master
NETWORKING=yes
HOSTNAME=master2
NETWORKING=yes
HOSTNAME=master3
配置/etc/hosts
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master2
192.168.244.102 master
192.168.244.104 master3
192.168.244.105 master2
127.0.0.1 localhost.localdomain master3
配置core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadooptmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
配置hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
配置mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
配置yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
hadoop-env.sh、yarn-env.sh，mapred-env.sh 的JAVA_HOME 配置为
export JAVA_HOME=/usr/java/jdk1.7.0_71
slaves配置为
master2
master3
使用命令sbin/start-dfs.sh;sbin/start-yarn.sh 启动
主节点jps查看不到任何信息
但是搜索进程 (ps -ef|grep java)可以看到
hadoop 1214 1 0 19:42 ? 00:00:12 /usr/java/jdk1.7.0_71/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1
-Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop-hadoop-namenode-master.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
hadoop 1394 1 0 19:42 ? 00:00:08 /usr/java/jdk1.7.0_71/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1
-Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/hadoop/hadoop/hadoop-2.7.1/logs -Dhadoop.log.file=hadoop-hadoop-secondarynamenode-master.log -Dhadoop.home.dir=/home/hadoop/hadoop/hadoop-2.7.1 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop/hadoop-2.7.1/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
hadoop 1956 1033 0 20:12 pts/0 00:00:00 grep java
在master3上面jps命令 打印
1067 DataNode
1297 Jps
在master2上面jps命令 打印
1153 NodeManager
1065 DataNode
1295 Jps
查看主节点日志 其他文件均正常，除了yarn-hadoop-resourcemanager-master.log 有错误信息
vi yarn-hadoop-resourcemanager-master.log
2015-12-02 19:42:38,272 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2015-12-02 19:42:38,272 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2015-12-02 19:42:38,272 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
Caused by: java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:879)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
... 4 more
Caused by: java.io.FileNotFoundException: /tmp/Jetty_master_8088_cluster____i4ls4w/webapp/webapps/cluster/.keep (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:857)
... 5 more
2015-12-02 19:42:38,277 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at master/127.0.0.1
************************************************************/
求各位指导！！！
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- windows复制文件到Linux或Windows，有关FTP的配置与使用
- bash 命令行编辑
- opencar模板 opencartchina
- opencar模板 opencartchina
- 使用Docker快速部署Storm环境
- 由一个线程例子引发的思考（转载）
- Centos7.0 修改防火墙为iptables
- OpenJudge_P1759 最长上升子序列(DP)
- linux学习（三）如何在linux系统下利用vi编辑C/C++程序
- spatialhadoop2.3源码阅读(四) FileMBR类
- 1044. Shopping in Mars (25) 尺取法
- Tomcat服务器安装与环境配置
- OpenJudge_P1768 最大子矩阵
- kaldi can link with openfst-1.5.0 without patch
- Linux学习笔记（6）之vi编辑器基本操作
- CodeForces 551D GukiZ and Binary Operations DP+矩阵乘法
- Linux下Rsync+sersync实现数据双向实时同步
- linux下shell命令之hwclock
- docker 中 NGINX+PHP+MYSQL+REDIS+Elasticsearch 环境搭建 (windows系统)
- 如何建立linux下的 LNMP 环境 + VSFTPD