Hadoop2.6.5安装部署及环境变量配置
2017-11-09 18:00
405 查看
系统环境:
四台虚拟机
192.168.1.167 vm4.com192.168.1.31 vm3.com
192.168.1.62 vm2.com
192.168.1.39 vm1.com
系统版本
[root@vm1 ~]# cat /etc/centos-release CentOS Linux release 7.0.1406 (Core)
部署前准备
vm1 可以免密登陆其他三台机器四台机器互相解析
[root@vm1 ~]# cat /etc/hosts 192.168.1.167 vm4.com 192.168.1.31 vm3.com 192.168.1.62 vm2.com 192.168.1.39 vm1.com
防火墙关闭状态
systemctl stop firewalld.service systemctl disable firewalld.service
新建hadoop用户
useradd -u 5000 hadoop 设置密码 echo "hadoop"|passwd --stdin hadoop
JDK环境部署(四台机器都需要):
在官网下载 jdk-8u151-linux-x64.tar.gztar zxvf /root/jdk-8u151-linux-x64.tar.gz mv /root/jdk1.8.0_151 /usr/local/jdk1.8
配置环境变量
vim /etc/profile.d/java.sh #### JAVA_HOME=/usr/local/jdk1.8 JAVA_BIN=/usr/local/jdk1.8/bin JRE_HOME=/usr/local/jdk1.8/jre PATH=$PATH:/usr/local/jdk1.8/bin:/usr/local/jdk1.8/jre/bin CLASSPATH=/usr/local/jdk1.8/jre/lib:/usr/local/jdk1.8/lib:/usr/local/jdk1.8/jre/lib/charsets.jar #### #source生效 source /etc/profile.d/java.sh
测试
[root@vm1 /]# java -version java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot( 4000 TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
JDK环境部署ok
Hadoop环境部署:
1.下载安装hadoop
hadoop官网:http://hadoop.apache.orgwget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz tar zxvf hadoop-2.6.5.tar.gz cd hadoop-2.6.5 vim etc/hadoop/hadoop-env.sh 修改这一行: export JAVA_HOME=/usr/local/jdk1.8 然后测试 [root@vm2 hadoop]# ./bin/hadoop version Hadoop 2.6.5 Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997 Compiled by sjlee on 2016-10-02T23:43Z Compiled with protoc 2.5.0 From source with checksum f05c9fa095a395faa9db9f7ba5d754 This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar
2.集群分布式环境配置
hadoop环境变量配置vim ~/.bashrc
添加一行
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin
source 生效
source ~/.bashrc
mv hadoop-2.6.5 /usr/local/hadoop chown -R hadoop.hadoop /usr/local/hadoop-2.6.5/
编辑配置文件
配置文件目录
[root@vm2 hadoop]# pwd /usr/local/hadoop/etc/hadoop
1.编辑slaves文件
将作为 DataNode 的主机名写入该文件,每行一个[root@vm2 hadoop]# cat slaves vm2.com vm3.com vm4.com
2.编辑core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://vm1.com:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> </configuration>
3.编辑hdfs-site.xml
dfs.replication 为 Slave 节点个数<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>vm1.com:50090</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/data</value> </property> </configuration>
4.编辑mapred-site.xml
cp mapred-site.xml.template mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>vm1.com:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>vm1.com:19888</value> </property> </configuration>
5.编辑yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>vm1.com</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
至此 hadoop启动所必需的配置已全部配置好
将整个hadoop目录打包用scp命令发送至其他节点
启动hadoop
首次启动需要先在 Master 节点执行 NameNode 的格式化:hdfs namenode -format
成功的话,会看到 “successfully formatted” 和 “Exitting with status 0” 的提示,若为 “Exitting with status 1” 则是出错。
接着开启 NameNode 和 DataNode 守护进程。
以下三个脚本的存放路径在
[root@vm1 sbin]# pwd /usr/local/hadoop/sbin
前面已经添加到了环境变量 可以直接执行
start-dfs.sh start-yarn.sh mr-jobhistory-daemon.sh start historyserver
通过命令 jps 可以查看各个节点所启动的进程。正确的话,在 vm1.com节点上可以看到 NameNode、ResourceManager、SecondrryNameNode、JobHistoryServer 进程如下:
[root@vm1 ~]# jps 4050 NameNode 4229 SecondaryNameNode 4663 JobHistoryServer 4378 ResourceManager 7498 Jps
在其他三个datanode节点可以看到 DataNode 和 NodeManager 进程
[root@vm2 sbin]# jps 12301 NodeManager 13358 Jps 12207 DataNode
以上 缺少任一进程都表示出错
另外还需要在 Master 节点上通过命令 hdfs dfsadmin -report 查看 DataNode 是否正常启动
[root@vm1 ~]# hdfs dfsadmin -report Configured Capacity: 160982630400 (149.93 GB) Present Capacity: 154752630784 (144.12 GB) DFS Remaining: 154750676992 (144.12 GB) DFS Used: 1953792 (1.86 MB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Live datanodes (3): Name: 192.168.1.167:50010 (vm4.com) Hostname: vm4.com Decommission Status : Normal Configured Capacity: 53660876800 (49.98 GB) DFS Used: 651264 (636 KB) Non DFS Used: 2076315648 (1.93 GB) DFS Remaining: 51583909888 (48.04 GB) DFS Used%: 0.00% DFS Remaining%: 96.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Nov 09 17:40:28 CST 2017 Name: 192.168.1.31:50010 (vm3.com) Hostname: vm3.com Decommission Status : Normal Configured Capacity: 53660876800 (49.98 GB) DFS Used: 651264 (636 KB) Non DFS Used: 2076905472 (1.93 GB) DFS Remaining: 51583320064 (48.04 GB) DFS Used%: 0.00% DFS Remaining%: 96.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Nov 09 17:40:28 CST 2017 Name: 192.168.1.62:50010 (vm2.com) Hostname: vm2.com Decommission Status : Normal Configured Capacity: 53660876800 (49.98 GB) DFS Used: 651264 (636 KB) Non DFS Used: 2076778496 (1.93 GB) DFS Remaining: 51583447040 (48.04 GB) DFS Used%: 0.00% DFS Remaining%: 96.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Nov 09 17:40:28 CST 2017
相关文章推荐
- JBOSS的下载安装、环境变量配置以及部署
- hadoop 安装环境变量配置检测
- 一路Hadoop-- 6、在Linux上安装JDK-2,配置jdk的环境变量
- hadoop分布式安装部署详细视频教程(网盘附配好环境的CentOS虚拟机文件/hadoop配置文件)
- JBOSS的下载安装、环境变量配置以及部署
- jboss安装、环境变量配置、部署
- hadoop分布式安装部署详细视频教程(网盘附配好环境的CentOS虚拟机文件/hadoop配置文件)
- JBOSS的下载安装、环境变量配置以及部署
- ubuntu 下部署 javaWeb 项目配置 jdk环境变量和安装tomcat7
- centos 7 安装+hadoop2.6.5环境部署(伪分布式)
- JBOSS的下载安装、环境变量配置以及部署
- JBOSS EAP6.2.0的下载安装、环境变量配置以及部署
- jdk下载安装与环境变量配置教程
- yum安装jdk环境变量配置
- Ubuntu配置hadoop单机+伪分布式环境+eclipse--安装hadoop(一)
- Ubuntu 9.10下安装JDK1.6.0_18 并配置环境变量
- Maven安装及环境变量配置
- Linux上安装JDK环境变量配置
- hadoop环境安装配置介绍与步骤
- Gradle的安装和环境变量的配置