hadoop-2.7.4 安装部署: HA+Federation
2017-10-05 20:52
330 查看
1.准备工作
准备四台主机 hadoop1,hadoop2,hadoop3,hadoop4。其中hadoop1需要设置免密码登录到其他三台主机(格式化文件系统,开启yarn等等需要使用),namespace ID 相同的两个 nameNode 要实现相互免密码登录,是因为某一台发生故障时需要切换到另外一台(failover)。每一台主机都需要在 /etc/hosts 文件输入四台主机的ip及对应主机名。
下表为各个主机需要安装的组件:
2.配置文件
hadoop-env.sh 设置一下JAVA_HOME即可
core-site.xml
具体参见 [ HDFS High Availability Using the Quorum Journal Manager ]
hdfs-site.xml
mapred-site.xml
yarn-site.xml
fairscheduler.xml
slave
将配置文件分发到各个主机,配置文件在各个主机略有不同,需要更改,更改地方已在配置文件中指明
3.启动
新手一枚,如果错误,曲解的地方,还望指教
准备四台主机 hadoop1,hadoop2,hadoop3,hadoop4。其中hadoop1需要设置免密码登录到其他三台主机(格式化文件系统,开启yarn等等需要使用),namespace ID 相同的两个 nameNode 要实现相互免密码登录,是因为某一台发生故障时需要切换到另外一台(failover)。每一台主机都需要在 /etc/hosts 文件输入四台主机的ip及对应主机名。
下表为各个主机需要安装的组件:
主机 | namenode | datanode | journalnode | ResourceManager | cluster ID | namespace |
---|---|---|---|---|---|---|
hadoop1 | nn1 | 不存在 | 不存在 | 存在 | hadoop-cluster | hadoop-cluster1 |
hadoop2 | nn2 | 存在 | 存在 | 不存在 | hadoop-cluster | hadoop-cluster1 |
hadoop3 | nn3 | 存在 | 存在 | 不存在 | hadoop-cluster | hadoop-cluster2 |
hadoop4 | nn4 | 存在 | 存在 | 不存在 | hadoop-cluster | hadoop-cluster2 |
表1
2.配置文件
hadoop-env.sh 设置一下JAVA_HOME即可
core-site.xml
<property> <!-- 该值要和这个HA的nameservice ID一致 即:hdfs-site.xml中属性dfs.nameservices的值---> <name>fs.defaultFS</name> <!-- 示例 --> <value>hdfs://hadoop-cluster1</value> </property> <property> <!--配置zookeeper节点,多个用逗号隔开--> <name>ha.zookeeper.quorum</name> <value>zookeeperhome:2181</value> </property>
具体参见 [ HDFS High Availability Using the Quorum Journal Manager ]
hdfs-site.xml
<property> <name>dfs.nameservices</name> <!--有两个HA组成的集群,所以要讲--> <value>hadoop-cluster1,hadoop-cluster2</value> <description>Comma-separated list of nameservices</description> </property> <!--hadoop cluster1--> <!--配置集群的各个节点--> <property> <name>dfs.ha.namenodes.hadoop-cluster1</name> <value>nn1,nn2</value> <description>The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).</description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-cluster1.nn1</name> <value>hadoop1:8020</value> <description>RPC address for nomenode1 of hadoop-cluster1</description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-cluster1.nn2</name> <value>hadoop2:8020</value> <description>RPC address for nomenode2 of hadoop-test </description> </property> <property> <name>dfs.namenode.http-address.hadoop-cluster1.nn1</name> <value>hadoop1:50070</value> <description>The address and the base port where the dfs namenode1 web ui will listen on. </description> </property> <property> <name>dfs.namenode.http-address.hadoop-cluster1.nn2</name> <value>hadoop2:50070</value> <description>< 4000 /span>he address and the base port where the dfs namenode2 web ui will listen on. </description> </property> <!-- hadoop cluster2 --> <property> <name>dfs.ha.namenodes.hadoop-cluster2</name> <value>nn3,nn4</value> <description>The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). </description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-cluster2.nn3</name> <value>hadoop3:8020</value> <description> RPC address for nomenode1 of hadoop-cluster1 </description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-cluster2.nn4</name> <value>hadoop4:8020</value> <description> RPC address for nomenode2 of hadoop-test </description> </property> <property> <name>dfs.namenode.http-address.hadoop-cluster2.nn3</name> <value>hadoop3:50070</value> <description> The address and the base port where the dfs namenode1 web ui will listen on. </description> </property> <property> <name>dfs.namenode.http-address.hadoop-cluster2.nn4</name> <value>hadoop4:50070</value> <description> The address and the base port where the dfs namenode2 web ui will listen on. </description> </property> <property> <name>dfs.namenode.name.dir</name> <!--namenode存放数据目录,需要先创建--> <value>file:///home/test/hadoop/hdfs/name</value> <description>Determines where on the local filesystem the DFS name nodeshould store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <!--这里需注意每个HA要有唯一的名称空间,即url,最后一段一定要是唯一的,而每个HA的两个namenode的名称空间要是相同的。--> <value>qjournal://hadoop2:8485;hadoop3:8485;hadoop4:8485/hadoop-cluster1</value> <description>A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. </description> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///home/test/hadoop/hdfs/data</value> <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. </description> </property> <property> <!--开启failover--> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> <description> Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. </description> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/test/hadoop/hdfs/journal/</value> </property> <property> <name>dfs.client.failover.proxy.provider.hadoop-cluster1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.client.failover.proxy.provider.hadoop-cluster2</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!--下面两项为配置fencing method,在failover中使用,必需得到的代表success的返回值,用来防止客户端读到过期数据--> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <!--密钥 故障切换--> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/exampleuser/.ssh/id_rsa</value> </property>
mapred-site.xml
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>hadoop1:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hadoop1:19888</value> </property>
yarn-site.xml
<property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>hadoop1</value> </property> <property> <description>The address of the applications manager interface in the RM.</description> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property> <property> <description>The address of the scheduler interface.</description> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> </property> <property> <description>The http address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name& dd95 gt; <value>${yarn.resourcemanager.hostname}:8088</value> </property> <property> <description>The https adddress of the RM web application.</description> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname}:8090</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> </property> <property> <description>The address of the RM admin interface.</description> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> </property> <property> <description>The class to use as the resource scheduler.</description> <name>yarn.resourcemanager.scheduler.class</name> <!--需要配置 fairscheduler.xml--> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> <property> <description>fair-scheduler conf location</description> <name>yarn.scheduler.fair.allocation.file</name> <value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value> </property> <property> <description>List of directories to store localized files in. An application's localized file directory will be found in:${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. </description> <name>yarn.nodemanager.local-dirs</name> <value>/home/chinahadoop/hadoop/yarn/local</value> </property> <property> <description>Whether to enable log aggregation</description> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/tmp/logs</value> </property> <property> <description>Amount of physical memory, in MB, that can be allocated for containers.</description> <name>yarn.nodemanager.resource.memory-mb</name> <value>30720</value> </property> <property> <description>Number of CPU cores that can be allocated for containers.</description> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>12</value> </property> <property> <description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
fairscheduler.xml
<queue name="infrastructure"> <minResources>102400 mb, 50 vcores </minResources> <maxResources>153600 mb, 100 vcores </maxResources> <maxRunningApps>200</maxRunningApps> <minSharePreemptionTimeout>300</minSharePreemptionTimeout> <weight>1.0</weight> <aclSubmitApps>root,yarn,search,hdfs</aclSubmitApps> </queue> <queue name="tool"> <minResources>102400 mb, 30 vcores</minResources> <maxResources>153600 mb, 50 vcores</maxResources> </queue> <queue name="sentiment"> <minResources>102400 mb, 30 vcores</minResources> <maxResources>153600 mb, 50 vcores</maxResources> </queue>
slave
hadoop2 hadoop3 hadoop4
将配置文件分发到各个主机,配置文件在各个主机略有不同,需要更改,更改地方已在配置文件中指明
3.启动
注意:所有操作均在Hadoop部署目录下进行。 首先在每个HA中任选一个namenode,初始化HA在zookeeper的state bin/hdfs zkfc -formatZK 启动Hadoop集群: ------------------------------------------------------------------- (1) 启动nn1与nn2 Step1 : 在各个JournalNode节点上,输入以下命令启动journalnode服务: sbin/hadoop-daemon.sh start journalnode Step2: 在[nn1]上,对其进行格式化,并启动: 注意:两个HA的clusterId要一致,这样的的话, Federation才会认为是一个集群 bin/hdfs namenode -format -clusterId hadoop-cluster sbin/hadoop-daemon.sh start namenode Step3: 在[nn2]上,同步nn1的元数据信息: bin/hdfs namenode -bootstrapStandby Step4: 启动[nn2]: sbin/hadoop-daemon.sh start namenode 经过以上四步操作,nn1和nn2均处理standby状态 可选操作,如果没有配置zookeeper,可进行step5。如果已配置,zookeeper会自己选出activeNamenode Step5: 将[nn1]切换为Active bin/hdfs haadmin -ns hadoop-cluster1 -transitionToActive nn1 ------------------------------------------------------------------- (2) 启动nn3与nn4 Step1: 在[nn3]上,对其进行格式化,并启动: bin/hdfs namenode -format -clusterId hadoop-cluster sbin/hadoop-daemon.sh start namenode Step2: 在[nn4]上,同步nn3的元数据信息: bin/hdfs namenode -bootstrapStandby Step3: 启动[nn4]: sbin/hadoop-daemon.sh start namenode 经过以上三步操作,nn3和nn4均处理standby状态 可选操作,与前面一致 Step4: 将[nn3]切换为Active bin/hdfs haadmin -ns hadoop-cluster2 -transitionToActive nn3 ------------------------------------------------------------------- (3)启动所有datanode Step6: 在[nn1]上,启动所有datanode sbin/hadoop-daemons.sh start datanode ------------------------------------------------------------------- (4)启动所有namenode failvoer sbin/hadoop-daemon.sh start zkfc ------------------------------------------------------------------- (5)在[nn1]上启动yarn sbin/start-yarn.sh ------------------------------------------------------------------- (6)关闭Hadoop集群: 在[nn1]上,输入以下命令 sbin/stop-yarn.sh sbin/stop-dfs.sh
新手一枚,如果错误,曲解的地方,还望指教
相关文章推荐
- 部署hadoop2.7.2 集群 基于zookeeper配置HDFS HA+Federation
- Hadoop2.x HA(High Avalability,高可用性) 安装部署
- hadoop HA架构安装部署(QJM HA)
- hadoop ha集群安装部署
- hadoop2.7.1 HA安装部署(转)
- 【Hadoop2.7.0、Zookeeper3.4.6、JDK1.7】搭建完全分布式的hadoop,HA部署安装,自动备援
- Hadoop2.x分布式安装以及HA部署
- hadoop 集群安装(二) HA部署
- hadoop 2.x HA(QJM)安装部署规划
- hadoop2.X的ha完全分布式安装和部署
- hbase安装及部署(hadoop2.7.4集群)
- hadoop部署(单机、HA、HA+federation)
- hadoop-2.4.1 HA 分布式集群安装部署
- Hadoop+hive 集群安装部署 (一)
- Spark/Shark/Hadoop1安装部署文档总结(Hadoop2同理,只需要换成Spark的Hadoop2版本即可)
- Hadoop系列之一:hadoop部署安装
- linux上部署hadoop集群 HA-QJM篇
- Hadoop HA 与 Federation
- Hadoop实战-初级部分 之 Hadoop安装部署
- Ubuntu16.04安装配置hadoop-2.7.4