Spark On Yarn（HDFS HA）详细配置
2014-02-22 23:34
731 查看
######################################################################################################
一、服务器分布及相关说明
######################################################################################################
1、服务器角色
2、Hadoop（HDFS HA）总体架构
######################################################################################################
二、基础环境部署
######################################################################################################
1、JDK安装
http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.tar.gz
# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local
# cd /usr/local
# ln -s jdk1.7.0_45 jdk
# vim /etc/profile
export JAVA_HOME=/usr/local/jdk
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin
# source /etc/profile
=========================================================================================
2、Scala安装
http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
# tar xvzf scala-2.10.3.tgz -C/usr/local
# cd /usr/local
# ln -s scala-2.10.3 scala
# vim /etc/profile
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
# source /etc/profile
=========================================================================================
3、SSH免密码登录
可参考文章：
http://blog.csdn.net/codepeak/article/details/14447627
......
=========================================================================================
4、主机名设置
# vim /etc/hosts
172.18.35.29 namenode1
172.18.35.30 namenode2
172.18.34.232 datanode1
172.18.24.136 datanode2
######################################################################################################
三、ZooKeeper集群部署
######################################################################################################
1、ZooKeeper安装
http://apache.dataguru.cn/zookeeper/stable/zookeeper-3.4.5.tar.gz
# tar xvzf zookeeper-3.4.5.tar.gz -C/usr/local
# cd /usr/local
# ln -s zookeeper-3.4.5 zookeeper
# vim /etc/profile
export ZOO_HOME=/usr/local/zookeeper
export ZOO_LOG_DIR=/data/hadoop/zookeeper/logs
export PATH=$PATH:$ZOO_HOME/bin
# source /etc/profile
=========================================================================================
2、ZooKeeper配置与启动
# mkdir -p/data/hadoop/zookeeper/{data,logs}
# vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/hadoop/zookeeper/data
clientPort=2181
server.1=172.18.35.29:2888:3888
server.2=172.18.35.30:2888:3888
server.3=172.18.34.232:2888:3888
在172.18.35.29上执行：
echo 1 > /data/hadoop/zookeeper/data/myid
在172.18.35.30 上执行：
echo 2 > /data/hadoop/zookeeper/data/myid
在172.18.34.232 上执行：
echo 3 > /data/hadoop/zookeeper/data/myid
## 启动ZooKeeper集群
# cd /usr/local/zookeeper && bin/zkServer.sh start
# ./bin/zkCli.sh -server localhost:2181
测试zookeeper集群是否建立成功，如无报错表示集群创建成功
# bin/zkServer.sh status
######################################################################################################
四、Hadoop（HDFS HA）集群部署
######################################################################################################
1、hadoop环境安装
Hadoop的源码编译部分可以参考：
/article/4424430.html
# tar xvzf hadoop-2.2.0.tgz -C/usr/local
# cd /usr/local
# ln -s hadoop-2.2.0 hadoop
# vim /etc/profile
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_PID_DIR=/data/hadoop/pids
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# mkdir -p /data/hadoop/{pids,storage}
# mkdir -p/data/hadoop/storage/{hdfs,tmp,journal}
# mkdir -p/data/hadoop/storage/tmp/nodemanager/{local,remote,logs}
# mkdir -p/data/hadoop/storage/hdfs/{name,data}
=========================================================================================
2、core.site.xml配置
# vim/usr/local/hadoop/etc/hadoop/core-site.xml
=========================================================================================
3、hdfs-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/hdfs-site.xml
=========================================================================================
4、mapred-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/mapred-site.xml
=========================================================================================
5、yarn-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/yarn-site.xml
【注意：上面的第68`96行部分，需要根据服务器的硬件配置进行修改】
=========================================================================================
6、配置hadoop-env.sh、mapred-env.sh、yarn-env.sh【在开头添加】
文件路径：
/usr/local/hadoop/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/etc/hadoop/mapred-env.sh
/usr/local/hadoop/etc/hadoop/yarn-env.sh
添加内容：
export JAVA_HOME=/usr/local/jdk
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_PID_DIR=/data/hadoop/pids
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
=========================================================================================
7、数据节点配置
# vim /usr/local/hadoop/etc/hadoop/slaves
datanode1
datanode2
=========================================================================================
8、集群启动
(1)、在namenode1上执行，创建命名空间
# hdfs zkfc -formatZK
----------------------------------------------------------------------------------------------------------------------------------------------
(2)、在对应的节点上启动日志程序journalnode
# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode
----------------------------------------------------------------------------------------------------------------------------------------------
(3)、格式化主NameNode节点（namenode1）
# hdfs namenode -format
----------------------------------------------------------------------------------------------------------------------------------------------
(4)、启动主NameNode节点
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
----------------------------------------------------------------------------------------------------------------------------------------------
(5)、格式备NameNode节点（namenode2）
# hdfs namenode -bootstrapStandby
----------------------------------------------------------------------------------------------------------------------------------------------
(6)、启动备NameNode节点（namenode2）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
----------------------------------------------------------------------------------------------------------------------------------------------
(7)、在两个NameNode节点（namenode1、namenode2）上执行
# cd /usr/local/hadoop && sbin/hadoop-daemon.shstart zkfc
----------------------------------------------------------------------------------------------------------------------------------------------
(8)、启动所有的DataNode节点（datanode1、datanode2）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode
----------------------------------------------------------------------------------------------------------------------------------------------
(9)、启动Yarn（namenode1）
# cd /usr/local/hadoop && sbin/start-yarn.sh
主NameNode节点上的信息：
DataNode节点上的信息：
----------------------------------------------------------------------------------------------------------------------------------------------
(10)、测试Yarn是否可用
# hdfs dfs -put/usr/local/hadoop/etc/hadoop/yarn-site.xml /tmp
# hadoop jar/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount /tmp/yarn-site.xml /mytest
-----------------------------------------------------------------------------------------------------------------------------------------------
(11)、HDFS的HA功能测试
切换前的状态：
# kill -9 11466
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
切换后的状态：
-----------------------------------------------------------------------------------------------------------------------------------------------
(12)、后续维护
HDFS的关闭与启动：
# cd /usr/local/hadoop && sbin/stop-dfs.sh
# cd /usr/local/hadoop && sbin/start-dfs.sh
YARN的关闭与启动：
# cd /usr/local/hadoop && sbin/stop-yarn.sh
# cd /usr/local/hadoop && sbin/start-yarn.sh
【注意】
需要在NameNode节点上执行。
######################################################################################################
五、Spark集群部署
######################################################################################################
1、Spark安装与配置
Spark的源码编译请参考：
/article/4424431.html
# tar xvzf spark-0.9.0-incubating.tgz -C/usr/local
# cd /usr/local
# ln -s spark-0.9.0-incubating spark
# vim /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
# source /etc/profile
# cd /usr/local/spark/conf
# mkdir -p /data/spark/tmp
----------------------------------------------------------------------------------------------------------------------------------------------
# vim spark-env.sh
export JAVA_HOME=/usr/local/jdk
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/usr/local/hadoop
SPARK_LOCAL_DIR="/data/spark/tmp"
SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTi
meStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC-XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFracti
on=60"
----------------------------------------------------------------------------------------------------------------------------------------------
# vim slaves
datanode1
datanode2
# cd /usr/local/spark && sbin/start-all.sh
=========================================================================================
2、相关测试
(1)、本地模式
# bin/run-exampleorg.apache.spark.examples.SparkPi local
----------------------------------------------------------------------------------------------------------------------------------------------
(2)、普通集群模式
# bin/run-exampleorg.apache.spark.examples.SparkPi spark://namenode1:7077
# bin/run-exampleorg.apache.spark.examples.SparkLR spark://namenode1:7077
# bin/run-exampleorg.apache.spark.examples.SparkKMeans spark://namenode1:7077file:/usr/local/spark/data/kmeans_data.txt 2 1
----------------------------------------------------------------------------------------------------------------------------------------------
(3)、结合HDFS的集群模式
# hadoop fs -put README.md .
# MASTER=spark://namenode1:7077bin/spark-shell
scala> val file =sc.textFile("hdfs://namenode1:9000/user/root/README.md")
scala> val count = file.flatMap(line=> line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)
scala> count.collect()
scala> :quit
----------------------------------------------------------------------------------------------------------------------------------------------
(4)、基于YARN模式
#SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar\
bin/spark-classorg.apache.spark.deploy.yarn.Client \
--jarexamples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar \
--classorg.apache.spark.examples.SparkPi \
--args yarn-standalone \
--num-workers 3 \
--master-memory 4g \
--worker-memory 2g \
--worker-cores 1
----------------------------------------------------------------------------------------------------------------------------------------------
(5)、最终的目录结构及相关配置
目录结构：
配置文件“/etc/profile”中的环境变量设置：
本文出自 “人生理想在于坚持不懈” 博客，请务必保留此出处http://sofar.blog.51cto.com/353572/1362240
一、服务器分布及相关说明
######################################################################################################
1、服务器角色
| 服务角色/服务器 | 172.18.35.29（namenode1） | 172.18.35.30（namenode2） | 172.18.34.232（datanode1） | 172.18.24.136（datanode2） |
| NameNode | YES | YES | NO | NO |
| DataNode | NO | NO | YES | YES |
| JournalNode | YES | YES | YES | NO |
| ZooKeeper | YES | YES | YES | NO |
| ZKFC | YES | YES | NO | NO |
######################################################################################################
二、基础环境部署
######################################################################################################
1、JDK安装
http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.tar.gz
# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local
# cd /usr/local
# ln -s jdk1.7.0_45 jdk
# vim /etc/profile
export JAVA_HOME=/usr/local/jdk
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin
# source /etc/profile
=========================================================================================
2、Scala安装
http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
# tar xvzf scala-2.10.3.tgz -C/usr/local
# cd /usr/local
# ln -s scala-2.10.3 scala
# vim /etc/profile
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
# source /etc/profile
=========================================================================================
3、SSH免密码登录
可参考文章：
http://blog.csdn.net/codepeak/article/details/14447627
......
=========================================================================================
4、主机名设置
# vim /etc/hosts
172.18.35.29 namenode1
172.18.35.30 namenode2
172.18.34.232 datanode1
172.18.24.136 datanode2
######################################################################################################
三、ZooKeeper集群部署
######################################################################################################
1、ZooKeeper安装
http://apache.dataguru.cn/zookeeper/stable/zookeeper-3.4.5.tar.gz
# tar xvzf zookeeper-3.4.5.tar.gz -C/usr/local
# cd /usr/local
# ln -s zookeeper-3.4.5 zookeeper
# vim /etc/profile
export ZOO_HOME=/usr/local/zookeeper
export ZOO_LOG_DIR=/data/hadoop/zookeeper/logs
export PATH=$PATH:$ZOO_HOME/bin
# source /etc/profile
=========================================================================================
2、ZooKeeper配置与启动
# mkdir -p/data/hadoop/zookeeper/{data,logs}
# vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/hadoop/zookeeper/data
clientPort=2181
server.1=172.18.35.29:2888:3888
server.2=172.18.35.30:2888:3888
server.3=172.18.34.232:2888:3888
在172.18.35.29上执行：
echo 1 > /data/hadoop/zookeeper/data/myid
在172.18.35.30 上执行：
echo 2 > /data/hadoop/zookeeper/data/myid
在172.18.34.232 上执行：
echo 3 > /data/hadoop/zookeeper/data/myid
## 启动ZooKeeper集群
# cd /usr/local/zookeeper && bin/zkServer.sh start
# ./bin/zkCli.sh -server localhost:2181
测试zookeeper集群是否建立成功，如无报错表示集群创建成功
# bin/zkServer.sh status
######################################################################################################
四、Hadoop（HDFS HA）集群部署
######################################################################################################
1、hadoop环境安装
Hadoop的源码编译部分可以参考：
/article/4424430.html
# tar xvzf hadoop-2.2.0.tgz -C/usr/local
# cd /usr/local
# ln -s hadoop-2.2.0 hadoop
# vim /etc/profile
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_PID_DIR=/data/hadoop/pids
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# mkdir -p /data/hadoop/{pids,storage}
# mkdir -p/data/hadoop/storage/{hdfs,tmp,journal}
# mkdir -p/data/hadoop/storage/tmp/nodemanager/{local,remote,logs}
# mkdir -p/data/hadoop/storage/hdfs/{name,data}
=========================================================================================
2、core.site.xml配置
# vim/usr/local/hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://appcluster</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/data/hadoop/storage/tmp</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>172.18.35.29:2181,172.18.35.30:2181,172.18.34.232:2181</value> </property> <property> <name>ha.zookeeper.session-timeout.ms</name> <value>2000</value> </property> <property> <name>fs.trash.interval</name> <value>4320</value> </property> <property> <name>hadoop.http.staticuser.use</name> <value>root</value> </property> <property> <name>hadoop.proxyuser.hadoop.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hadoop.groups</name> <value>*</value> </property> <property> <name>hadoop.native.lib</name> <value>true</value> </property> </configuration>
=========================================================================================
3、hdfs-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:/data/hadoop/storage/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/data/hadoop/storage/hdfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.blocksize</name> <value>67108864</value> </property> <property> <name>dfs.datanode.du.reserved</name> <value>53687091200</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.permissions.enabled</name> <value>false</value> </property> <property> <name>dfs.nameservices</name> <value>appcluster</value> </property> <property> <name>dfs.ha.namenodes.appcluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.appcluster.nn1</name> <value>namenode1:8020</value> </property> <property> <name>dfs.namenode.rpc-address.appcluster.nn2</name> <value>namenode2:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.appcluster.nn1</name> <value>namenode1:53310</value> </property> <property> <name>dfs.namenode.servicerpc-address.appcluster.nn2</name> <value>namenode2:53310</value> </property> <property> <name>dfs.namenode.http-address.appcluster.nn1</name> <value>namenode1:8080</value> </property> <property> <name>dfs.namenode.http-address.appcluster.nn2</name> <value>namenode2:8080</value> </property> <property> <name>dfs.datanode.http.address</name> <value>0.0.0.0:8080</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://namenode1:8485;namenode2:8485;datanode1:8485/appcluster</value> </property> <property> <name>dfs.client.failover.proxy.provider.appcluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence(root:36000)</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_dsa_nn1</value> </property> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/data/hadoop/storage/hdfs/journal</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>ha.failover-controller.cli-check.rpc-timeout.ms</name> <value>60000</value> </property> <property> <name>ipc.client.connect.timeout</name> <value>60000</value> </property> <property> <name>dfs.image.transfer.bandwidthPerSec</name> <value>41943040</value> </property> <property> <name>dfs.namenode.accesstime.precision</name> <value>3600000</value> </property> <property> <name>dfs.datanode.max.transfer.threads</name> <value>4096</value> </property> </configuration>
=========================================================================================
4、mapred-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>namenode1:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>namenode1:19888</value> </property> </configuration>
=========================================================================================
5、yarn-site.xml配置
# vim/usr/local/hadoop/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>namenode1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>namenode1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>namenode1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>namenode1:8033</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>namenode1:8034</value>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>namenode1:80</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>namenode1:80</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/local</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>${hadoop.tmp.dir}/nodemanager/remote</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>16</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>50320</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>40960</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>8</value>
</property>
</configuration>【注意：上面的第68`96行部分，需要根据服务器的硬件配置进行修改】
=========================================================================================
6、配置hadoop-env.sh、mapred-env.sh、yarn-env.sh【在开头添加】
文件路径：
/usr/local/hadoop/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/etc/hadoop/mapred-env.sh
/usr/local/hadoop/etc/hadoop/yarn-env.sh
添加内容：
export JAVA_HOME=/usr/local/jdk
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_PID_DIR=/data/hadoop/pids
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
=========================================================================================
7、数据节点配置
# vim /usr/local/hadoop/etc/hadoop/slaves
datanode1
datanode2
=========================================================================================
8、集群启动
(1)、在namenode1上执行，创建命名空间
# hdfs zkfc -formatZK
----------------------------------------------------------------------------------------------------------------------------------------------
(2)、在对应的节点上启动日志程序journalnode
# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode
----------------------------------------------------------------------------------------------------------------------------------------------
(3)、格式化主NameNode节点（namenode1）
# hdfs namenode -format
----------------------------------------------------------------------------------------------------------------------------------------------
(4)、启动主NameNode节点
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
----------------------------------------------------------------------------------------------------------------------------------------------
(5)、格式备NameNode节点（namenode2）
# hdfs namenode -bootstrapStandby
----------------------------------------------------------------------------------------------------------------------------------------------
(6)、启动备NameNode节点（namenode2）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
----------------------------------------------------------------------------------------------------------------------------------------------
(7)、在两个NameNode节点（namenode1、namenode2）上执行
# cd /usr/local/hadoop && sbin/hadoop-daemon.shstart zkfc
----------------------------------------------------------------------------------------------------------------------------------------------
(8)、启动所有的DataNode节点（datanode1、datanode2）
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode
----------------------------------------------------------------------------------------------------------------------------------------------
(9)、启动Yarn（namenode1）
# cd /usr/local/hadoop && sbin/start-yarn.sh
主NameNode节点上的信息：
DataNode节点上的信息：
----------------------------------------------------------------------------------------------------------------------------------------------
(10)、测试Yarn是否可用
# hdfs dfs -put/usr/local/hadoop/etc/hadoop/yarn-site.xml /tmp
# hadoop jar/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount /tmp/yarn-site.xml /mytest
-----------------------------------------------------------------------------------------------------------------------------------------------
(11)、HDFS的HA功能测试
切换前的状态：
# kill -9 11466
# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode
切换后的状态：
-----------------------------------------------------------------------------------------------------------------------------------------------
(12)、后续维护
HDFS的关闭与启动：
# cd /usr/local/hadoop && sbin/stop-dfs.sh
# cd /usr/local/hadoop && sbin/start-dfs.sh
YARN的关闭与启动：
# cd /usr/local/hadoop && sbin/stop-yarn.sh
# cd /usr/local/hadoop && sbin/start-yarn.sh
【注意】
需要在NameNode节点上执行。
######################################################################################################
五、Spark集群部署
######################################################################################################
1、Spark安装与配置
Spark的源码编译请参考：
/article/4424431.html
# tar xvzf spark-0.9.0-incubating.tgz -C/usr/local
# cd /usr/local
# ln -s spark-0.9.0-incubating spark
# vim /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
# source /etc/profile
# cd /usr/local/spark/conf
# mkdir -p /data/spark/tmp
----------------------------------------------------------------------------------------------------------------------------------------------
# vim spark-env.sh
export JAVA_HOME=/usr/local/jdk
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/usr/local/hadoop
SPARK_LOCAL_DIR="/data/spark/tmp"
SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTi
meStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC-XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFracti
on=60"
----------------------------------------------------------------------------------------------------------------------------------------------
# vim slaves
datanode1
datanode2
# cd /usr/local/spark && sbin/start-all.sh
=========================================================================================
2、相关测试
(1)、本地模式
# bin/run-exampleorg.apache.spark.examples.SparkPi local
----------------------------------------------------------------------------------------------------------------------------------------------
(2)、普通集群模式
# bin/run-exampleorg.apache.spark.examples.SparkPi spark://namenode1:7077
# bin/run-exampleorg.apache.spark.examples.SparkLR spark://namenode1:7077
# bin/run-exampleorg.apache.spark.examples.SparkKMeans spark://namenode1:7077file:/usr/local/spark/data/kmeans_data.txt 2 1
----------------------------------------------------------------------------------------------------------------------------------------------
(3)、结合HDFS的集群模式
# hadoop fs -put README.md .
# MASTER=spark://namenode1:7077bin/spark-shell
scala> val file =sc.textFile("hdfs://namenode1:9000/user/root/README.md")
scala> val count = file.flatMap(line=> line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)
scala> count.collect()
scala> :quit
----------------------------------------------------------------------------------------------------------------------------------------------
(4)、基于YARN模式
#SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar\
bin/spark-classorg.apache.spark.deploy.yarn.Client \
--jarexamples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar \
--classorg.apache.spark.examples.SparkPi \
--args yarn-standalone \
--num-workers 3 \
--master-memory 4g \
--worker-memory 2g \
--worker-cores 1
----------------------------------------------------------------------------------------------------------------------------------------------
(5)、最终的目录结构及相关配置
目录结构：
配置文件“/etc/profile”中的环境变量设置：
本文出自 “人生理想在于坚持不懈” 博客，请务必保留此出处http://sofar.blog.51cto.com/353572/1362240
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- spark与磁盘交互，spark读hdfs
- 成功编译hadoop eclipse插件。此方式适用于win 7
- 在standalone模式下运行yarn 0.9.0对HDFS上的数据进行计算
- 大规模Hadoop集群实践：腾讯分布式数据仓库（TDW）
- 用Hadoop构建电影推荐系统
- Hadoop 2.x.0集群安装与配置
- Tachyon：吞吐量超过HDFS 300多倍 来自伯克利的分布式文件系统
- Hadoop专业解决方案-第12章 为Hadoop应用构建企业级的安全解决方案
- Hadoop虽然强大，但不是万能的
- HDFS采用主从（Master/Slave）结构模型
- HDFS设计优劣
- libhdfs 的C API
- HDFS HA方案
- 恢复HDFS误删数据
- Hortonworks 联手 Red Hat，将构建更强大的 Hadoop
- 国内第一篇详细讲解hadoop2的automatic HA+Federation+Yarn配置的教程
- 读<王垠:一种新的操作系统设计>
- hdfs-site.xml
- HDFS中hsync方法介绍
- Flume-NG + HDFS + HIVE 日志收集分析