Linux搭建Hadoop开发环境

Linux搭建Hadoop开发环境
2.Hadoop环境搭建安装配置：
[1].官网下载Hadoop-2.7.5安装包： hadoop-2.7.5.tar.gz

[2].把Hadoop-2.7.5安装包利用Xftp5工具上传到：/usr/local/hadoop

[3].登录Liunx服务器，利用Xhell5进入：cd /usr/local/hadoop：
[root@marklin hadoop]# cd /usr/local/hadoop
       [root@marklin hadoop]#
       并使用tar -xvf 解压：tar -xvf hadoop-2.7.5.tar.gz，
[root@marklin hadoop]# tar -xvf hadoop-2.7.5.tar.gz

[4].配置Hadoop环境变量，输入：vim /etc/profile
  #Setting HADOOP_HOME PATH
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.5
export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin::${HADOOP_HOME}/lib
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export HADOOP_MAPARED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
 

保存配置，输入：source /etc/profile
[root@marklin ~]# source /etc/profile
       [root@marklin ~]#

[5].Hadoop修改配置文件：
(1). 配置core-site.xml，在Hadoop安装目录[/usr/local/hadoop/hadoop-2.7.5/etc/hadoop]下 输入： vim core-site.xml
[root@marklin ~]# cd /usr/local/hadoop/hadoop-2.7.5/etc/hadoop
 输入：vim core-site.xml

并配置：
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://127.0.0.1:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
</configuration>

同时在文件路径：/usr/local/hadoop/repository/hdfs，创建tmp目录： mkdir tmp
（2） 修改 hdfs-site.xml，并配置：vim hdfs-site.xml
[root@marklin hadoop]# vim hdfs-site.xml
[root@marklin hadoop]# 

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/name</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/data</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>127.0.0.1:50070</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>127.0.0.1:9001</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

同时在文件路径：/usr/local/hadoop/repository/hdfs，创建name和data目录： mkdir  name 和mkdir  data
（3） 创建mapred-site.xml文件，输入：cp mapred-site.xml.template mapred-site.xml
[root@marklin hadoop]# cp mapred-site.xml.template mapred-site.xml

编辑mapred-site.xml文件，并配置：
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>hdfs://127.0.0.1:8021/</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>127.0.0.1:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>127.0.0.1:19888</value>
    </property>
    <property>
        <name>mapred.system.dir</name>
        <value>/usr/local/hadoop/repository/mapreduce/system</value>
        <final>true</final>
    </property>
    <property>
        <name>mapred.local.dir</name>
        <value>/usr/local/hadoop/repository/mapreduce/local</value>
        <final>true</final>
    </property>
</configuration>

（4） 修改 yarn-site.xml，并输入：:vim yarn-site.xml
[root@marklin hadoop]# vim yarn-site.xml
[root@marklin hadoop]#

并配置：
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>127.0.0.1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>768</value>
    </property>
</configuration>

【6】在Hadoop文件目录[/usr/local/hadoop/hadoop-2.7.5/etc/hadoop]下，
对应的 hadoop-env.sh,mapred-env.sh以及yarn-env.sh文件配置JAVA_HOME：export JAVA_HOME=/usr/local/java/jdk1.8.0_162
输入：vim hadoop-env.sh ：
[root@marklin hadoop]# vim hadoop-env.sh
[root@marklin hadoop]#

export JAVA_HOME=/usr/local/java/jdk1.8.0_162

输入：vim mapred-env.sh：
export JAVA_HOME=/usr/local/java/jdk1.8.0_162

[root@marklin hadoop]# vim mapred-env.sh
[root@marklin hadoop]#

输入：vim yarn-env.sh
export JAVA_HOME=/usr/local/java/jdk1.8.0_162

[root@marklin hadoop]# vim yarn-env.sh
[root@marklin hadoop]#

 
【6】开放端口：50070
（1）启动防火墙：systemctl start firewalld.service
[root@marklin ~]# systemctl start firewalld.service
[root@marklin ~]#

（2）启动防火墙：firewall-cmd --zone=public --add-port=50070/tcp --permanent
[root@marklin ~]# firewall-cmd --zone=public --add-port=50070/tcp --permanent
[root@marklin ~]#

(3)启动：firewall-cmd --reload
[root@marklin ~]# firewall-cmd --reload
[root@marklin ~]# 

（4）格式化：hdfs namenode -format
[root@marklin ~]# hdfs namenode -format
[root@marklin ~]#

（5）启动脚本：start-all.sh
[root@marklin ~]# start-all.sh
[root@marklin ~]#