apache版本的hadoop源码编译

apache版本的hadoop源码编译

标签（空格分隔）： hadoop

VMware Workstation 10.0.1（本课程学习实验全部在虚拟机里面完成）

系统：Centos6.5 2.6.32-358.el6.x86_64

内存：最低2GB 处理器：1 网卡：NAT模式（自动获取IP）

1，修改主机名

# vi /etc/sysconfig/network

HOSTNAME=hadoop-yarn.beifeng.com

2,关闭selinux和防火墙

# chkconfig iptables off
# vi /etc/sysconfig/selinux

SELINUX=disabled

3,添加DNS

# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DNS1=8.8.8.8
DNS2=8.8.4.4

4,新建几个目录

[root@localhost /]# mkdir -p /opt/softwares
[root@localhost /]# mkdir -p /opt/tools
[root@localhost /]# mkdir -p /opt/modules

5,解压jdk包到指定目录下

[root@hadoop-yarn softwares]# tar -zxf jdk-7u67-linux-x64.tar.gz -C /opt/modules/

6，删除系统自带的jdk

# rpm -qa|grep java
# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
# rpm -e --nodeps tzdata-java-2012j-1.el6.noarch

7,修改环境变量 /etc/profile

vi /etc/profile

export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile
# java -version //java命令验证一下版本

8,安装依赖包

# yum -y install autoconf automake libtool cmake
# yum -y install ncurses-devel
# yum -y install openssl-devel
# yum -y install lzo-devel zlib-devel gcc gcc-c++

9,安装protobuf

[root@hadoop-yarn softwares]# tar -zxf protobuf-2.5.0.tar.gz -C /opt/modules/

[root@hadoop-yarn modules]# cd protobuf-2.5.0/
# ./configure
# make
# make install

10,安装findbugs

下载解压apache-maven-3.0.5-bin.tar.gz

[root@hadoop-yarn softwares]# tar -zxf findbugs-1.3.9.tar.gz -C /opt/modules/

vi /etc/profile   //增加环境变量
------------------------------------
export FINDBUGS_HOME=/opt/modules/findbugs-1.3.9
export PATH=$PATH:$FINDBUGS_HOME/bin
------------------------------------

# source /etc/profile
# findbugs -version  //验证

11,配置maven环境

下载解压apache-maven-3.0.5-bin.tar.gz

[root@hadoop-yarn softwares]# tar -zxf apache-maven-3.0.5-bin.tar.gz -C /opt/modules/

vi /etc/profile   //增加环境变量
------------------------------------
export MAVEN_HOME=/opt/modules/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin
------------------------------------

# source /etc/profile

[root@hadoop-yarn /]# mkdir ~/.m2/     //home目录下
[root@hadoop-yarn /]# cp /opt/modules/apache-maven-3.0.5/conf/settings.xml   /root/.m2/

上传部分包（为节省时间我已经部分包打包成mvn-resp.tar.gz）
# tar -zxf mvn-resp.tar.gz   //解压以后在当前目录会有一个.m2的隐藏目录
# cd .m2
# cp -r repository /root/.m2/

12,编译安装hadoop

hadoop 下载网址（所有的）： http://archive.apache.org/dist/hadoop/common/ 源码包管理地址： https://github.com/apache/hadoop[/code] 
# tar -zxf hadoop-2.5.0-src.tar.gz -C /opt/modules/


使用maven命令编译hadoop
# cd hadoop-2.5.0-src/
# mvn package -Pdist,native -DskipTests -Dtar




13．移动目录

# cd /彩电opt/modules/hadoop-2.5.0-src/hadoop-dist/target
# mv hadoop-2.5.0.tar.gz  /opt/softwares/hadoop-2.5.0-bf.tar.gz

移出我们编译好的hadoop目录
# cd /opt/modules/hadoop-2.5.0-src/hadoop-dist/target
# mv hadoop-2.5.0/  /opt/modules/
（/opt/modules/hadoop-2.5.0/这就是我们hadoop的编译好的目录）


Hadoop默认情况下是不支持snappy压缩的，如果需要支持snappy的话，需要自行编译hadoop2.x源码完成：

1) 安装sanppy

2) 编译haodop 2.x源码

mvn package -Pdist,native -DskipTests -Dtar -Drequire.snappy

编译完成以后会生成新的/opt/modules/hadoop-2.5.0-src/target/hadoop-2.5.0/lib/native，然后替换原来的即可。

编译需要较长时间。

**报错Out Of Memory，由于内存原因导致，加export MAVEN_OPTS=”-Xms256m -Xmx512m”

**如果出现类似下面这样的错误，多尝试几次”mvn package -Pdist,native -DskipTests –Dtar”命令，这是因为网络不稳定造成的。

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Could not transfer artifact com.thoughtworks.qdox:qdox:jar:1.12 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused: Connection refused -> [Help 1]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Could not transfer artifact com.thoughtworks.qdox:qdox:jar:1.12 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused: Connection refused -> [Help 1]