您的位置:首页 > 运维架构

编译hadoop-2.2.0源码,RHEL6.3x64

2014-03-28 16:17 169 查看
编译环境:
OS: RHEL6.3.x86_64
Kernel:2.6.32-358.el6.x86_64
JDK: 1.6.0_35 x86_64
编译前准备:
1.安装依赖的rpm包
yum -y install lzo-devel zlib-devel gcc autoconf automake libtool openssl-devel ncurses-devel cmake
2.安装maven
下载地址: http://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
解压到/usr/local/maven目录下,然后编辑/etc/profile,在其中编辑MAVEN_HOME的环境变量,保存后执行:
source /etc/profile 或 . /etc/profile
使之生效。
验证生效与否:
# mvn -v
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 23:22:22+0800)
Maven home: /usr/local/maven
Java version: 1.6.0_35, vendor: Sun Microsystems Inc.
Java home: /usr/java/jdk1.6.0_35/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: “unix"
3.安装protobuf http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar xf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make install

开始编译Hadoop
1.下载hadoop-2.2.0 源码。
地址:http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
2.tar xf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
3.在编译前,首先修复一个2.2.0版本的一个bug,否则会在编译过程中遇到如下报错:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
[ERROR] /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
cannot access org.mortbay.component.AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle
not found

bug patch地址:https://issues.apache.org/jira/browse/hadoop-10110
方法:修改hadoop源码目录下的hadoop-common-project/hadoop-auth/pom.xml
添加或修改如下部分内容:

<dependency>
<groupId>org.mortbay.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>

4.在hadoop源码目录下执行:

# mvn package -Pdist,native -DskipTests -Dtar -e -X

[INFO] Apache Hadoop Main ................................ SUCCESS [4.419s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [3.358s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [4.488s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.640s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [3.900s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.072s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [4.491s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.014s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:48.272s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [10.646s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.066s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [1:54.818s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [20.414s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [14.205s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [5.866s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.079s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.361s]
[INFO] hadoop-yarn-api ................................... SUCCESS [51.017s]
[INFO] hadoop-yarn-common ................................ SUCCESS [33.649s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.230s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [13.812s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.117s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.240s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [19.048s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.012s]
[INFO] hadoop-yarn-client ................................ SUCCESS [6.523s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.174s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [3.049s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.169s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [29.535s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.287s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.211s]
[INFO] hadoop-yarn-project ............................... SUCCESS [6.975s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [24.512s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.984s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [13.070s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [6.213s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [5.803s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.423s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [8.608s]
[INFO] hadoop-mapreduce .................................. SUCCESS [5.522s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [6.932s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [10.974s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [3.047s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [8.680s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [6.245s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [3.823s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [4.784s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [7.803s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [2.759s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.046s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [20.150s]
[INFO] Apache Hadoop Client .............................. SUCCESS [8.368s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.278s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:56.567s
[INFO] Finished at: Wed Dec 25 23:13:59 CST 2013
[INFO] Final Memory: 59M/121M
[INFO] ————————————————————————————————————

直到最后出现BUILD SUCCESS,说明编译成功。
源码目录下 hadoop-dist/target/hadoop-2.2.0.tar.gz 就是我们最终编译成功的hadoop包。
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: