mac os x 编译hadoop
2014-06-03 12:39
381 查看
查阅有关官方介绍 http://wiki.apache.org/hadoop/HowToContribute 中有说明：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持
Cygwin 和 Mac OS X 。搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。
[一]、环境说明：
Hadoop 2.2.0
Mac OS X 10.9.1
详细的环境依赖（protoc、cmake 等）参见：Hadoop2.2.0源码编译 （http://www.micmiu.com/opensource/hadoop/hadoop-build-source-2-2-0/）中介绍。
[二]、Mac OSX 编译本地库的步骤：
1、checkout Hadoop 2.2.0的源码
2、patch 相关补丁
官方讨论地址：https://issues.apache.org/jira/browse/HADOOP-9648 里面有详细介绍
补丁下载链接：https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
ps：如果要回退patch 执行：
3、编译本地库
在Hadoop源码的根目录下执行编译本地库命令：
编译成功看到如下日志信息：
编译通过后可在 <HADOOP源码根目录>/hadoop-dist/target/hadoop-2.2.0/lib/ 目录下看到如下内容：
查看源代码
打印帮助
然后把 上面生成的本地库 copy到部署环境相应的位置，再建立软连接即可：
错误处理
运行
clean install package -Pdist -P-cbuild -DskipTests -Dtar
报各种错误
1、报错[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
修改hadoop-common-project/hadoop-common/pom.xml 文件中，env.JAVA_HOME改为java.home
2、报错
/hadoop-2.2.0-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression (‘void’ and ‘int’)
[exec] if(setnetgrent(cgroup) == 1) {
[exec] ~~~~~~~~~~~~~~~~~~~ ^ ~
[exec] 1 error generated.
[exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[exec] make: *** [all] Error 2
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]
修改mvn3的配置文件：/opt/local/share/java/maven3/settings.xml
在<mirrors>…</mirrors>里添加国内源：
<mirrors>
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
</mirrors>
在<profiles>…</profiles>标签中增加以下内容：
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7<k>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
注意修改jdk version number
将刚才的maven 配置文件拷贝到当前用户的home目录下:
settings.xml copy 到 your_hadoop_usr_home/.m2/
cp settings.xml ~/.m2
3、报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space
分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
4、报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] Failure executing javac, but could not parse the error:
执行maven clean，然后再
export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
三、最重要的一点，build your code是使用这个command line(Only for Mac OS):
mvn clean install -P-cbuild
编译之前， 你在hadoop-2.2.0-src目录（/Users/JuneMAC/hadoop/release-2.2.0）下执行
mvn clean install –DskipTests
上面的成功后，执行下面这个，生成安装包
mvn clean install package -Pdist -P-cbuild -DskipTests -Dtar
执行完成后，可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
下找到
hadoop-2.2.0.tar.gz
将上面这个编译好的源码包解压到：
/Users/JuneMAC/hadoop/
然后进行相关配置
解压之后的源码包和官网下载下来的源码包相对比，没有lib目录
相关解释：
“Here we use the additional options to stop compiling the native code.
this is the key reason why we need use -P-cbuild option”
上面这个是原因，好像不是很重要。实际上如果指定-Pdist,native 生成native lib 不成功，查阅有关官方介绍发现：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin 和 Mac OS X 。
Cygwin 和 Mac OS X 。搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。
[一]、环境说明：
Hadoop 2.2.0
Mac OS X 10.9.1
详细的环境依赖（protoc、cmake 等）参见：Hadoop2.2.0源码编译 （http://www.micmiu.com/opensource/hadoop/hadoop-build-source-2-2-0/）中介绍。
[二]、Mac OSX 编译本地库的步骤：
1、checkout Hadoop 2.2.0的源码
1 | $svn co https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0/ |
官方讨论地址：https://issues.apache.org/jira/browse/HADOOP-9648 里面有详细介绍
补丁下载链接：https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
1 | #切换到hadoop 源码的根目录 |
2 | $wget https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch |
3 | $patch -p1 < HADOOP-9648.v2.patch |
patch -RE -p1 < HADOOP-9648.v2.patch即可。
3、编译本地库
在Hadoop源码的根目录下执行编译本地库命令：
1 | $ mvn package -Pdist,native -DskipTests -Dtar |
1 | [INFO] ------------------------------------------------------------------------ |
2 | [INFO] Reactor Summary: |
3 | [INFO] |
4 | [INFO] Apache Hadoop Main ................................ SUCCESS [1.511s] |
5 | [INFO] Apache Hadoop Project POM ......................... SUCCESS [0.493s] |
6 | [INFO] Apache Hadoop Annotations ......................... SUCCESS [0.823s] |
7 | [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.561s] |
8 | [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.245s] |
9 | [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [2.465s] |
10 | [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [0.749s] |
11 | [INFO] Apache Hadoop Auth ................................ SUCCESS [0.832s] |
12 | [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.070s] |
13 | [INFO] Apache Hadoop Common .............................. SUCCESS [1:00.030s] |
14 | [INFO] Apache Hadoop NFS ................................. SUCCESS [0.285s] |
15 | [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.049s] |
16 | [INFO] Apache Hadoop HDFS ................................ SUCCESS [1:13.339s] |
17 | [INFO] Apache Hadoop HttpFS .............................. SUCCESS [20.259s] |
18 | [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [0.767s] |
19 | [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [0.279s] |
20 | [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.046s] |
21 | [INFO] hadoop-yarn ....................................... SUCCESS [0.239s] |
22 | [INFO] hadoop-yarn-api ................................... SUCCESS [7.641s] |
23 | [INFO] hadoop-yarn-common ................................ SUCCESS [5.479s] |
24 | [INFO] hadoop-yarn-server ................................ SUCCESS [0.114s] |
25 | [INFO] hadoop-yarn-server-common ......................... SUCCESS [1.743s] |
26 | [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [6.381s] |
27 | [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.259s] |
28 | [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [0.578s] |
29 | [INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.303s] |
30 | [INFO] hadoop-yarn-client ................................ SUCCESS [0.233s] |
31 | [INFO] hadoop-yarn-applications .......................... SUCCESS [0.062s] |
32 | [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [0.253s] |
33 | [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.074s] |
34 | [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [1.504s] |
35 | [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.242s] |
36 | [INFO] hadoop-yarn-site .................................. SUCCESS [0.172s] |
37 | [INFO] hadoop-yarn-project ............................... SUCCESS [1.235s] |
38 | [INFO] hadoop-mapreduce-client-common .................... SUCCESS [3.664s] |
39 | [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [0.183s] |
40 | [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [0.495s] |
41 | [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [1.296s] |
42 | [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [0.580s] |
43 | [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [0.213s] |
44 | [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [0.344s] |
45 | [INFO] hadoop-mapreduce .................................. SUCCESS [1.303s] |
46 | [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [0.257s] |
47 | [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.925s] |
48 | [INFO] Apache Hadoop Archives ............................ SUCCESS [0.282s] |
49 | [INFO] Apache Hadoop Rumen ............................... SUCCESS [0.403s] |
50 | [INFO] Apache Hadoop Gridmix ............................. SUCCESS [0.283s] |
51 | [INFO] Apache Hadoop Data Join ........................... SUCCESS [0.197s] |
52 | [INFO] Apache Hadoop Extras .............................. SUCCESS [0.241s] |
53 | [INFO] Apache Hadoop Pipes ............................... SUCCESS [8.249s] |
54 | [INFO] Apache Hadoop OpenStack support ................... SUCCESS [0.492s] |
55 | [INFO] Apache Hadoop Client .............................. SUCCESS [0.373s] |
56 | [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.133s] |
57 | [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [0.439s] |
58 | [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [0.596s] |
59 | [INFO] Apache Hadoop Tools ............................... SUCCESS [0.044s] |
60 | [INFO] Apache Hadoop Distribution ........................ SUCCESS [0.194s] |
61 | [INFO] ------------------------------------------------------------------------ |
62 | [INFO] BUILD SUCCESS |
63 | [INFO] ------------------------------------------------------------------------ |
64 | [INFO] Total time : 3:44.266s |
65 | [INFO] Finished at: Fri Jan 17 10:06:17 CST 2014 |
66 | [INFO] Final Memory: 66M/123M |
67 | [INFO] ------------------------------------------------------------------------ |
68 | micmiu-mbp:trunk micmiu$ |
查看源代码
打印帮助
1 | micmiu-mbp:lib micmiu$ tree |
2 | . |
3 | |____.DS_Store |
4 | |____native |
5 | | |____libhadoop.1.0.0.dylib |
6 | | |____libhadoop.a |
7 | | |____libhadoop.dylib |
8 | | |____libhadooppipes.a |
9 | | |____libhadooputils.a |
10 | | |____libhdfs.0.0.0.dylib |
11 | | |____libhdfs.a |
12 | | |____libhdfs.dylib |
1 | $ ls -s libhadoop.1.0.0.dylib libhadoop.so |
2 | $ ls -s libhdfs.0.0.0.dylib libhdfs.so |
运行
clean install package -Pdist -P-cbuild -DskipTests -Dtar
报各种错误
1、报错[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
修改hadoop-common-project/hadoop-common/pom.xml 文件中，env.JAVA_HOME改为java.home
2、报错
/hadoop-2.2.0-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression (‘void’ and ‘int’)
[exec] if(setnetgrent(cgroup) == 1) {
[exec] ~~~~~~~~~~~~~~~~~~~ ^ ~
[exec] 1 error generated.
[exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[exec] make: *** [all] Error 2
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]
修改mvn3的配置文件：/opt/local/share/java/maven3/settings.xml
在<mirrors>…</mirrors>里添加国内源：
<mirrors>
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
</mirrors>
在<profiles>…</profiles>标签中增加以下内容：
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7<k>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
注意修改jdk version number
将刚才的maven 配置文件拷贝到当前用户的home目录下:
settings.xml copy 到 your_hadoop_usr_home/.m2/
cp settings.xml ~/.m2
3、报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space
分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
4、报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] Failure executing javac, but could not parse the error:
执行maven clean，然后再
export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
三、最重要的一点，build your code是使用这个command line(Only for Mac OS):
mvn clean install -P-cbuild
编译之前， 你在hadoop-2.2.0-src目录（/Users/JuneMAC/hadoop/release-2.2.0）下执行
mvn clean install –DskipTests
上面的成功后，执行下面这个，生成安装包
mvn clean install package -Pdist -P-cbuild -DskipTests -Dtar
执行完成后，可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
下找到
hadoop-2.2.0.tar.gz
将上面这个编译好的源码包解压到：
/Users/JuneMAC/hadoop/
然后进行相关配置
解压之后的源码包和官网下载下来的源码包相对比，没有lib目录
相关解释：
“Here we use the additional options to stop compiling the native code.
this is the key reason why we need use -P-cbuild option”
上面这个是原因，好像不是很重要。实际上如果指定-Pdist,native 生成native lib 不成功，查阅有关官方介绍发现：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin 和 Mac OS X 。
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- mac os x 编译hadoop
- mac OS X Yosemite 上编译hadoop 2.6.0/2.7.0及TEZ 0.5.2/0.7.0 注意事项
- 在Mac OS X 64bit系统上编译hadoop 2.2源码
- Mac OS X 10.10编译Hadoop 2.6.0笔记
- Mac OSX下编译 Hadoop 2.6.4
- Mac OS X Lion 下编译Android 源码
- Mac OS X下搭建Android Source编译环境的问题及解决方法
- C++插件框架已在 Mac OS X 下编译通过
- Mac OS X 10.6下android源码编译
- [整] Mac OS X 10.6 下 android 源码编译
- Mac OS X 下编译android源码
- [转] 在 Mac OS X 下编译 Objective-C 运行时
- Mac OS X 平台上编译WebCore 注意事项
- 在Mac os x Lion下编译OpenCASCADE
- (转载)在 Mac OS X 上编译 V8
- (转载)在 Mac OS X 上编译 V8
- 在Mac OS X下编译 boost|building boost library under mac os x
- mac os X 下编译jrtplib
- Apple Mac OS 下编译Emacs 23
- 在Mac OS X上安装Xcode以及编译Objective-C