hadoop2.3.0编译及安装配置

一、Hadoop 2.3.0源码编译

1.工具

----------------------------------------------------------------------------------

Requirements:

* Unix System

* JDK 1.6+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer (if compiling native code)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------

安装jdk
http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz?AuthParam=1395828777_fcd88f7a3b2c6b67e39c64adec67acc3
安装maven
http://apache.dataguru.cn/maven/maven-3/3.1.1/source/apache-maven-3.1.1-src.tar.gz
安装Findbugs
http://ncu.dl.sourceforge.net/project/findbugs/findbugs/2.0.3/findbugs-2.0.3.tar.gz
安装protobuf
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2
hadoop2.3.0源码
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.3.0/hadoop-2.3.0-src.tar.gz
参考：http://wiki.apache.org/hadoop/HowToContribute

2.解压源码，然后用mvn编译mvn package -Pdist,native,docs -DskipTests -Dtar

二、Hadoop 2.3.0集群配置

配置项参考：http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

安装包下载地址：http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.3.0/hadoop-2.3.0.tar.gz

1.jdk配置

2.ssh无密码登陆

3.配置hdfs-site.xml ,core-site.xml,mapred-site.xml,yarn-site.xml,slaves,yarn-env.sh,haoop-env.sh（jdk位置）

4.启动hadoop start-dfs.sh start-yarn.sh

附：

hadoop编译参考：

http://www.micmiu.com/bigdata/hadoop/hadoop-build-source-2-2-0/

http://blog.chinaunix.net/uid-9460004-id-4047994.html

hadoop子项目配置参考：

http://m.blog.csdn.net/blog/jinnchang/8900992

http://wenku.baidu.com/view/fe1b2f22de80d4d8d15a4f6e.html

注：11/09/20 17:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes
where applicable 出现这样的错误

1.指定本地库的路径,hadoop-env.sh

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

2.库不兼容本机机器系统，本地maven编译hadoop项目，将编译所得的本地库替换libhadoop.so就可以了。

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html