hadoop(1.1.2) eclipse plugin编译

-------------------------------------------------------------------
下载安装eclipse

-------------------------------------------------------------------

$sudo apt-get install eclipse
#会自动安装eclipse-jdt，其jar文件 ubuntu 12.10 在'/usr/share/eclipse/dropins/jdt/plugins'里，ubuntu 12.04在'/usr/lib/eclipse/dropins/jdt/plugins'里

#这里使用ubuntu 12.04

$sudo vim /etc/profile

ADD:

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk

-------------------------------------------------------------------
下载安装hadoop

-------------------------------------------------------------------

$sudo tar -xzf hadoop-1.1.2.tar.gz -C /usr/local

$sudo chown -R hadoop:hadoop /usr/local/hadoop-1.1.2

$sudo vim /etc/profile
增加:

export HADOOP_HOME=/usr/local/hadoop-1.1.2

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export CLASSPATH=$HADOOP_HOME/hadoop-core-1.1.2.jar:$HADOOP_HOME:$HADOOP_HOME/lib:$CLASSPATH

-------------------------------------------------------------------
编译hadoop eclipse插件

-------------------------------------------------------------------

(1)

$mkdir -p $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/hadoop-core-1.1.2.jar                          $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/commons-cli-1.2.jar                       $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/commons-httpclient-3.0.1.jar        $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/commons-configuration-1.6.jar    $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/commons-lang-2.4.jar                    $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/jackson-mapper-asl-1.8.8.jar        $HADOOP_HOME/build/contrib/eclipse-plugin/lib

$cp $HADOOP_HOME/lib/jackson-core-asl-1.8.8.jar              $HADOOP_HOME/build/contrib/eclipse-plugin/lib
或

$cp $HADOOP_HOME/hadoop-core-1.1.2.jar $HADOOP_HOME/lib/commons-cli-1.2.jar  $HADOOP_HOME/lib/commons-httpclient-3.0.1.jar $HADOOP_HOME/lib/commons-configuration-1.6.jar $HADOOP_HOME/lib/commons-lang-2.4.jar  $HADOOP_HOME/lib/jackson-mapper-asl-1.8.8.jar $HADOOP_HOME/lib/jackson-core-asl-1.8.8.jar
   $HADOOP_HOME/build/contrib/eclipse-plugin/lib

(2)
修改build.xml和MANIFEST.MF:

$cd $HADOOP_HOME/src/contrib/eclipse-plugin

$vim build.xml
注释以下两行：

    <copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>

    <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

$vim META-INF/MANIFEST.MF
修改：

Bundle-ClassPath: classes/,

 lib/hadoop-core.jar
为：

Bundle-ClassPath: classes/,

 lib/hadoop-core-1.1.2.jar,

 lib/commons-cli-1.2.jar,

 lib/commons-httpclient-3.0.1.jar,

 lib/commons-configuration-1.6.jar,

 lib/commons-lang-2.4.jar,

 lib/jackson-mapper-asl-1.8.8.jar,

 lib/jackson-core-asl-1.8.8.jar

(3)
#把相应eclipse-jdt到jar文件复制到/usr/lib/eclipse/plugins

$sudo cp -r /usr/lib/eclipse/dropins/jdt/plugins/*  /usr/lib/eclipse/plugins

$ant -Declipse.home=/usr/lib/eclipse -Dbuild.dir=/usr/local/hadoop-1.1.2/build/contrib/eclipse-plugin -Dversion=1.1.2 jar

#成功后，在/usr/local/hadoop-1.1.2/build/contrib/eclipse-plugin目录下有相应的jar。

-------------------------------------------------------------------
配置eclipse

-------------------------------------------------------------------

$sudo cp /usr/local/hadoop-1.1.2/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.1.2.jar /usr/lib/eclipse/plugins

$eclipse
#可以在左侧的Project Explore看到"DFS Locations".

#如果没有，请重新安装一次eclipse

-------------------------------------------------------------------
在eclipse运行hadoop程序

-------------------------------------------------------------------

(1)Window -> Preferences -> Hadoop Map/Reduce 输入hadoop安装目录

(2)Window -> Show View -> other -> MapReduce Tools -> MapReduce Locations 点击OK

(3)点击右侧大象 New Hadoop Location ，配置如下：

Location name : 可任意
Map/Reduce Master

--Host: localhost (JobTracker进程 所在主机)

--Port: 8021 (JobTracker进程 接受请求的RPC端口)
DFS Master

--Host: localhost (NameNode进程 所在主机)

--Port: 8020 (NameNode进程 接受请求到RPC端口)
User Name: hadoop (Hadoop client用户名)
SOCKS proxy:这里不使用(SOCKS proxy can be configured if you cannot access the Map/Reduce location directly because your machine is not directly connected to the location. )

(4)启动hadoop集群

a.伪分布式集群部署：
----hadoop-env.sh:

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk

----core-site.xml:

<configuration>

  <property>

    <name>hadoop.tmp.dir</name>

    <value>/home/hadoop/hadoop-tmp</value>

  </property>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost:8020/</value>

  </property>

</configuration>

----mapred-site.xml:

<configuration>

  <property>

    <name>mapred.job.tracker</name>

    <value>localhost:8021</value>

  </property>

</configuration>
b.配置ssh:
~$ssh-keygen -t rsa -P 'hadoop' -f ~/.ssh/id_rsa_hadoop
#生成一个rsa密钥对，私钥在id_rsa_hadoop，公钥在id_rsa_hadoop.pub，密码为"hadoop"

~$cat ~/.ssh/id_rsa_hadoop.pub >> ~/.ssh/authorized_keys
#如果是集群，则需要把authorized_keys复制到所有节点

~$ssh-agent bash
#启动ssh代理

~$ssh-add ~/.ssh/id_rsa_hadoop
#输入rsa密码
c.启动集群：
~$hadoop namenode -format

~$start-all.sh

~$jps
显示如下，则成功：

24749 Jps

2526 

24608 TaskTracker

24272 SecondaryNameNode

23769 NameNode

24015 DataNode

24359 JobTracker
d.运行基准测试程序wordcount
~$hadoop fs -mkdir /user/hadoop/wordcount/input

~$hadoop fs -copyFromLocal /home/hadoop/examples.desktop /user/hadoop/wordcount/input/

~$hadoop jar $HADOOP_HOME/hadoop-examples-1.1.2.jar wordcount /user/hadoop/wordcount/input /user/hadoop/wordcount/output
~$hadoop fs -cat /user/hadoop/wordcount/output/part*

#可以看到正常的输出

(5)测试eclipse和hadoop的连接
在eclipse DFS Locations按F5 ，可以看到：



(6)在eclipse 上运行wordcount程序：
wordcount的源代码：$HADOOP_HOME/src/examples/org/apache/hadoop/examples/WordCount.java
在eclipse新建一个MAP/REDUCE project：



在MP-Wordcount工程下的src目录下新建一个java类：



复制wordcount的源代码进去，把"package org.apache.hadoop.examples;"替换成"package test;"



在Project Explore窗口里，右击"WordCount.java"，选择"Run AS"->"Run on hadoop"

这个时候报错如下：

Usage: wordcount <in> <out>

在Project Explore窗口里，右击"WordCount.java"，选择"Run AS"->"Run Configurations",配置java application





点"Apply"，再点"Run"
报错：

Output directory /user/hadoop/wordcount/output already exists

~$hadoop fs -rmr /user/hadoop/wordcount/output
#再次进行"Run As"->"Run Configurations"->"Run"

~$hadoop fs -cat /user/hadoop/wordcount/output/part*
#可以看到正确的输出