hadoop wordcount练习

今天下载安装、运行了下hadoop，跟着doc走，但有时也会出现一些问题，所以在这里记录一下：

1. 下载hadoop

    我这里下载的是hadoop-0.21.0，之前在公司使用的是0.18，现在新版本的hadoop已经升级到了1.0.0,对API做了新的处理，下载地址：https://www.apache.org/dyn/closer.cgi/hadoop/core/

2. 解压、配置环境变量

    tar -zxvf hadoop-0.21.0.tar.gz

    ln -s hadoop-0.21.0 hadoop

    cd hadoop

    cd conf

修改hadoop-env.sh

配置JAVA_HOME

配置core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000/</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

mapred-site.xml

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

3. 格式化

    ./bin/hadoop namenode -format

    启动hadoop

    ./bin/hadoop start-all.sh

    验证启动是否成功

    jps

4.  hadoop分为单机模式、伪分布式模式、集群模式，这里 采用伪分布式模式，例子可参考hadoop快速入门文档:http://hadoop.apache.org/common/docs/r0.19.2/cn/quickstart.html

以下便是对wordcount的操作记录：

1> 编译WordCount.java

    javac -classpath $classpath -d wordcount_classes WordCount.java

2>打jar包

    jar -cvf wordcount.jar -C wordcount_classes

    ./bin/hadoop fs -put wordcount.jar .

3>增加file01 file02到input

    ./bin/hadoop fs -mkdir input

    ./bin/hadoop fs -put file01 input

    ./bin/hadoop fs -put file02 input

   查看input目录文件

    ./bin/hadoop fs -ls input

4>执行wordcount统计

    ./bin/hadoop jar wordcount.jar WordCount input output

5>查看统计结果

    ./bin/hadoop fs -cat /user/gavin/output/part-00000

Bye    1
Goodbye    1
Hadoop    2
Hello    2
World    2

期间遇到的问题是output目录已存在：

gavin@ubuntu:~/programs/hadoop$ ./bin/hadoop jar wordcount.jar WordCount input output
11/12/24 23:01:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/12/24 23:01:18 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/12/24 23:01:19 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/12/24 23:01:19 INFO mapreduce.JobSubmitter: Cleaning up the staging area hdfs://localhost:9000/tmp/hadoop-gavin/mapred/staging/gavin/.staging/job_201112242122_0007
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/gavin/output already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:375)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:334)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
at WordCount.main(WordCount.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

 这是只要删除output目录即可

    ./bin/hadoop fs -rmr output

再重新执行jar命令就可以看到输出结果

参考:

Hadoop Map/Reduce教程
http://hadoop.apache.org/common/docs/r0.19.2/cn/mapred_tutorial.html#%E7%9B%AE%E7%9A%84
hadoop 快速入门
http://hadoop.apache.org/common/docs/r0.19.2/cn/quickstart.html

hadoop 命令手册
http://hadoop.apache.org/common/docs/r0.19.2/cn/commands_manual.html
Hadoop Shell命令
http://hadoop.apache.org/common/docs/r0.19.2/cn/hdfs_shell.html
Hadoop分布式文件系统：架构和设计
http://hadoop.apache.org/common/docs/r0.19.2/cn/hdfs_design.html

hadoop下载
https://www.apache.org/dyn/closer.cgi/hadoop/core/