hadoop 2.7.4 下运行WordCount例子笔记

1.源码如下：

package com.mapred.core;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.junit.Test;

import java.io.IOException;

public class WordCount {

@Test
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
//FileSystem fs = FileSystem.get(new URI("hdfs://192.168.70.128:9000"),conf); //理解为一个访问端到服务端的连接
Job job = new Job(conf);
//指明程序的入口
job.setJarByClass(WordCount.class);

//指明输入的数据
FileInputFormat.setInputPaths(job, new Path(args[0]));
//组织mapper和reducer
//设置mapper
job.setMapperClass(WordCountMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);

//设置reducer
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);

//指明数据输出的路径
FileOutputFormat.setOutputPath(job, new Path(args[1]));
//提交任务运行
job.waitForCompletion(true);
}
}

class WordCountMapper extends Mapper<LongWritable,Text,Text,LongWritable>{

@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String val = value.toString();
String[] words = val.split(" ");
for(String word : words){
context.write(new Text(word), new LongWritable(1));
}
}

}

class WordCountReducer extends Reducer<Text,LongWritable,Text,LongWritable>{
@Override
protected void reduce(Text key, Iterable<LongWritable> values,Context context)
throws IOException, InterruptedException {
long sum = 0;
for(LongWritable value : values){
sum += value.get();
}
context.write(key, new LongWritable(sum));
}

}

2.运行步骤笔记：

运行WordCount步骤：
1.将项目打成jar包，比如打成mapredProject.jar包。

2.上传mapredProject.jar到/soft目录

3.在/soft目录创建输入数据文件input.txt。查看input.txt的文件内容, more /soft/input.txt:
lengend
i
am
a
hero
i
am
a
fool
i
am
a
apple
but
you
are
a
bastard

4.在hdfs中创建输入数据存放目录：  hadoop fs -mkdir -p  /wordcount/in

5.将/soft/input.txt上传到hdfs文件系统的wordcount目录下： hadoop fs -put /soft/input.txt /wordcount/in

6.运行WordCount项目： hadoop jar /soft/mapredProject.jar  /wocount/in /wocount/output

7.等待完成，控制台会展示进度，作为示例，某次运行进度可能如下所示：
[hadoop@node1 soft]$ hadoop jar /soft/mapredProject.jar  /wocount/in /wocount/output
18/01/10 21:22:13 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.209.129:8032
18/01/10 21:22:14 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/01/10 21:22:17 INFO input.FileInputFormat: Total input paths to process : 1
18/01/10 21:22:17 INFO mapreduce.JobSubmitter: number of splits:1
18/01/10 21:22:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1515211219380_0005
18/01/10 21:22:21 INFO impl.YarnClientImpl: Submitted application application_1515211219380_0005
18/01/10 21:22:21 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1515211219380_0005/ 18/01/10 21:22:21 INFO mapreduce.Job: Running job: job_1515211219380_0005
18/01/10 21:23:18 INFO mapreduce.Job: Job job_1515211219380_0005 running in uber mode : false
18/01/10 21:23:18 INFO mapreduce.Job:  map 0% reduce 0%
18/01/10 21:23:51 INFO mapreduce.Job:  map 100% reduce 0%
18/01/10 21:24:10 INFO mapreduce.Job:  map 100% reduce 100%
18/01/10 21:24:12 INFO mapreduce.Job: Job job_1515211219380_0005 completed successfully
18/01/10 21:24:13 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=264
FILE: Number of bytes written=242009
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=171
HDFS: Number of bytes written=76
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=30472
Total time spent by all reduces in occupied slots (ms)=15714
Total time spent by all map tasks (ms)=30472
Total time spent by all reduce tasks (ms)=15714
Total vcore-milliseconds taken by all map tasks=30472
Total vcore-milliseconds taken by all reduce tasks=15714
Total megabyte-milliseconds taken by all map tasks=31203328
Total megabyte-milliseconds taken by all reduce tasks=16091136
Map-Reduce Framework
Map input records=19
Map output records=19
Map output bytes=220
Map output materialized bytes=264
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=12
Reduce shuffle bytes=264
Reduce input records=19
Reduce output records=12
Spilled Records=38
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=249
CPU time spent (ms)=2860
Physical memory (bytes) snapshot=288591872
Virtual memory (bytes) snapshot=4164571136
Total committed heap usage (bytes)=141230080
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=68
File Output Format Counters
Bytes Written=76

8.查看输出： hadoop fs -ls  /wocount/output，输出结果可能如下所示：
Found 2 items
-rw-r--r--   2 hadoop supergroup          0 2018-01-10 21:24 /wocount/output/_SUCCESS
-rw-r--r--   2 hadoop supergroup         76 2018-01-10 21:24 /wocount/output/part-r-00000

9.打开上述步骤列出的输入文件： hadoop fs -cat /wocount/output/part-r-00000 ， 可以看到统计的单词以及对应的数目如下所示：

1
a	4
am	3
apple	1
are	1
bastard	1
but	1
fool	1
hero	1
i	3
lengend	1
you	1