Hadoop 2.5.1学习笔记3：关于Combiner

如果把前面的例子加上Combiner.class

public static class Combiner extends Reducer<Text, Text, Text, Text> {
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
long count = 0;
for (Text val : values) {
count+=Long.parseLong(val.toString());
}
context.write(key, new Text(""+count));
}

}

然后指定 job.setCombinerClass(Combiner.class);

可以观察下两个的效率区别：
4/11/07 14:49:25 INFO mapreduce.Job: Counters: 38

File System Counters

FILE: Number of bytes read=52642504

FILE: Number of bytes written=95200714

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=608036374

HDFS: Number of bytes written=423

HDFS: Number of read operations=22

HDFS: Number of large read operations=0

HDFS: Number of write operations=5

Map-Reduce Framework

Map input records=2923923

Map output records=2923923

Map output bytes=20467464

Map output materialized bytes=26315322

Input split bytes=212

Combine input records=0

Combine output records=0

Reduce input groups=38

Reduce shuffle bytes=26315322

Reduce input records=2923923

Reduce output records=38

Spilled Records=5847846

Shuffled Maps =2

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms)=252

CPU time spent (ms)=0

Physical memory (bytes) snapshot=0

Virtual memory (bytes) snapshot=0

Total committed heap usage (bytes)=1150484480

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=236907275

File Output Format Counters

Bytes Written=423

使用后的：
14/11/07 16:04:49 INFO mapreduce.Job: Counters: 38

File System Counters

FILE: Number of bytes read=16224

FILE: Number of bytes written=704061

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=608036374

HDFS: Number of bytes written=423

HDFS: Number of read operations=22

HDFS: Number of large read operations=0

HDFS: Number of write operations=5

Map-Reduce Framework

Map input records=2923923

Map output records=2923923

Map output bytes=20467464

Map output materialized bytes=523

Input split bytes=212

Combine input records=2923923

Combine output records=39

Reduce input groups=38

Reduce shuffle bytes=523

Reduce input records=39

Reduce output records=38

Spilled Records=78

Shuffled Maps =2

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms)=281

CPU time spent (ms)=0

Physical memory (bytes) snapshot=0

Virtual memory (bytes) snapshot=0

Total committed heap usage (bytes)=1154875392

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=236907275

File Output Format Counters

Bytes Written=423

第一次耗费 28秒

第二次耗费21秒。