An Example of Hadoop MapReduce Counter
2011-10-09 16:25
260 查看
MapReduce Counter
Hadoop MapReduce Counter provides a way to measure the progress or the number of operations that occur within MapReduce programs. Basically, MapReduce framework provides a number of built-in counters to measure basic I/O operations, such as FILE_BYTES_READ/WRITTENand Map/Combine/Reduce input/output records. These counters are very useful especially when you evaluate some MapReduce programs. Besides, the MapReduce Counter allows users to employ your own counters. Since MapReduce Counters are automatically aggregated
over Map and Reduce phases, it is one of the easiest way to investigate internal behaviors of MapReduce programs. In this post, I’m going to introduce how to use your own MapReduce Counter. The example sources described in this post are based on Hadoop 0.21
API.
Incrementing your counter
For your own MapReduce counter, you first define a enum typeas follow:
public static enum MATCH_COUNTER { INCOMING_GRAPHS, PRUNING_BY_NCV, PRUNING_BY_COUNT, PRUNING_BY_ISO, ISOMORPHIC };
And then, when you want to increment your own counter, you should call the increment method
as follows:
context.getCounter(MATCH_COUNTER.INCOMING_GRAPHS).increment(1);
You can access context instance
within setup, cleanup, map,
and reduce method
in Mapper or Reducer class. You can get a desired counter via calling context.getCounter method
with some enum value.
Finding your counter
You can get some Counters froma finished job as follows:
Configuration conf = new Configuration(); Cluster cluster = new Cluster(conf); Job job = Job.getInstance(cluster,conf); result = job.waitForCompletion(true); ... Counters counters = job.getCounters();
The instance of Counters class
contains all of the counters obtained from a job. So, when you want to get your own counter, you should call findCounter method
with a enum type
as follows:
Counter c1 = counters.findCounter(MATCH_COUNTER.INCOMING_GRAPHS); System.out.println(c1.getDisplayName()+":"+c1.getValue());
The below example shows how to get built-in counter groups that Hadoop provides basically.
for (CounterGroup group : counters) { System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")"); System.out.println(" number of counters in this group: " + group.size()); for (Counter counter : group) { System.out.println(" - " + counter.getDisplayName() + ": " + counter.getName() + ": "+counter.getValue()); } }
原文地址:http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/
相关文章推荐
- Hadoop中MapReduce中的counter的意义
- hadoop下运行mapreduce程序Failed to set permissions of path:的解决
- The Next Generation of Apache Hadoop MapReduce
- 解决Hadoop运行jar包时MapReduce任务启动前OutOfMemoryError:Java heap space问题
- logstash 如何处理 mongodb 导出来的 _id value数据。 how to custom fields of logstash by mongo mapreduce exported data.(example format: {_id:"xxx"} , value:{})
- Hadoop中MapReduce中的counter的意义
- An Example of How Oracle Works
- Next Generation of Apache Hadoop MapReduce – The Scheduler
- An example of a trading strategy coded using Quantmod Package in R
- How to sort an arbitrarily large set of data using Hadoop?
- Writing an Hadoop MapReduce Program in Python
- 用Python语言写Hadoop MapReduce程序Writing an Hadoop MapReduce Program in Python
- An example of MFC ( help myself with beginning in this field )
- Hadoop中MapReduce中的counter的意义
- An example of the <sql> and <include> tags
- Writing an Hadoop MapReduce Program in Python
- An example of using libexif to set JPEG Orientation Tag
- Basic Example of an Array OF AWK 【每日一译】--2013-01-27
- Django: An example of Django template
- hadoop-mapreduce-example中实例介绍