关于mapreduce程序之Mapper、Reducer以及Driver驱动的若干框架总结
2013-08-21 20:23
393 查看
第一套:
为旧的API,此时Maper和Reducer为接口,且该接口均在org.apache.hadoop.mapred.*;中
Mapper:
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
public class ExampleMapper extends MapRecuceBase implements Mapper<LongWritable,Text,Text,IntWritable>
{
public void map(LongWritable key,Text value,OutputCollector<Text,IntWritable> output,Reporterreporter)
throwsIOException
{
.......
}
}
Reducer:
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Reducer;
public class ExampleReducer extends MapReduceBase implements
Reducer<Text,IntWritable,Text,IntWritable>
{
public void reduce(Text key,Iterator<IntWritable> values,OutputCollector<Text,IntWritable>,Reporter
reporter)
{
........
}
}
Driver:
public class ExampleDrvier{
public static void main(String args[]) throws IOException{
JobConf=new JobConf(ExampleDrvier.class);
conf.setJobName(“Example”);
........
JobClient.runJob(conf);
}
}
或者:
public class ExampleDriver{
public int run (String args[]) throws Exception{
JobConf conf=new JobConf( getConf(), getClass());
conf.setJobName("example");
..........
JobClient.runJob(conf);
return 0;
}
public static void main(String args[]) throws Exception{
int exitCode=ToolRunner.run(new ExampleDriver(),args);
System.exit(exitCode);
}
}
第二套:
为新的API,此时Maper和Reducer为虚类,且该接口均在org.apache.hadoop.mapreduce.*;中
Mapper:
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Mapper.Context;
public class ExampleMapper extends Mapper<LongWritable,Text,Text,IntWritable>
{
public void map(LongWritable key,Text value,Context context) throwsIOException,InterruptedException
{
..........
}
}
Reducer:
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Reducer.Context;
class ExampleReducer extends Reducer<Text ,Text,Text,Text>
{
public void reduce(Text key,Iterable<Text> values,Context context) throwsIOException,InterruptedException
{
..........
}
}
Driver:
public class ExampleDriver{
public static void main(String args[]) throws Exception{
Job job=new Job();
job.setJarByClass(ExampleDriver.class);
............
job.setMapperClass();
.......
a98c
......
System.exit(job.waitForCompletion(true)?0:1);
}
}
或者
public class ExampleDriver extends Configured implements Tool
{
public int run(String args[])throws Exception
{
Configuration conf=getConf();
Job job=new Job(conf,"Example"); //任务名
job.setJarByClass(ExampleDriver.class); //指定class
............
job.waitForCompletion(true);
return job.isSuccessful()?0:1;
}
public static void main(String args[]) throws Exception
{
int res=ToolRunner.run(new Configuration(),new ExampleDriver(),args);
System.exit(res);
}
}
为旧的API,此时Maper和Reducer为接口,且该接口均在org.apache.hadoop.mapred.*;中
Mapper:
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
public class ExampleMapper extends MapRecuceBase implements Mapper<LongWritable,Text,Text,IntWritable>
{
public void map(LongWritable key,Text value,OutputCollector<Text,IntWritable> output,Reporterreporter)
throwsIOException
{
.......
}
}
Reducer:
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Reducer;
public class ExampleReducer extends MapReduceBase implements
Reducer<Text,IntWritable,Text,IntWritable>
{
public void reduce(Text key,Iterator<IntWritable> values,OutputCollector<Text,IntWritable>,Reporter
reporter)
{
........
}
}
Driver:
public class ExampleDrvier{
public static void main(String args[]) throws IOException{
JobConf=new JobConf(ExampleDrvier.class);
conf.setJobName(“Example”);
........
JobClient.runJob(conf);
}
}
或者:
public class ExampleDriver{
public int run (String args[]) throws Exception{
JobConf conf=new JobConf( getConf(), getClass());
conf.setJobName("example");
..........
JobClient.runJob(conf);
return 0;
}
public static void main(String args[]) throws Exception{
int exitCode=ToolRunner.run(new ExampleDriver(),args);
System.exit(exitCode);
}
}
第二套:
为新的API,此时Maper和Reducer为虚类,且该接口均在org.apache.hadoop.mapreduce.*;中
Mapper:
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Mapper.Context;
public class ExampleMapper extends Mapper<LongWritable,Text,Text,IntWritable>
{
public void map(LongWritable key,Text value,Context context) throwsIOException,InterruptedException
{
..........
}
}
Reducer:
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Reducer.Context;
class ExampleReducer extends Reducer<Text ,Text,Text,Text>
{
public void reduce(Text key,Iterable<Text> values,Context context) throwsIOException,InterruptedException
{
..........
}
}
Driver:
public class ExampleDriver{
public static void main(String args[]) throws Exception{
Job job=new Job();
job.setJarByClass(ExampleDriver.class);
............
job.setMapperClass();
.......
a98c
......
System.exit(job.waitForCompletion(true)?0:1);
}
}
或者
public class ExampleDriver extends Configured implements Tool
{
public int run(String args[])throws Exception
{
Configuration conf=getConf();
Job job=new Job(conf,"Example"); //任务名
job.setJarByClass(ExampleDriver.class); //指定class
............
job.waitForCompletion(true);
return job.isSuccessful()?0:1;
}
public static void main(String args[]) throws Exception
{
int res=ToolRunner.run(new Configuration(),new ExampleDriver(),args);
System.exit(res);
}
}
相关文章推荐
- 027_编写MapReduce的模板类Mapper、Reducer和Driver
- 关于程序中Activity若干问题知识点总结
- 关于编译Qt以及驱动的一点总结吧
- 027_编写MapReduce的模板类Mapper、Reducer和Driver
- android关于类在序列化以及AIDL中的若干总结点
- 关于在IDEA上进行聚合工程以及SSM框架搭建的总结
- Hadoop 调试第一个mapreduce程序过程详细记录总结以及权限问题 Permission denied: user=dr.who
- 关于编译Qt以及驱动的一点总结吧 - Rollen Holt - 博客园
- 【ssh2学习记录】3、关于ssh2框架整合以及hibernate的一些总结
- 关于platform设备驱动框架的总结
- 关于微信小程序循环以及嵌套循环的总结
- 关于写程序时「对数据库操作」的一些总结。
- 关于jsp网页以及web.xml文件配置总结
- javaweb学习总结一(eclipse常用快捷键、debug调试以及junit测试框架)
- 关于QT——按键驱动以及温度传感的结合
- 数据库分库分表(sharding)系列(三) 关于使用框架还是自主开发以及sharding实现层面的考量(不太懂)
- 2017年11月21日学习总结__关于函数的定义的说明以及递归函数
- 过滤驱动实现目录重定向之(完整版本程序下载以及使用)
- 关于基于xfire webservice框架开发webservice的总结
- 《HBase in Action》 第三章节的学习总结 ---- 如何编写和运行基于HBase的MapReduce程序