Hadoop之Mapreduce------>入门级程序WordCount代码编写
2016-12-07 21:50
519 查看
一 、Mapper编写
继承Mapper类------>重写map方法------>实现具体业务逻辑------>将新的key,value输出
public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//接收数据V1
String line = value.toString();
//按照空格切分成单词
String[] words = line.split(" ");
for (String w : words) {
//出现一次记1
context.write(new Text(w), new LongWritable(1));
}
}
}
一 、Reducer编写
继承Reducer类------>重写reduce方法------>实现具体业务逻辑------>将新的key,value输出
public class WCReducer extends Reducer<Text,LongWritable,Text,LongWritable> {
@Override
protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
//接受数据
// Text k3 = key;
//定义一个计数器
long counter = 0;
//遍历values集合,并求和
for(LongWritable i:values){
counter += i.get();
}
//将新的key,value输出
context.write(key,new LongWritable(counter));
}
}
三 、主函数编写
public class WordCount {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
//构建job类
Job job = Job.getInstance(new Configuration());
//注意main方法所在的类,别忘记设置setJarByClass
job.setJarByClass(WordCount.class);
//设置mapper相关属性
job.setMapperClass(WCMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
//设置源文件
FileInputFormat.setInputPaths(job,new Path("/wordcount"));
//设置Rueduce相关属性
job.setReducerClass(WCReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
//设置输出文件
FileOutputFormat.setOutputPath(job,new Path("/wcout126"));
//提交任务
job.waitForCompletion(true);
}
}
继承Mapper类------>重写map方法------>实现具体业务逻辑------>将新的key,value输出
public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//接收数据V1
String line = value.toString();
//按照空格切分成单词
String[] words = line.split(" ");
for (String w : words) {
//出现一次记1
context.write(new Text(w), new LongWritable(1));
}
}
}
一 、Reducer编写
继承Reducer类------>重写reduce方法------>实现具体业务逻辑------>将新的key,value输出
public class WCReducer extends Reducer<Text,LongWritable,Text,LongWritable> {
@Override
protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
//接受数据
// Text k3 = key;
//定义一个计数器
long counter = 0;
//遍历values集合,并求和
for(LongWritable i:values){
counter += i.get();
}
//将新的key,value输出
context.write(key,new LongWritable(counter));
}
}
三 、主函数编写
public class WordCount {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
//构建job类
Job job = Job.getInstance(new Configuration());
//注意main方法所在的类,别忘记设置setJarByClass
job.setJarByClass(WordCount.class);
//设置mapper相关属性
job.setMapperClass(WCMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
//设置源文件
FileInputFormat.setInputPaths(job,new Path("/wordcount"));
//设置Rueduce相关属性
job.setReducerClass(WCReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
//设置输出文件
FileOutputFormat.setOutputPath(job,new Path("/wcout126"));
//提交任务
job.waitForCompletion(true);
}
}
相关文章推荐
- Hadoop之Mapreduce------>入门级程序WordCount原理
- MapReduce编写wordcount程序代码实现
- hadoop mapreduce wordcount编写
- Hadoop 1.x 编写自己的WordCount程序
- 第一个MapReduce程序----wordcount(编写并运行)
- 【学习笔记】用Hadoop在MapReduce中WordCount简单程序运行详细流程
- hadoop之WordCount代码编写
- hadoop学习之HDFS(2.1):linux下eclipse中配置hadoop-mapreduce开发环境并运行WordCount.java程序
- Hadoop 6、第一个mapreduce程序 WordCount
- Ubuntu14.04中eclipse下编写mapreduce例子程序WordCount
- HADOOP 分布式集群环境下第一个mapReduce程序—WordCount
- 用Python编写MapReduce的WordCount实例程序
- hadoop学习笔记(三)mapreduce程序wordcount
- win7下安装hadoop 2.6.0 的eclipse插件并编写运行WordCount程序
- hadoop程序MapReduce之WordCount
- Hadoop之WordCount计数器程序编写并打包
- 配置Hadoop2.x的HDFS、MapReduce来运行WordCount程序
- Hadoop 6、第一个mapreduce程序 WordCount