eclipse hadoop远程调试
2015-07-31 21:39
375 查看
一、插件安装
我用的hadoop-2.6.0,插件下载地址下载后扔到eclipse/dropins目录下即可,当然eclipse/plugins也是可以的,前者更为轻便,推荐;重启Eclipse,即可在透视图(Perspective)中看到Map/Reduce
二、配置
点击蓝色的小象图标,新建一个Hadoop连接:注意,一定要填写正确,修改了某些端口,以及默认运行的用户名等
正常情况下,可以在项目区域可以看到
这样可以正常的进行HDFS分布式文件系统的管理:上传,删除等操作。
为下面测试做准备,需要先建了一个目录 user/root/input,然后上传两个txt文件到此目录:
三、Hadoop工程
新建一个Map/Reduce Project工程,设定好本地的hadoop目录新建一个测试类HadoopTest:
import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; import org.apache.log4j.Logger; public class HadoopTest{ private static final Logger log = Logger.getLogger(HadoopTest.class); public static class Mapper1 extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { log.info("Map key : " + key); log.info("Map value : " + value); StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { String wordStr = itr.nextToken(); word.set(wordStr); log.info("Map word : " + wordStr); context.write(word, one); } } } public static class Reduce1 extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { log.info("Reduce key : " + key); log.info("Reduce value : " + values); int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); log.info("Reduce sum : " + sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args) .getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: WordCountTest <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(HadoopTest.class); job.setMapperClass(Mapper1.class); job.setCombinerClass(Reducer1.class); job.setReducerClass(Reducer1.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
右键,选择“Run Configurations”,弹出窗口,点击“Arguments”选项卡,在“Program argumetns”处预先输入参数:
备注:参数为了在本地调试使用,而非真实环境。
然后,点击“Apply”,然后“Close”。现在可以右键,选择“Run on Hadoop”,运行。
成功之后,在Eclipse下刷新HDFS目录,可以看到生成了ouput目录:
相关文章推荐
- 我是菜鸟:java内存及内存溢出异常
- Spring--quartz中cronExpression配置说明
- [转]基于全注解的Spring3.1 mvc、myBatis3.1、Mysql的轻量级项目
- EJB(企业级JavaBean)之Session Bean
- java concurrent并发详解
- java进阶第一篇
- java List在arrayList()括号里初始化List
- [转]Spring 注解总结
- junit测试spring相关配置
- 【Struts2框架】(一)原理分析
- Java基础之详细理解回收机制
- JAVA国际化程序实现
- SSH学习八 Spring框架 -- 控制反转实现Ioc
- java重载方法重写、继承封装
- Java反序列化测试
- Java大数应用
- Exponentiation(java 大实数)
- java类加载全过程
- Spring是什么?
- spring与hibernate整合(三种方式)