Linux巩固记录(5) hadoop 2.7.4下自己编译代码并运行MapReduce程序
2017-09-02 17:27
531 查看
程序代码为 ~\hadoop-2.7.4\share\hadoop\mapreduce\sources\hadoop-mapreduce-examples-2.7.4-sources\org\apache\hadoop\examples\WordCount.java
第一次 删除了package
View Code
为啥要删除package,就是因为有包路径的时候 调用方式就要 xxx.xxxxx.xxx来执行,而且打包的时候就不能只打class了,目录结构也要一并打进去
同理,自己写的代码也可按照这个方式执行
顺便提一点,如果只是打jar包 用
但是如果要修改MANIFEST.MF,在里面指定mainClass,按照如下方式
这样就可以直接用 java -jar test.jar 运行了,后面不用跟具体的类
第一次 删除了package
package org.apache.hadoop.examples; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length < 2) { System.err.println("Usage: wordcount <in> [<in>...] <out>"); System.exit(2); } Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); for (int i = 0; i < otherArgs.length - 1; ++i) { FileInputFormat.addInputPath(job, new Path(otherArgs[i])); } FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
View Code
[root@master classes]# [root@master classes]# tree . 0 directories, 0 files [root@master classes]# javac -classpath .:/home/jars/* -d /home/classes/ /home/javaFile/WordCount.java [root@master classes]# tree . └── org └── apache └── hadoop └── examples ├── WordCount.class ├── WordCount$IntSumReducer.class └── WordCount$TokenizerMapper.class 4 directories, 3 files [root@master classes]# jar -cvf wordcount.jar ./* added manifest adding: org/(in = 0) (out= 0)(stored 0%) adding: org/apache/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/examples/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out= 764)(deflated 57%) adding: org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out= 749)(deflated 58%) adding: org/apache/hadoop/examples/WordCount.class(in = 1988) (out= 1050)(deflated 47%) [root@master classes]# /home/hadoop-2.7.4/bin/hadoop jar /home/classes/wordcount.jar org.apache.hadoop.examples.WordCount /hdfs-input.txt /result-package 17/09/02 02:20:41 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.80:8032 17/09/02 02:20:43 INFO input.FileInputFormat: Total input paths to process : 1 17/09/02 02:20:43 INFO mapreduce.JobSubmitter: number of splits:1 17/09/02 02:20:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504320356950_0011 17/09/02 02:20:43 INFO impl.YarnClientImpl: Submitted application application_1504320356950_0011 17/09/02 02:20:43 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504320356950_0011/ 17/09/02 02:20:43 INFO mapreduce.Job: Running job: job_1504320356950_0011 17/09/02 02:20:51 INFO mapreduce.Job: Job job_1504320356950_0011 running in uber mode : false 17/09/02 02:20:51 INFO mapreduce.Job: map 0% reduce 0% 17/09/02 02:20:58 INFO mapreduce.Job: map 100% reduce 0% 17/09/02 02:21:05 INFO mapreduce.Job: map 100% reduce 100% 17/09/02 02:21:06 INFO mapreduce.Job: Job job_1504320356950_0011 completed successfully 17/09/02 02:21:06 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=118 FILE: Number of bytes written=241857 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=174 HDFS: Number of bytes written=76 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=3828 Total time spent by all reduces in occupied slots (ms)=4312 Total time spent by all map tasks (ms)=3828 Total time spent by all reduce tasks (ms)=4312 Total vcore-milliseconds taken by all map tasks=3828 Total vcore-milliseconds taken by all reduce tasks=4312 Total megabyte-milliseconds taken by all map tasks=3919872 Total megabyte-milliseconds taken by all reduce tasks=4415488 Map-Reduce Framework Map input records=6 Map output records=12 Map output bytes=118 Map output materialized bytes=118 Input split bytes=98 Combine input records=12 Combine output records=9 Reduce input groups=9 Reduce shuffle bytes=118 Reduce input records=9 Reduce output records=9 Spilled Records=18 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=186 CPU time spent (ms)=1200 Physical memory (bytes) snapshot=297316352 Virtual memory (bytes) snapshot=4159815680 Total committed heap usage (bytes)=139595776 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=76 File Output Format Counters Bytes Written=76 [root@master classes]# /home/hadoop-2.7.4/bin/hadoop fs -ls / Found 4 items -rw-r--r-- 2 root supergroup 76 2017-09-02 00:57 /hdfs-input.txt drwxr-xr-x - root supergroup 0 2017-09-02 02:21 /result-package drwxr-xr-x - root supergroup 0 2017-09-02 02:12 /result-self-compile drwx------ - root supergroup 0 2017-09-02 02:11 /tmp [root@master classes]# [root@master classes]#
为啥要删除package,就是因为有包路径的时候 调用方式就要 xxx.xxxxx.xxx来执行,而且打包的时候就不能只打class了,目录结构也要一并打进去
同理,自己写的代码也可按照这个方式执行
顺便提一点,如果只是打jar包 用
jar -cvf test.jar XXX.class
但是如果要修改MANIFEST.MF,在里面指定mainClass,按照如下方式
#解压文件 jar -xf test.jar #在MANIFEST.MF 增加mainclass Manifest-Version: 1.0 Created-By: 1.6.0_20 (Sun Microsystems Inc.) Main-class: WordCount #再打包 jar -cvfm test.jar MANIFEST.MF XXXX.class
这样就可以直接用 java -jar test.jar 运行了,后面不用跟具体的类
相关文章推荐
- 使用命令行编译打包运行自己的MapReduce程序 Hadoop2.4.1
- 使用命令行编译打包运行自己的MapReduce程序 Hadoop2.7.2
- 使用命令行编译打包运行自己的MapReduce程序 hadoop2.7.3
- 使用命令行编译打包运行自己的MapReduce程序 Hadoop2.4.1
- 使用命令行编译打包运行自己的MapReduce程序 Hadoop2.4.1
- 使用命令行编译打包运行自己的MapReduce程序 Hadoop2.6.0
- Windows下使用eclipse编译打包运行自己的MapReduce程序 Hadoop2.6.0
- Linux巩固记录(4) 运行hadoop 2.7.4自带demo程序验证环境
- Ubuntu系统下的Hadoop集群(2)_使用命令行编译打包运行自己的MapReduce程序
- Hadoop学习全程记录——在Eclipse中运行第一个MapReduce程序
- Hadoop MapReduce示例程序WordCount.java手动编译运行解析
- hadoop学习之HDFS(2.1):linux下eclipse中配置hadoop-mapreduce开发环境并运行WordCount.java程序
- gcc编译静态库到自己的程序 解决在不同linux下因libc版本问题而不能运行
- 【图文详细】使用Eclipse编译运行MapReduce程序_Hadoop_2.4.1
- Ubuntu14.0.4下,shell编译-打包-运行Hadoop2.7.2的MapReduce程序
- Hadoop MapReduce示例程序WordCount.java手动编译运行解析
- Hadoop学习全程记录——在Eclipse中运行第一个MapReduce程序
- 使用Eclipse编译运行MapReduce程序 Hadoop2.4.1
- Hadoop学习全程记录——在Eclipse中运行第一个MapReduce程序
- Ubuntu系统下的Hadoop集群(4)_使用Eclipse编译运行MapReduce程序