Hadoop经典案例Spark实现(七)——日志分析:分析非结构化文件
2016-05-17 23:09
701 查看
相关文章推荐
Hadoop经典案例Spark实现(一)——通过采集的气象数据分析每年的最高温度
Hadoop经典案例Spark实现(二)——数据去重问题
Hadoop经典案例Spark实现(三)——数据排序
Hadoop经典案例Spark实现(四)——平均成绩
Hadoop经典案例Spark实现(五)——求最大最小值问题
Hadoop经典案例Spark实现(六)——求最大的K个值并排序
Hadoop经典案例Spark实现(七)——日志分析:分析非结构化文件
Hadoop经典案例Spark实现(七)——日志分析:分析非结构化文件
1、需求:根据tomcat日志计算url访问了情况,具体的url如下,
要求:区别统计GET和POST URL访问量
结果为:访问方式、URL、访问量
测试数据集:
由于Tomcat日志是不规则的,需要先过滤清洗数据。
2、Hadoop之MapReduce实现:
Map类
Reduce类
启动类
3、Spark实现之Scala版本
//textFile() 加载数据
val data = sc.textFile("/spark/seven.txt")
//filter 过滤长度小于0, 过滤不包含GET与POST的URL
val filtered = data.filter(_.length()>0).filter( line => (line.indexOf("GET")>0 || line.indexOf("POST")>0) )
//转换成键值对操作
val res = filtered.map( line => {
if(line.indexOf("GET")>0){ //截取 GET 到URL的字符串
(line.substring(line.indexOf("GET"),line.indexOf("HTTP/1.0")).trim,1)
}else{ //截取 POST 到URL的字符串
(line.substring(line.indexOf("POST"),line.indexOf("HTTP/1.0")).trim,1)
}//最后通过reduceByKey求sum
}).reduceByKey(_+_)
//触发action事件执行
res.collect()
Scala函数式编程的代码简洁且优雅,在JDK1.8之后的也会有类似的新特性。
对比输出结果与MR是一致的
Hadoop经典案例Spark实现(一)——通过采集的气象数据分析每年的最高温度
Hadoop经典案例Spark实现(二)——数据去重问题
Hadoop经典案例Spark实现(三)——数据排序
Hadoop经典案例Spark实现(四)——平均成绩
Hadoop经典案例Spark实现(五)——求最大最小值问题
Hadoop经典案例Spark实现(六)——求最大的K个值并排序
Hadoop经典案例Spark实现(七)——日志分析:分析非结构化文件
Hadoop经典案例Spark实现(七)——日志分析:分析非结构化文件
1、需求:根据tomcat日志计算url访问了情况,具体的url如下,
要求:区别统计GET和POST URL访问量
结果为:访问方式、URL、访问量
测试数据集:
196.168.2.1 - - [03/Jul/2014:23:36:38 +0800] "GET /course/detail/3.htm HTTP/1.0" 200 38435 0.038 182.131.89.195 - - [03/Jul/2014:23:37:43 +0800] "GET /html/notes/20140617/888.html HTTP/1.0" 301 - 0.000 196.168.2.1 - - [03/Jul/2014:23:38:27 +0800] "POST /service/notes/addViewTimes_23.htm HTTP/1.0" 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:39:03 +0800] "GET /html/notes/20140617/779.html HTTP/1.0" 200 69539 0.046 196.168.2.1 - - [03/Jul/2014:23:43:00 +0800] "GET /html/notes/20140318/24.html HTTP/1.0" 200 67171 0.049 196.168.2.1 - - [03/Jul/2014:23:43:59 +0800] "POST /service/notes/addViewTimes_779.htm HTTP/1.0" 200 1 0.003 196.168.2.1 - - [03/Jul/2014:23:45:51 +0800] "GET /html/notes/20140617/888.html HTTP/1.0" 200 70044 0.060 196.168.2.1 - - [03/Jul/2014:23:46:17 +0800] "GET /course/list/73.htm HTTP/1.0" 200 12125 0.010 196.168.2.1 - - [03/Jul/2014:23:46:58 +0800] "GET /html/notes/20140609/542.html HTTP/1.0" 200 94971 0.077 196.168.2.1 - - [03/Jul/2014:23:48:31 +0800] "POST /service/notes/addViewTimes_24.htm HTTP/1.0" 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:48:34 +0800] "POST /service/notes/addViewTimes_542.htm HTTP/1.0" 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:49:31 +0800] "GET /notes/index-top-3.htm HTTP/1.0" 200 53494 0.041 196.168.2.1 - - [03/Jul/2014:23:50:55 +0800] "GET /html/notes/20140609/544.html HTTP/1.0" 200 183694 0.076 196.168.2.1 - - [03/Jul/2014:23:53:32 +0800] "POST /service/notes/addViewTimes_544.htm HTTP/1.0" 200 2 0.004 196.168.2.1 - - [03/Jul/2014:23:54:53 +0800] "GET /service/notes/addViewTimes_900.htm HTTP/1.0" 200 151770 0.054 196.168.2.1 - - [03/Jul/2014:23:57:42 +0800] "GET /html/notes/20140620/872.html HTTP/1.0" 200 52373 0.034 196.168.2.1 - - [03/Jul/2014:23:58:17 +0800] "POST /service/notes/addViewTimes_900.htm HTTP/1.0" 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:58:51 +0800] "GET /html/notes/20140617/888.html HTTP/1.0" 200 70044 0.057 186.76.76.76 - - [03/Jul/2014:23:48:34 +0800] "POST /service/notes/addViewTimes_542.htm HTTP/1.0" 200 2 0.003 186.76.76.76 - - [03/Jul/2014:23:46:17 +0800] "GET /course/list/73.htm HTTP/1.0" 200 12125 0.010 8.8.8.8 - - [03/Jul/2014:23:46:58 +0800] "GET /html/notes/20140609/542.html HTTP/1.0" 200 94971 0.077
由于Tomcat日志是不规则的,需要先过滤清洗数据。
2、Hadoop之MapReduce实现:
Map类
import java.io.IOException; import javax.naming.spi.DirStateFactory.Result; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class LogMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private IntWritable val = new IntWritable(1); @Override protected void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException { String line = value.toString().trim(); String tmp = handlerLog(line); if(tmp.length()>0){ context.write(new Text(tmp), val); } } //127.0.0.1 - - [03/Jul/2014:23:36:38 +0800] "GET /course/detail/3.htm HTTP/1.0" 200 38435 0.038 private String handlerLog(String line){ String result = ""; try{ if(line.length()>20){ if(line.indexOf("GET")>0){ result = line.substring(line.indexOf("GET"), line.indexOf("HTTP/1.0")).trim(); }else if(line.indexOf("POST")>0){ result = line.substring(line.indexOf("POST"), line.indexOf("HTTP/1.0")).trim(); } } }catch (Exception e) { System.out.println(line); } return result; } public static void main(String[] args){ String line = "127.0.0.1 - - [03/Jul/2014:23:36:38 +0800] \"GET /course/detail/3.htm HTTP/1.0\" 200 38435 0.038"; System.out.println(new LogMapper().handlerLog(line)); } }
Reduce类
import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class LogReducer extends Reducer<Text, IntWritable, Text, IntWritable> { @Override protected void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable val : values){ sum += val.get(); } context.write(key, new IntWritable(sum)); } }
启动类
import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class JobMain { /** * @param args */ public static void main(String[] args)throws Exception { Configuration configuration = new Configuration(); Job job = new Job(configuration,"log_job"); job.setJarByClass(JobMain.class); job.setMapperClass(LogMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setReducerClass(LogReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); Path path = new Path(args[1]); FileSystem fs = FileSystem.get(configuration); if(fs.exists(path)){ fs.delete(path, true); } FileOutputFormat.setOutputPath(job, path); System.exit(job.waitForCompletion(true)?0:1); } }
3、Spark实现之Scala版本
//textFile() 加载数据
val data = sc.textFile("/spark/seven.txt")
//filter 过滤长度小于0, 过滤不包含GET与POST的URL
val filtered = data.filter(_.length()>0).filter( line => (line.indexOf("GET")>0 || line.indexOf("POST")>0) )
//转换成键值对操作
val res = filtered.map( line => {
if(line.indexOf("GET")>0){ //截取 GET 到URL的字符串
(line.substring(line.indexOf("GET"),line.indexOf("HTTP/1.0")).trim,1)
}else{ //截取 POST 到URL的字符串
(line.substring(line.indexOf("POST"),line.indexOf("HTTP/1.0")).trim,1)
}//最后通过reduceByKey求sum
}).reduceByKey(_+_)
//触发action事件执行
res.collect()
Scala函数式编程的代码简洁且优雅,在JDK1.8之后的也会有类似的新特性。
对比输出结果与MR是一致的
(POST /service/notes/addViewTimes_779.htm,1), (GET /service/notes/addViewTimes_900.htm,1), (POST /service/notes/addViewTimes_900.htm,1), (GET /notes/index-top-3.htm,1), (GET /html/notes/20140318/24.html,1), (GET /html/notes/20140609/544.html,1), (POST /service/notes/addViewTimes_542.htm,2), (POST /service/notes/addViewTimes_544.htm,1), (GET /html/notes/20140609/542.html,2), (POST /service/notes/addViewTimes_23.htm,1), (GET /html/notes/20140617/888.html,3), (POST /service/notes/addViewTimes_24.htm,1), (GET /course/detail/3.htm,1), (GET /course/list/73.htm,2), (GET /html/notes/20140617/779.html,1), (GET /html/notes/20140620/872.html,1)
相关文章推荐
- 实例讲解linux用户,组,文件目录权限
- LINUX SHELL控制结构while
- carson常用linux命令整理
- 谢欣伦 - OpenDev原创教程 - 媒体开发库libMedia
- linux shell编程学习--循环流程while,for,until命令
- 初进测试——培训
- virtualbox 动态增大虚拟硬盘大小的方法和步骤(linux VM)
- 阿里云CentOS6.5安装jdk1.7
- Linux下进程的创建(system(); fork(); exec*())
- OpenCV中感兴趣区域的选取与检测(二)
- Linux使用lvresize扩展或缩减lv大小
- 内核态使用infiniband
- basic file operations
- Linux Shell for循环写法总结
- Linux 的cp命令
- C/C++/linux推荐读物
- 【Android 第三方SDK】breakpad在linux下编译
- Nginx
- linux中KVM桥接网卡br0
- 清除linux系统的多余引导