利用Spark解析Tomcat日志，并将统计结果存入Mysql数据库

原文：http://www.voidcn.com/blog/Netbug_NB/article/p-5992917.html



本文试图实现的需求场景为：以学习Spark知识点为目的，编写Scala利用Spark解析800M的tomcat日志文件，打印一段时间内ERROR级别记录的前10行，统计每分钟的日志记录数，并将统计结果存入mysql数据库中。之前曾用JAVA写过一次同样的处理逻辑，但在学习了Scala之后，真的感觉在计算方面Scala要比JAVA方便的多。没有学习Scala语言的同学速度速度了啊……

技术要点

将日志文件写入HDFS中，相对路径PATH为“nova.log”

注意JAVA堆栈异常日志的处理

将解析后的异常日志全部存到SparkSQL中或Hive数据仓库中

通过编写SQL查询一段时间内ERROR级别记录的前10行

统计每分钟的日志记录数，并将统计结果存入mysql数据库中,便于上层应用直接使用计算结果

解析前后对比

解析前：


解析后：

解析代码

LoggerApp.scala：

<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> java.text.SimpleDateFormat
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> java.util.Date

<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.SparkConf
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.SparkContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.rdd.RDD
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.rdd.RDD.rddToPairRDDFunctions
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.Row
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.SQLContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.hive.HiveContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StringType
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StructField
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StructType
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.TimestampType

/**
* 日志解析
*/
object LoggerApp {
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">main</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(args: Array[String])</span>:</span> Unit = {
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"<!--开始解析-->"</span>)
val reg = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"^(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}) (\\[.*\\]) (.*) (.*) - ([\\s\\S]*)$"</span>
val path = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"nova.log"</span>
val sc = new SparkContext(new SparkConf().setAppName(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"日志解析"</span>))
val textRDD = sc.textFile(path)

/**
* 处理一条日志包括多行的情况
*/
var key = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>
val formatRDD = textRDD.map { x =>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">if</span> (x.matches(reg)) {
key = x
Pair.apply(key, <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>)
} <span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">else</span> {
Pair.apply(key, x)
}
}.reduceByKey((a, b) => { a + <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"\n"</span> + b }).map(x => x._1 + x._2)

/**
* 将字符串转换为Logger
*/
val loggerRDD: RDD[Logger] = formatRDD.map { x =>
{
val reg.r(time, thread, level, logger, msg) = x //通过正则取值
val log = new Logger(formatDate(time), thread, level, logger, msg)
log
}
}.cache()

/**
* TODO 通过类的反射机制来定义数据库Scheme，但在scala语言中不知道为啥就是不成功，此处浪费了许久留着以后研究吧
*/
/*val sqlc = new SQLContext(sc)
sqlc.createDataFrame(loggerRDD, classOf[Logger]).registerTempTable(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"logger"</span>)*/

/**
* 定义数据库Scheme
*/
val schemaString = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"time thread level logger msg"</span>
val schema =
StructType(
schemaString.split(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">" "</span>).map(fieldName =>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">if</span> (<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"time"</span>.equals(fieldName))
StructField(fieldName, TimestampType, true)
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">else</span>
StructField(fieldName, StringType, true)))
/**
* 将Logger转换为Row
*/
val rowRDD = loggerRDD.map { log =>
Row(
formatDate(log.time),
log.thread,
log.level,
log.logger,
log.msg)
}
/**
* 利用SQL进行查询过滤
*/
//    val sqlc = bySQLContext(sc, rowRDD, schema);
val sqlc = byHiveContext(sc, rowRDD, schema);
val df = sqlc.sql(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"select * from logger where level='ERROR' and time between '2016-03-21 11:00:00' and '2016-03-21 12:00:00' order by time"</span>)
val errLogRDD = df.map { x =>
new Logger(
formatDate(x.getTimestamp(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">0</span>)),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">1</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transpa
11170
rent;">2</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">3</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">4</span>))
}
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">for</span> (log <- errLogRDD.take(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">10</span>)) {
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"time:"</span> + formatDateToStr(log.time))
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"thread:"</span> + log.thread)
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"level:"</span> + log.level)
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"logger:"</span> + log.logger)
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"msg:"</span> + log.msg)
}
println(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"<!--解析结束-->"</span>)
}
/**
* 创建临时表
*/
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">bySQLContext</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(sc: SparkContext, rowRDD: RDD[Row], schema: StructType)</span>:</span> SQLContext = {
val sqlc = new SQLContext(sc)
sqlc.createDataFrame(rowRDD, schema).registerTempTable(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"logger"</span>)
sqlc
}
/**
* 创建永久表，需要提前搭建好Spark与Hive的集成环境
*/
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">byHiveContext</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(sc: SparkContext, rowRDD: RDD[Row], schema: StructType)</span>:</span> SQLContext = {
val sqlc = new HiveContext(sc)
sqlc.sql(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"drop table if exists logger"</span>)
sqlc.sql(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"CREATE TABLE IF NOT EXISTS logger (time TIMESTAMP, thread STRING, level STRING, logger STRING, msg STRING)"</span>)
sqlc.createDataFrame(rowRDD, schema).write.mode(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"overwrite"</span>).saveAsTable(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"logger"</span>)
sqlc
}
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">formatDate</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(str: String)</span>:</span> Date = {
new SimpleDateFormat(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"yyyy-MM-dd HH:mm:ss.SSS"</span>).parse(str)
}
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">formatDate</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(timestamp: java.sql.Timestamp)</span>:</span> Date = {
new SimpleDateFormat(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"yyyy-MM-dd HH:mm:ss.SSS"</span>).parse(timestamp.toString())
}
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">formatDate</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(date: Date)</span>:</span> java.sql.Timestamp = {
new java.sql.Timestamp(date.getTime)
}
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">formatDateToStr</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(date: Date)</span>:</span> String = {
new SimpleDateFormat(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"yyyy-MM-dd HH:mm:ss.SSS"</span>).format(date)
}
}

Logger.scala：

import java.util.Date

<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">class</span> Logger extends Serializable {
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">var</span> time: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">Date</span> = <span class="hljs-literal" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">null</span>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">var</span> thread: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span> = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">var</span> level: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span> = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">var</span> logger: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span> = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">var</span> msg: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span> = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">""</span>
def <span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>(time: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">Date</span>, thread: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span>, level: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span>, logger: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span>, msg: <span class="hljs-built_in" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 134, 179); background: transparent;">String</span>) {
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>()
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>.time = time;
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>.thread = thread;
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>.level = level;
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>.logger = logger;
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">this</span>.msg = msg;
}
}

统计并写入Mysql

LoggerMysqlApp.scala：

<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.SparkContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.SparkConf
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.hive.HiveContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.SQLContext
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> java.util.Date
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> java.text.SimpleDateFormat
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StringType
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StructType
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.StructField
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.types.IntegerType
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> org.apache.spark.sql.Row
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">import</span> java.util.Properties

object LoggerMysqlApp {
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">main</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(args: Array[String])</span>:</span> Unit = {
val sc = new SparkContext(new SparkConf().setAppName(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"输出写入Mysql"</span>))
/**
* 从hive中加载数据
*/
val hivec = new HiveContext(sc)
val df = hivec.sql(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"select * from logger"</span>)
val loggerRDD = df.rdd.map { x =>
new Logger(
LoggerApp.formatDate(x.getTimestamp(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">0</span>)),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">1</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">2</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">3</span>),
x.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">4</span>))
}
val resultRDD = loggerRDD.map { logger =>
Pair(formatDateToStr(logger.time), <span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">1</span>)
}.reduceByKey((a, b) =>
{ a + b }).map(f =>
Row(f._1, f._2)).sortBy(f => f.getInt(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">1</span>), false, <span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">2</span>)
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">for</span> (r <- resultRDD.take(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">10</span>)) {
println(r.getString(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">0</span>) + <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">":"</span> + r.getInt(<span class="hljs-number" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(64, 160, 112); background: transparent;">1</span>))
}
/**
* 定义数据库Scheme
*/
val schemaString = <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"time count"</span>
val schema =
StructType(
schemaString.split(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">" "</span>).map(fieldName =>
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">if</span> (<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"time"</span>.equals(fieldName))
StructField(fieldName, StringType, true)
<span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">else</span>
StructField(fieldName, IntegerType, true)))
/**
* TODO计算每分钟日志的个数
*/
val connectionProperties = new Properties()
connectionProperties.setProperty(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"user"</span>, <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"root"</span>)
connectionProperties.setProperty(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"password"</span>, <span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"."</span>)
new SQLContext(sc).createDataFrame(resultRDD, schema).write.jdbc(
<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"jdbc:mysql://192.168.136.128:3306/logger"</span>,
<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"logger"</span>,
connectionProperties);
}
<span class="hljs-function" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; background: transparent;"><span class="hljs-keyword" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(149, 65, 33); background: transparent;">def</span> <span class="hljs-title" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(25, 70, 157); background: transparent;">formatDateToStr</span><span class="hljs-params" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(0, 0, 255); background: transparent;">(date: Date)</span>:</span> String = {
new SimpleDateFormat(<span class="hljs-string" style="border: 0px; margin: 0px; padding: 0px; font-weight: inherit; font-style: inherit; font-family: inherit; vertical-align: baseline; color: rgb(33, 145, 97); background: transparent;">"yyyy-MM-dd HH:mm"</span>).format(date)
}
}