filebeat+logstash对springcloud项目日志进行解析

项目地址：https://gitee.com/lwydyby/springcloud-adplatform

该项目的日志目前可以分为三种：

(1)项目启动日志

(2)controller层的操作日志（通过aop实现记录）

(3)项目非controller层异常日志

由于分为3种格式，则需要三种macth进行匹配：

grok {
match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type}  com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}",
"message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)",
"message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type}  %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]
}

使用logstash自带表达式的格式一般为：%{SYNTAX:SEMANTIC}

即： SYNTAX

代表匹配值的类型，例如，

0.11

可以

NUMBER

类型所匹配，

10.222.22.25

可以使用

IP

匹配。

*

SEMANTIC

表示存储该值的一个变量声明，

client_ip_address`这个字段里边，类似数据库的列名

具体可参考：http://blog.csdn.net/liukuan73/article/details/52318243

而自定义匹配需要使用正则表达式，格式为

( ?<列名>正则表达式)

注：[\s\S]*为匹配剩余所有的字符，我这里是用来匹配错误日志的详细信息的

由于controller层日志是以json格式进行的存储，除使用grok之外还需使用 json进行解析

json {
source => "log_json"
target => "content"
remove_field=>["logjson"]
}

完整配置文件为：

input {
beats {
port => 5044
}
}
filter{
grok {
match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type}  com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}",
"message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)",
"message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type}  %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]
}
json {
source => "log_json"
target => "content"
remove_field=>["logjson"]
}
}
output {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "%{[fields][logIndex]}-%{+YYYY.MM.dd}"
document_type => "%{[fields][docType]}"
}
}

除需要配置logstash之外，还需要对filebeat进行配置，以设置如何发送日志（处理多行日志）：

filebeat:
prospectors:
-
paths:
- "/data/logs/*.log"
fields:
logIndex: userservice
docType: springcloud
project: adplatform
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
timeout: 5s
output:
logstash:
hosts: ["localhost:5044"]

日志文件样例：

2018-02-22 16:09:59.876 [main] INFO  o.s.integration.channel.PublishSubscribeChannel - Channel 'user-service:dev:8763.errorChannel' has 1 subscriber(s).
2018-02-22 16:09:59.876 [main] INFO  o.s.integration.endpoint.EventDrivenConsumer - started _org.springframework.integration.errorLogger
2018-02-22 16:09:59.913 [main] INFO  o.s.i.endpoint.SourcePollingChannelAdapter - started sleuthStreamSpanReporter.poll.inboundChannelAdapter
2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147482647
2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147483647
2018-02-22 16:09:59.966 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8763"]
2018-02-22 16:09:59.991 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8763"]
2018-02-22 16:09:59.992 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2018-02-22 16:10:00.081 [main] INFO  o.s.b.c.e.tomcat.TomcatEmbeddedServletContainer - Tomcat started on port(s): 8763 (http)
2018-02-22 16:10:00.082 [main] INFO  o.s.c.n.e.s.EurekaAutoServiceRegistration - Updating port to 8763
2018-02-22 16:10:00.088 [main] INFO  com.gameley.UserServiceApplication - Started UserServiceApplication in 281.136 seconds (JVM running for 285.806)
2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  o.a.c.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization started
2018-02-22 16:10:13.301 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization completed in 67 ms
2018-02-22 16:12:26.850 [http-nio-8763-exec-10] INFO  com.gameley.common.Interceptor.HttpAspect - {"address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":1041}
2018-02-22 16:12:27.971 [task-scheduler-5] INFO  o.s.integration.codec.kryo.CompositeKryoRegistrar - registering [40, java.io.File] with serializer org.springframework.integration.codec.kryo.FileSerializer
2018-02-22 16:12:28.276 [http-nio-8763-exec-9] INFO  com.gameley.common.Interceptor.HttpAspect - {"address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":7}