Logstash收集json格式日志文件如何写配置文件

1、日志格式

{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}

我们如果收集这个日志只是做简单的配置。如下：

input {
file {

path => ["/home/elk/logstash-5.6.3/request"]
type => "chenxun"
}
}

output {

stdout {
codec => rubydebug
}

elasticsearch {
hosts => "192.168.2.181:9200"

}
}

那么收集到的结果是：

{
"_index": "logstash-2017.11.22",
"_type": "chenxun",
"_id": "AV_iTR0AM1H1mf2je0nC",
"_version": 1,
"_score": 1,
"_source": {
"@version": "1",
"host": "Ubuntu-20170424",
"path": "/home/elk/logstash-5.6.3/request",
"@timestamp": "2017-11-22T05:57:05.383Z",
"message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
"type": "chenxun"
}
}

即会将json记录做为一个字符串放到”message”下,这不是我们想要的结果，是让logstash自动解析json记录，将各字段放入elasticsearch中。下面介绍如何配置.

1.直接设置codec => json

input {
file {

path => ["/home/elk/logstash-5.6.3/request"]
type => "chenxun"
codec => json

}
}

这个时候看看结果： 已经把json解析到各个字段中去了

{
"_index": "logstash-2017.11.22",
"_type": "136587",
"_id": "AV_iXHbGM1H1mf2jfF4d",
"_version": 1,
"_score": 1,
"_source": {
"48": "136587",
"10018": 0,
"10070": 0,
"10071": 0,
"10190": 0,
"10191": 0,
"path": "/home/elk/logstash-5.6.3/request",
"@timestamp": "2017-11-22T06:13:51.361Z",
"@version": "1",
"host": "Ubuntu-20170424",
"type": "136587"
}
}

可以设置编码格式：（收集中文日志）

codec => json {
charset => "UTF-8"
}

2、使用filter json

配置如下：

input {
file {

path => ["/home/elk/logstash-5.6.3/request"]

}
}

filter {
json {
source => "message"
#target => "doc"
#remove_field => ["message"]
}
}

output {

stdout {
codec => rubydebug
}

elasticsearch {
hosts => "192.168.2.181:9200"

}
}

输入结果：

{
"_index": "logstash-2017.11.22",
"_type": "136587",
"_id": "AV_igupKM1H1mf2jfxm2",
"_version": 1,
"_score": 1,
"_source": {
"48": "136587",
"10018": 0,
"10070": 0,
"10071": 0,
"10190": 0,
"10191": 0,
"path": "/home/elk/logstash-5.6.3/request",
"@timestamp": "2017-11-22T06:55:51.335Z",
"@version": "1",
"host": "Ubuntu-20170424",
"message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
"type": "136587"
}
}

可以看到，原始记录被保存，同时字段也被解析保存。如果确认不需要保存原始记录内容，可以加设置：remove_field => [“message”]

其中特别需要注意解析json数据的内容，logstash会在向es插入数据时默认会在_source下增加type,host,path三个字段，如果json内容中本身也含有type,host,path字段，那么解析后将覆盖掉logstash默认的这三个字段，尤其是type字段，这个同时也是做为index/type用的，覆盖掉后，插入进es中的index/type就是json数据记录中的内容，将不再是logstash config中配置的type值。