ELK 5.0.1+Filebeat5.0.1实时监控MongoDB日志并使用正则解析mongodb日志

    关于ELK5.0.1的安装部署，请参考博文（ ELK 5.0.1+Filebeat5.0.1
for LINUX RHEL6.6 监控MongoDB日志），

本文重点说明如何适用filebeat实时监控mongodb数据库日志及在logstash正则解析mongodb日志。

    部署完ELK5.0.1后，在需要监控mongodb日志的数据库服务器上部署filebeat来抓取日志，

首先需要修改filebeat配置文件：

[root@se122 filebeat-5.0.1]# pwd

/opt/filebeat-5.0.1

[root@se122 filebeat-5.0.1]# 

[root@se122 filebeat-5.0.1]# ls

data  filebeat  filebeat.full.yml  filebeat.template-es2x.json  filebeat.template.json  filebeat.yml  scripts

[root@se122 filebeat-5.0.1]# cat filebeat.yml 

filebeat :

 prospectors :

  -

   paths :

       - /root/rs0-0.log   #filebeat负责实时监控的mongodb日志

   document_type : mongodblog  #指定filebeat发送到logstash的mongodb日志的文档类型为document_type，一定要指定（logstash接收解析匹配要使用）

   input_type : log

 registry_file : 

   /opt/filebeat-5.0.1/data/registry

output.logstash:

  hosts: ["10.117.194.228:5044"] #logstash服务部署的机器IP地址及运行的服务端口号

[root@se122 filebeat-5.0.1]# 

其次修改logstash配置文件：

[root@rhel6 config]# pwd

/opt/logstash-5.0.1/config

[root@rhel6 config]# cat logstash_mongodb.conf 

#input {

# stdin {}

#}

input{

  beats {

    host => "0.0.0.0"

    port => 5044

    type => mongodblog  #指定filebeat输入的日志类型是mongodblog

  }

}

filter {

  if [type] == "mongodblog" { #过滤器，只处理filebeat发送过来的mogodblog日志数据

    grok {  #解析发送过来的mognodblog日志

       match => ["message","%{TIMESTAMP_ISO8601:timestamp}\s+%{MONGO3_SEVERITY:severity}\s+%{MONGO3_COMPONENT:component}\s+(?:\[%{DATA:context}\])?\s+%{GREEDYDATA:body}"]

    }

    if [component] =~ "WRITE" {

      grok { #第二层解析body部分，提取mongodblog中的command_type、db_name、command、spend_time字段

        match => ["body","%{WORD:command_type}\s+%{DATA:db_name}\s+\w+\:\s+%{GREEDYDATA:command}%{INT:spend_time}ms$"]

      }

    } else {

        grok {

          match => ["body","\s+%{DATA:db_name}\s+\w+\:\s+%{WORD:command_type}\s+%{GREEDYDATA:command}protocol.*%{INT:spend_time}ms$"]

        }

    }

    date {

      match => [ "timestamp", "UNIX", "YYYY-MM-dd HH:mm:ss", "ISO8601"]

      remove_field => [ "timestamp" ]

    }

  }

}

output{

elasticsearch {

hosts => ["192.168.144.230:9200"]

index => "mongod_log-%{+YYYY.MM}"

}

stdout {

codec => rubydebug

}

}

[root@rhel6 config]# 

    然后，确保ELK服务端的服务进程都已经开启，启动命令：

[elasticsearch@rhel6 ]$ /home/elasticsearch/elasticsearch-5.0.1/bin/elasticsearch

[root@rhel6 ~]# /opt/logstash-5.0.1/bin/logstash -f /opt/logstash-5.0.1/config/logstash_mongodb.conf 

[root@rhel6 ~]# /opt/kibana-5.0.1/bin/kibana

在远程端启动filebeat，开始监控mongodb日志：

[root@se122 filebeat-5.0.1]# /opt/filebeat-5.0.1/filebeat -e -c /opt/filebeat-5.0.1/filebeat.yml -d "Publish"

2017/02/16 05:50:40.931969 beat.go:264: INFO Home path: [/opt/filebeat-5.0.1] Config path: [/opt/filebeat-5.0.1] Data path: [/opt/filebeat-5.0.1/data] Logs path: [/opt/filebeat-5.0.1/logs]

2017/02/16 05:50:40.932036 beat.go:174: INFO Setup Beat: filebeat; Version: 5.0.1

2017/02/16 05:50:40.932167 logp.go:219: INFO Metrics logging every 30s

2017/02/16 05:50:40.932227 logstash.go:90: INFO Max Retries set to: 3

2017/02/16 05:50:40.932444 outputs.go:106: INFO Activated logstash as output plugin.

2017/02/16 05:50:40.932594 publish.go:291: INFO Publisher name: se122

2017/02/16 05:50:40.935437 async.go:63: INFO Flush Interval set to: 1s

2017/02/16 05:50:40.935473 async.go:64: INFO Max Bulk Size set to: 2048

2017/02/16 05:50:40.935745 beat.go:204: INFO filebeat start running.

2017/02/16 05:50:40.935836 registrar.go:66: INFO Registry file set to: /opt/filebeat-5.0.1/data/registry

2017/02/16 05:50:40.935905 registrar.go:99: INFO Loading registrar data from /opt/filebeat-5.0.1/data/registry

2017/02/16 05:50:40.936717 registrar.go:122: INFO States Loaded from registrar: 1

2017/02/16 05:50:40.936771 crawler.go:34: INFO Loading Prospectors: 1

2017/02/16 05:50:40.936860 prospector_log.go:40: INFO Load previous states from registry into memory

2017/02/16 05:50:40.936923 registrar.go:211: INFO Starting Registrar

2017/02/16 05:50:40.936939 sync.go:41: INFO Start sending events to output

2017/02/16 05:50:40.937148 spooler.go:64: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s

2017/02/16 05:50:40.937286 prospector_log.go:67: INFO Previous states loaded: 1

2017/02/16 05:50:40.937404 crawler.go:46: INFO Loading Prospectors completed. Number of prospectors: 1

2017/02/16 05:50:40.937440 crawler.go:61: INFO All prospectors are initialised and running with 1 states to persist

2017/02/16 05:50:40.937478 prospector.go:106: INFO Starting prospector of type: log

2017/02/16 05:50:40.937745 log.go:84: INFO Harvester started for file: /root/rs0-0.log

    我们看到，这里已经开始实时监控mongodb日志是/root/rs0-0.log；然后，我们去logstash开启的前台窗口，可以看到有如下信息：

{

        "severity" => "I",

          "offset" => 243843239,

      "spend_time" => "0",

      "input_type" => "log",

          "source" => "/root/rs0-0.log",

         "message" => "2017-02-04T14:03:30.025+0800 I COMMAND  [conn272] command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} protocol:op_query 0ms",

            "type" => "mongodblog",

            "body" => "command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} protocol:op_query 0ms",

         "command" => "{ replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} ",

            "tags" => [

        [0] "beats_input_codec_plain_applied"

    ],

       "component" => "COMMAND",

      "@timestamp" => 2017-02-04T06:03:30.025Z,

         "db_name" => "admin.$cmd",

    "command_type" => "replSetGetStatus",

        "@version" => "1",

            "beat" => {

        "hostname" => "se122",

            "name" => "se122",

         "version" => "5.0.1"

    },

            "host" => "se122",

         "context" => "conn272"

}

这说明logstash按照配置文件正常过滤并按照指定的正则解析了mongodblog日志，再到kibana创建索引：



然后，就能在kibana自定义视图查看到监控到的Mongodb日志了：