logstash解析haproxy的haproxy.log日志字段
2018-09-06 14:39
3051 查看
在工作中需要在elk中展示haproxy的日志,用于分析访问客户IP、URI、PV等;
日志样例:我只取出了2种类型的日志,在logstash解析时同时解析2种日志格式;
Sep 5 10:36:55 localhost haproxy[567197]: 221.238.230.162:49225 [05/Sep/2018:10:36:27.130] SBD-Security SBD-Reglog/shvnginx02 0/0/896 5489 -- 1401/2/2/0/0 0/0 Sep 5 10:36:54 localhost haproxy[567197]: 58.220.76.39:59768 [05/Sep/2018:10:36:54.026] SBD-Nginx imageservers/image01 0/0/0/2/2 200 5126 - - ---- 1353/1351/0/1/0 0/0 {|s06.abc001.cn} "GET /ftp_product_img/cn1100017322EA_1_thb.jpg?t=201709101850 HTTP/1.1"
filebeat先读取haproxy.log的日志,写入redis中:
filebeat.inputs: - type: log paths: - /var/log/haproxy/haproxy.log tags: ["sbd_haproxy"] fields: type: sbd_haproxy fields_under_root: true output.redis: hosts: ["10.78.1.181"] key: "sbd_haproxy" type: list
logstash向redis读取数据,解析过滤之后写入elastic中:
input { redis { host => "10.78.1.181" port => 6379 data_type => list key => "sbd_haproxy" } } filter { grok { match => ["message","%{HAPROXYHTTP}","message","%{HAPROXYTCP}"] } mutate { remove_field => ["host","captured_response_cookie","haproxy_time","haproxy_month","haproxy_hour","srvconn","backend_queue","retries","termination_state"] remove_field => ["haproxy_monthday","syslog_server","time_backend_connect","pid","srv_queue","beat","beconn","client_port","haproxy_milliseconds","@version"] remove_field => ["haproxy_minute","offset","haproxy_second","actconn","source","program","haproxy_year","feconn","http_version"] remove_field => ["message","prospector","time_duration","time_queue","syslog_timestamp","captured_request_cookie","prospector"] } date { match => ["accept_date","dd/MMM/yyyy:HH:mm:ss.SSS"] } mutate { remove_field => ["accept_date"] } } output { if [type] == "sbd_haproxy" { if [tags][0] == "sbd_haproxy" { elasticsearch { hosts => ["10.78.1.184:9200","10.78.1.185:9200","10.78.1.188:9200"] index => "%{type}-%{+YYYY.MM.dd}" } #stdout { codec => rubydebug } } } }
kibana展示:
相关文章推荐
- SQL Server 2008事务日志的[RowLog Contents 0] 字段没人能解析出来?
- logstash 使用grok正则解析日志
- Logstash5.4.1解析日志报错处理 推荐
- filebeat+logstash对springcloud项目日志进行解析
- Caffe命令行解析及log日志文件生成
- Logstash使用grok插件解析Nginx日志
- Logstash 实践之Spark Driver日志解析
- 解析训练日志文件train_output.log
- logstash使用grok正则解析日志
- Linux 详解 /var/log/xferlog的各个字段解析
- 3.zookeeper原理解析-数据存储之TxnLog事务日志
- /var/log/xferlog的各个字段解析
- Logstash 实践之Redis日志解析
- [log]logstash添加字段.geoip展示
- 日志字段解析
- nginx 日志 log_format 及字段说明
- ELK-Logstash收集haproxy日志
- Filebeat+Logstash+ElasticSearch+Kibana搭建Apache访问日志解析平台
- Logstash日志字段拆分grok
- logstash使用grok正则解析日志和kibana遇到的问题