ELK收集nginx日志并展示来源IP城市分布图

安装ELK

elasticsearch下载地址：

https://www.elastic.co/downloads/elasticsearch

logstash下载地址：

https://www.elastic.co/downloads/logstash

kibana下载地址：

https://www.elastic.co/downloads/kibana

安装参考（推荐官网下载压缩包再解压，brew安装会缺少x-pack插件）：

https://www.cnblogs.com/liuxiaoming123/p/8081883.html

操作步骤

1. elasticsearch部分

启动elasticsearch，访问http://localhost:9200

sh ./bin/elasticsearch

2.logstsh部分

日志文件格式

log_format  wwwlogs  '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for $request_time';

27.148.152.0 - - [07/Sep/2017:17:24:53 +0800] "GET /1.php HTTP/1.1" 200 25327 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
221.179.144.118 - - [07/Sep/2017:17:24:54 +0800] "GET /favicon.ico HTTP/1.1" 404 571 "http://192.168.44.111:88/1.php" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
180.130.2.40 - - [08/Sep/2017:17:14:00 +0800] "GET / HTTP/1.1" 403 169 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
117.30.72.137 - - [08/Sep/2017:17:14:00 +0800] "GET / HTTP/1.1" 403 169 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
183.136.190.62 - - [20/Aug/2019:04:23:00 +0800] "GET / HTTP/1.1" 403 571 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
47.101.202.76 - - [20/Aug/2019:05:29:45 +0800] "GET / HTTP/1.1" 403 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0"
183.136.190.62 - - [08/Sep/2019:16:27:28 +0800] "GET / HTTP/1.1" 403 571 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
123.125.67.162 - - [09/Sep/2019:17:35:29 +0800] "GET /robots.txt HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"

进入配置config文件夹，下载logstash提供的IP地址归类查询库

cd config/

wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

添加配置文件

// 新建配置文件
vim ./config/ip-test.conf

// 具体配置
input {
file {
path => "/usr/local/Cellar/logstash-6.6.0/config/nginx.log"
type => "nginx"
start_position => "beginning"
}
}

filter {
grok {
match => {"message" => "%{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} \"-\" \"%{DATA:http_user_agent}\""}
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/usr/local/Cellar/logstash-6.6.0/config/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
index => "logstash-map-%{+YYYY-MM}"
}
}

// 新建完记得执行检查配置文件命令，可以知道是否写错配置
./bin/logstash -f ./config/ip-test.conf -t

配置解释

• grok：用来正则匹配内容的插件
• geoip: 用来查询IP的插件
• source: 需要通过geoip插件处理的field，一般为ip，这里填ip对应的字段remote_addr
• target: 解析后的geoip地址数据，应该存放在哪一个字段中，默认是geoip这个字段
• database: ip地址归类查询库
• add_field: 这里两行是添加经纬度，地图中地区显示是根据经纬度来识

启动logstash，数据会传到elasticsearch保存

// 启动命令需要跟上配置文件路径 ./bin/logstash -f {configPath}
./bin/logstash -f ./config/ip-test.conf1.logstsh部分

elasticsearch数据展示

3.Kibana部分

在Kibana配置文件./config/kibana.yml的最后一行添加高德地图接口,如下

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&
z={z}'

删除bundles文件

rm -rf ./optimize/bundles

启动kibana，访问http://localhost:5601

./bin/kibana

设置Index Patterns，创建logstash*，选择@timestamp作为Time Filter field name

创建visualizations，选择coordinate Map类型，把logstash*选为分析对象，在左侧GeoCoordinates修改选项就可以看到IP来源分布图

右上角还可以增加时间筛选条件、自动刷新频率等

• 点赞
• 收藏
• 分享
• 文章举报