ELK 日志系统搭建--监控nginx

logstash安装

下载路径:https://www.elastic.co/downloads/logstash（安装方法参考官网安装步骤）

要读取nginx日志，配置nginx日志格式

vim nginx.conf

修改nginx记录日志格式，从http模块下

log_format  main  '$remote_addr | $time_local | $request | $uri | '
'$status | $body_bytes_sent | $bytes_sent | $gzip_ratio | $http_referer | '
'"$http_user_agent" | $http_x_forwarded_for | $upstream_addr | $upstream_response_time | $upstream_status | $request_time';

修改完成后保存，使用./nginx -s reload 重新加载

/etc/logstash/conf.d下创建nginx日志配置文件
touch nginx_access.conf
sudo vim nginx_access.conf
input {
file {
path => [ "/usr/local/nginx/logs/adsapi.access.log" ]
type => "nginx_access"
}
}
filter {
grok {
match => [
"message", "%{IPORHOST:clientip} \| %{HTTPDATE:timestamp} \| (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-) \| %{URIPATH:uripath} \| %{NUMBER:response} \| (?:%{NUMBER:body_bytes_sent}|-) \| (?:%{NUMBER:bytes_sent}|-) \| (?:%{NOTSPACE:gzip_ratio}|-) \| (?:%{QS:http_referer}|-) \| %{QS:user_agent} \| (?:%{QS:http_x_forwarded_for}|-) \| (%{URIHOST:upstream_addr}|-) \| (%{BASE16FLOAT:upstream_response_time}) \| %{NUMBER:upstream_status} \| (%{BASE16FLOAT:request_time})"
]
}
geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]","%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]","%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]","float" ]
}

date {
match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]

}
mutate {
remove_field => "timestamp"

}
}

output {
elasticsearch {
hosts => ["127.0.*.*:9200"]
index => "logstash-nginx-access-%{+YYYY.MM.dd}"
user => "****" //下文安装kibana会设置
password => "pwd"
}
stdout {

}
}

2 . elasticsearch安装

下载地址：https://www.elastic.co/downloads/elasticsearch，安装步骤参见官网

安装完成后，从etc/elasticsearch/ 目录下

vim elasticsearch.yml

cluster.name: elk
node.name: es2
path.data: /data/elasticsearch（存储目录一定要给elasticsearch账户授权）
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: *.*.*.*(服务器ip)
http.port: 9200

启动服务sudo service elasticsearch start

查看启动日志，或直接查看启动后的进程状态是否成功

elasticsearch (pid 19206) is running.

浏览器输入：http://ip地址:9200/,给出响应结果

3 . kibana安装

下载地址：https://www.elastic.co/downloads/kibana

安装x-pack ,下载地址：https://www.elastic.co/downloads/x-pack

自己安装的在/user/share/ 目录下

从etc/kibana/ 目录下修改kibana.yml文件

sudo vim kibana.yml
server.name: "*.*.*.*"// (服务器ip地址)
elasticsearch.url: "http://*.*.*.*:9200"
elasticsearch.username: "username"
elasticsearch.password: "pwd"
增加：
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

配置完成后，三个服务依次启动 elasticsearch–kibana –logstash

service elasticsearch start

service kibana start

initctl start logstash

4. 要外网访问需要配置nginx.conf,访问地址到kibana

upstream elk {
ip_hash;
server 127.0.0.1:5601;
}

server {
listen 80;
server_name 域名;
server_tokens off;

client_body_timeout 5s;
client_header_timeout 5s;

location / {
proxy_pass http://elk/; index index.html index.htm;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
}
}

5.  配置完成后，重新加载nginx,浏览器输入域名，填写安装x-pack的用户名和密码
6.  登录成功后，![这里写图片描述](https://img-blog.csdn.net/20171201115255853?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
7.  在Configure an index pattern功能下配置：logstash-nginx-access*
![这里写图片描述](https://img-blog.csdn.net/20171201115318889?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
8.创建成功后，选择discover模块就能查看到