搭建ELK收集Nginx日志
2017-06-20 10:19
295 查看
众所周知,ELK是日志收集套装,这里就不多做介绍了。
画了一个粗略的架构图,如下:
这里实际用了三个节点,系统版本为CentOS6.6,ES版本为2.3.5,logstash版本为2.4.0,kibana版本为4.5.4-1,nginx版本为1.8.1。
1、为三个节点安装java环境
2、三节点同步时间
3、安装elasticsearch集群,配置集群很简单,三节点保持集群名称相同即可,rpm包是提前在官网下载的
节点1,ES01:
节点2,ES02:
节点3,ES03:
启动服务:
通过cluster API查看集群状态:
4、为ES三个节点安装head插件
用浏览器访问head:
这个是我装完所有组件之后的状态,后面装完之后就不再贴head图了。
星形代表master
圆形代表slave
5、在节点1上安装logstash01
命令行验证logstash:
标准输入 --> 标准输出
标准输入 --> elasticsearch
从时间和内容可以看出,红色框的两条是我刚才添加的两条信息。
6、安装kibana
用浏览器访问 http://192.168.3.56:5601
7、安装redis
8、安装Nginx,使用nginx代理kibanna,并设置添加身份验证
9、将Nginx的日志格式转换为json格式
10、在需要收集日志也就是nginx server上安装filebeat
11、配置logstash01接收filebeat发出的日志,并输出到redis
12、在节点2上安装logstash02
13、配置logstash02从redis读取日志,并输出到elasticsearch中
14、登录配置kibana
配置完成后,就可以在Discover中看到nginx的日志了。
在Visualize里面可以画各种图,这里就不细说了。
展示一个我画的很简单的Dashboard:
画了一个粗略的架构图,如下:
这里实际用了三个节点,系统版本为CentOS6.6,ES版本为2.3.5,logstash版本为2.4.0,kibana版本为4.5.4-1,nginx版本为1.8.1。
192.168.3.56 ES01+logstash01+kibana+redis+nginx 192.168.3.49 ES02+logstash02 192.168.3.57 ES03
1、为三个节点安装java环境
# yum install -y java java-1.8.0-openjdk-devel # vim /etc/profile.d/java.sh export JAVA_HOME=/usr # source /etc/profile.d/java.sh
2、三节点同步时间
# ntpdate pool.ntp.org
3、安装elasticsearch集群,配置集群很简单,三节点保持集群名称相同即可,rpm包是提前在官网下载的
节点1,ES01:
# yum install -y elasticsearch-2.3.5.rpm # vim /etc/elasticsearch/elasticsearch.yml cluster.name: oupenges node.name: es01 network.host: 192.168.3.56 discovery.zen.ping.unicast.hosts: ["192.168.3.56", "192.168.3.49", "192.168.3.57"]
节点2,ES02:
# yum install -y elasticsearch-2.3.5.rpm # vim /etc/elasticsearch/elasticsearch.yml cluster.name: oupenges node.name: es02 network.host: 192.168.3.49 discovery.zen.ping.unicast.hosts: ["192.168.3.56", "192.168.3.49", "192.168.3.57"]
节点3,ES03:
# yum install -y elasticsearch-2.3.5.rpm # vim /etc/elasticsearch/elasticsearch.yml cluster.name: oupenges node.name: es03 network.host: 192.168.3.57 discovery.zen.ping.unicast.hosts: ["192.168.3.56", "192.168.3.49", "192.168.3.57"]
启动服务:
# service elasticsearch start # chkconfig elasticsearch on
通过cluster API查看集群状态:
# curl -XGET 'http://192.168.3.56:9200/_cluster/health?pretty=true' { "cluster_name" : "oupenges", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 56, "active_shards" : 112, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }
4、为ES三个节点安装head插件
# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
用浏览器访问head:
这个是我装完所有组件之后的状态,后面装完之后就不再贴head图了。
星形代表master
圆形代表slave
5、在节点1上安装logstash01
# yum install logstash-2.4.0.noarch.rpm
命令行验证logstash:
标准输入 --> 标准输出
# /opt/logstash/bin/logstash -e "input {stdin{}} output{stdout{ codec=>"rubydebug"}}" Settings: Default pipeline workers: 12 Pipeline main started hello { "message" => "hello", "@version" => "1", "@timestamp" => "2017-06-20T03:09:21.113Z", "host" => "uy-s-167" }
标准输入 --> elasticsearch
# /opt/logstash/bin/logstash -e 'input {stdin{}} output{ elasticsearch { hosts => ["192.168.3.56:9200"] index => "test"}}' Settings: Default pipeline workers: 12 Pipeline main started hello hi opera
从时间和内容可以看出,红色框的两条是我刚才添加的两条信息。
6、安装kibana
# yum install -y kibana-4.5.4-1.x86_64.rpm # vim /opt/kibana/config/kibana.yml elasticsearch.url: "http://192.168.3.56:9200" # service kibana start # chkconfig kibana on
用浏览器访问 http://192.168.3.56:5601
7、安装redis
# yum install -y redis # vim /etc/redis.conf daemonize yes bind 192.168.3.56 appendonly yes # service redis start # chkconfig redis on
8、安装Nginx,使用nginx代理kibanna,并设置添加身份验证
# wget http://nginx.org/download/nginx-1.8.1.tar.gz # tar xvf nginx-1.8.1.tar.gz # yum groupinstall -y "Development tools" # cd nginx-1.8.1/ # ./configure --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --user=nginx --group=nginx --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi --with-pcre # mkdir -pv /var/tmp/nginx/client/ # /usr/local/nginx/sbin/nginx # vim /usr/local/nginx/conf/nginx.conf 在http段添加一个server段 server { listen 8080; server_name 192.168.3.56; #当前主机名 auth_basic "Restricted Access"; auth_basic_user_file /usr/local/nginx/conf/htpasswd.users; #身份验证 location / { proxy_pass http://192.168.3.56:5601; #代理到kibana proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } # yum install -y httpd-tools # htpasswd -bc /usr/local/nginx/conf/htpasswd.users admin admin # cat /usr/local/nginx/conf/htpasswd.users admin:TvypNSDg6V3Rc # /usr/local/nginx/sbin/nginx -t # /usr/local/nginx/sbin/nginx -s reload
9、将Nginx的日志格式转换为json格式
# vim /usr/local/nginx/conf/nginx.conf log_format access1 '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"status":"$status"}'; access_log /var/log/nginx/access.log access1; # /usr/local/nginx/sbin/nginx -t # /usr/local/nginx/sbin/nginx -s reload
10、在需要收集日志也就是nginx server上安装filebeat
# yum install -y filebeat-1.2.3-x86_64.rpm # mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak # vim /etc/filebeat/filebeat.yml filebeat: prospectors: - paths: - /var/log/messages input_type: log document_type: nginxs1-system-message - paths: - /var/log/nginx/access.log input_type: log document_type: nginxs1-access-log registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["192.168.3.56:5044"] file: path: "/tmp/" filename: filebeat.txt shipper: logging: to_files: true files: path: /tmp/mybeat # service filebeat start # chkconfig filebeat on
11、配置logstash01接收filebeat发出的日志,并输出到redis
# vim /etc/logstash/conf.d/nginx.conf input { beats { port => 5044 codec => "json" }} output { if [type] == "nginxs1-system-message" { redis { data_type => "list" key => "nginxs1-system-message" host => "192.168.3.56" port => "6379" db => "0" }} if [type] == "nginxs1-access-log" { redis { data_type => "list" key => "nginxs1-access-log" host => "192.168.3.56" port => "6379" db => "0" }} file { path => "/tmp/nginx-%{+YYYY-MM-dd}messages.gz" } } # /etc/init.d/logstash configtest # service logstash restart
12、在节点2上安装logstash02
# yum install logstash-2.4.0.noarch.rpm
13、配置logstash02从redis读取日志,并输出到elasticsearch中
# vim /etc/logstash/conf.d/redis-to-es.conf input { redis { host => "192.168.3.56" port => "6379" db => "0" key => "nginxs1-system-message" data_type => "list" batch_count => 1 } redis { host => "192.168.3.56" port => "6379" db => "0" key => "nginxs1-access-log" data_type => "list" codec => "json" batch_count => 1 } } output { if [type] == "nginxs1-system-message" { elasticsearch { hosts => ["192.168.3.56:9200"] index => "nginxs1-system-message-%{+YYYY.MM.dd}" manage_template => true flush_size => 2000 idle_flush_time => 10 }} if [type] == "nginxs1-access-log" { elasticsearch { hosts => ["192.168.3.56:9200"] index => "logstash-nginxs1-access-log-%{+YYYY.MM.dd}" manage_template => true flush_size => 2000 idle_flush_time => 10 }} }
14、登录配置kibana
配置完成后,就可以在Discover中看到nginx的日志了。
在Visualize里面可以画各种图,这里就不细说了。
展示一个我画的很简单的Dashboard:
相关文章推荐
- ELK日志服务器的快速搭建并收集nginx日志 推荐
- elk系统搭建并收集nginx日志-主要步骤
- ubuntu搭建日志分析工具ELK收集Nginx日志
- 初探ELK-以收集 nginx 日志为例示范搭建一个 ELK 环境的基本步骤
- elk平台分析nginx日志的基本搭建
- elk+redis 搭建nginx日志分析平台
- ELK(一)ELK日志收集分析系统环境搭建
- 用ELK搭建简单的日志收集分析系统
- 使用elk+redis搭建nginx日志分析平台
- ELK5.0日志收集平台搭建指南
- 从0开始ELK日志收集系统介绍和搭建 JDK-1.6.0版
- ELK学习2_用Kibana和logstash快速搭建实时日志查询、收集与分析系统
- 使用elk+redis搭建nginx日志分析平台
- 结合Docker快速搭建ELK日志收集分析平台
- 使用elk+redis搭建nginx日志分析平台
- centos6.5下安装配置ELK及收集nginx日志
- ELK日志收集存储分析-----logstash+elasticsearch+kibana快速搭建日志平台
- ELK+redis搭建解析NGINX日志环境
- ELK集群部署及收集nginx日志
- ELK日志收集系统搭建