用Kibana+Logstash+Elasticsearch快速搭建实时日志查询、收集与分析系统

安装环境

先看看都需要安装什么软件包ruby 运行Kibana 必须，rubygems 安装ruby扩展必须bundler 功能类似于yumJDK 运行java程序必须redis 用来处理日志队列logstash 收集、过滤日志ElasticSearch 全文搜索服务(logstash集成了一个)kibana 页面展示192.168.18.240 logstash index，kibana，JDK192.168.18.241 logstash agent，JDK192.168.18.242 redis192.168.18.243 ElasticSearch，JDK先安装redis (192.168.18.242)#wget http://redis.googlecode.com/files/redis-2.6.12.tar.gz # tar zxvf redis-2.6.12.tar.gz# mv redis-2.6.12 redis# cd redis# make -j24# make install# vi /root/soft/redis/redis.conf修改level为 loglevel verbose# redis-server /root/soft/redis/redis.conf &看看 redis服务的状态# lsof -i:6379安装elasticsearch (192.168.18.243)

elasticsearch会依赖于java

3ff0

# vi /etc/apt/sources.list deb http: //ftp.debian.org/debian/ squeeze main non-free deb-src http: //ftp.debian.org/debian/ squeeze main non-free # apt-get update # apt-cache search sun-java # apt-get install sun-java6-jdk sun-java6-jre # java -version # wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.6.zip #unzip elasticsearch-0.20.6.zip# mv elasticsearch-0.20.6 /usr/local/share/elasticsearch # cd /usr/local/share/elasticsearch/bin/ # ./elasticsearch -f

在 logstash index上安装基础的软件环境： (192.168.18.240)

# vi /etc/apt/sources.list deb http: //ftp.debian.org/debian/ squeeze main non-free deb-src http: //ftp.debian.org/debian/ squeeze main non-free # apt-get update # apt-cache search sun-java # apt-get install sun-java6-jdk sun-java6-jre # java -version 开始安装logstash ( 其实logstash 就是一个java脚本，不需要安装... 下载即用 )# wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar # vi /root/soft/redis.confinput {redis {host => '192.168.18.242'data_type => 'list'port => "6379"key => 'logstash:redis'type => 'redis-input'}}output {elasticsearch {host => '192.168.18.243'port => "9300"}}# java -jar /root/soft/logstash-1.1.9-monolithic.jar agent -f /root/soft/redis.conf -- web --backend elasticsearch:///?local 现在可以通过浏览器访问一下 http://192.168.18.240:9292 看看logstash是的页面是个什么样子 配置logstash的agent (192.168.18.241)安装sun-java6-jre sun-java6-jdk#wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar # vi /root/soft/redis.confinput {stdin {type => "stdin-type"}file {type => "linux-syslog"# Wildcards work, here :)path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]}}output {redis {host => '192.168.18.242'data_type => 'list'key => 'logstash:redis'}}# java -jar /root/soft/logstash-1.1.9-monolithic.jar agent -f /root/soft/redis.conf &OK，最后就是 Kibana了 ，我把Kibana装在了 logstash index上面下载地址为 http://kibana.org/intro.html # apt-get install ruby rubygems # gem install bundler# bundle install ( /var/lib/gems/1.8/bin/bundle install )以上为ruby运行环境wget https://github.com/rashidkpc/Kibana/archive/v0.2.0.tar.gz # tar zxvf Kibana-0.2.0.tar.gz# cd Kibana-0.2.0直接安装就好了，非常简单，因为之前咱们已经安装好了 bundle编辑配置文件，指定 elasticsearch 的位置[192.168.18.240 root@nodec:/soft/Kibana-0.2.0]# vim KibanaConfig.rb.....Elasticsearch = "192.168.18.243:9200"KibanaPort = 5601KibanaHost = '0.0.0.0'.....主要是这几个参数启动的话需要ruby[192.168.18.240 root@nodec:/soft/Kibana-0.2.0]# /usr/bin/ruby kibana.rb &[192.168.18.240 root@nodec:/soft/Kibana-0.2.0]# == Sinatra/1.3.5 has taken the stage on 5601 for development with backup from Thin>> Thin web server (v1.5.0 codename Knife)>> Maximum connections set to 1024>> Listening on 0.0.0.0:5601, CTRL+C to stop如果ruby的东西都不缺的话，启动会很顺利，ok 现在看看5601端口的状态[192.168.233.128 root@nodec:/soft/Kibana-0.2.0]# lsof -i:5601COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEruby 3116 root 5u IPv4 28947 TCP *:esmagent (LISTEN)访问一下 试试看 http://192.168.18.240:5601 尝试搜索一下php的错误日志，比如mysql呵呵，要的就是这个效果，日志会实时的汇总到 logstash index 上供我们查询，当然这只是开始使用logstash的第一步而已，更多的高级功能可以看看官方文档http://logstash.net/docs/1.1.9/