ELK日志分析系统搭建部署

Logstash+ElasticSearch+Kibana3架构日志系统一，分析系统简介
日志监控和分析在保障业务稳定运行时，起到了很重要的作用，不过一般情况下日志都分散在各个生产服务器，且开发人员无法登陆生产服务器，这时候就需要一个集中式的日志收集装置，对日志中的关键字进行监控，触发异常时进行报警，并且开发人员能够查看相关日志。logstash+elasticsearch+kibana3就是实现这样功能的一套系统，并且功能更强大。 logstash是一个管理日志和事件的工具，你可以收集它们，解析它们，并存储它们以供以后使用(例如日志搜索)，logstash有一个内置的web界面，用来搜索你的所有日志。logstash在部署时有两种运行模式：standalone和centralized： * standalone：standalone的意思是所有的事情都在一台服务器上运行，包括日志收集、日志索引、前端WEB界面都部署在一台机器上。 * centralized：就是多服务器模式，从很多服务器运输(ship)日志到一台总的日志(collector)服务器上用来索引和查找。 需要注意的是logstash本身并没有什么shipper和collector这种说法，因为不论是运输日志的进程还是汇集总的日志的进程运行的都是同一个程序，只是使用的配置文件不同而已。elasticsearch: 基于lucene的开源搜索引擎，是一个分布式的搜索分析系统，主要特点有：realtime data、real time analytics、distributed、high availability、multi-tenancy、fulltext search、document oriented、conflict management、schema free、restful api等等。kibana3:可视化日志和数据系统，作为WEB前端可以很容易的和elasticsearch系统结合。kibana有版本2和版本3的区分，版本2采用ruby编写，部署起来很麻烦，需要安装很多ruby依赖包(目前网上多是这个版本的部署)，版本3采用纯html+css编写，因此部署起来很方便，解压即用。 二，测试步骤1，测试架构两台机器试验，CentOS6.5 192.168.6.140：logstash agent，redis，jdk192.168.6.135：logstash index， elasticsearch ， kibana，jdk其中redis上安装agent端，agent和index的logstash安装是一样的，唯一不同点是conf文件不同。
2，redis和logstash agent安装（192.168.6.140）1），安装java环境http://www.java.com/en/download/manual.jsp rpm -ivh jdk-7u67-linux-x64.rpm Preparing... ########################################### [100%] 1:jdk ########################################### [100%]Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar... #java -versionjava version "1.7.0_67"Java(TM) SE Runtime Environment (build 1.7.0_67-b01)Java HotSpot(TM) 64-Bit Server VM (build24.65-b04, mixed mode) 2），安装rediswget http://download.redis.io/releases/redis-2.8.17.tar.gz tar -zxvf redis-2.8.17.tar.gzcd /redis-2.8.17.tar.gz 先安装tcl,否则下面会报错yum install tcl -ymake MALLOC=libc make testmake install # pwd/soft/redis-2.8.17/utils./install_server.sh Welcome to the redis service installerThis script will help you easily set up a runningredis server Please select the redis port for this instance: [6379]Selecting default: 6379Please select the redis config file name[/etc/redis/6379.conf] Selected default - /etc/redis/6379.confPlease select the redis log file name[/var/log/redis_6379.log] Selected default - /var/log/redis_6379.logPlease select the data directory for this instance[/var/lib/redis/6379] Selected default - /var/lib/redis/6379Please select the redis executable path[/usr/local/bin/redis-server] Selected config:Port :6379Config file :/etc/redis/6379.confLog file :/var/log/redis_6379.logData dir :/var/lib/redis/6379Executable :/usr/local/bin/redis-serverCli Executable : /usr/local/bin/redis-cliIs this ok? Then press ENTER to go on or Ctrl-C to abort.Copied /tmp/6379.conf => /etc/init.d/redis_6379Installing service...Successfully added to chkconfig!Successfully added to runlevels 345!Starting Redis server...Installation successful! # pwd/soft/redis-2.8.17/src[root@logserver src]# ./redis-cli -h 127.0.0.1 -p 6379127.0.0.1:6379> pingPONG127.0.0.1:6379> set name fooOK127.0.0.1:6379> get name"foo"127.0.0.1:6379> bye(error) ERR unknown command 'bye'127.0.0.1:6379> quit 3），安装logstashagent # rpm -ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpm Preparing... ########################################### [100%] 1:logstash ########################################### [100%][root@logserver soft]# rpm -ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpm Preparing... ########################################### [100%] 1:logstash-contrib ########################################### [100%]

修改配置文件

cat /etc/logstash/conf.d/logstash.agent.confinput { file { path =>"/var/log/messages" type => "syslog" } file { path => "/var/log/httpd/access_log" type => "apache" } } output { redis { host => "192.168.6.140" data_type =>"list" key => "logstash" }}启动logstash agent
service logstash start
root 14203 0.7 32.6 1397132 158811pxs/0 SNl 14:23 1:02 /usr/java/jdk1.7.0_67/bin/java -Djava.io.tmpdir=/var/lib/logstash-Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l/var/log/logstash/logstash.log 2，logstashindex 安装（192.168.6.135）1），安装java环境http://www.java.com/en/download/manual.jsp rpm -ivh jdk-7u67-linux-x64.rpm Preparing... ###########################################[100%] 1:jdk ########################################### [100%]Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar... #java -versionjava version "1.7.0_67"Java(TM) SE Runtime Environment (build1.7.0_67-b01)Java HotSpot(TM) 64-Bit Server VM (build24.65-b04, mixed mode) 2），安装logstash# rpm -ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpm Preparing... ###########################################[100%] 1:logstash ########################################### [100%][root@logserver soft]# rpm -ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpm Preparing... ###########################################[100%] 1:logstash-contrib ########################################### [100%]

修改配置文件

cat/etc/logstash/conf.d/logstash.index.conf
input {
redis
{
host => "192.168.6.140"
data_type =>"list"
port => "6379"
key => "logstash"
type => "redis-input"
}

}

output
{
elasticsearch {
host => "192.168.6.135"
port => "9300"
}
}

启动logstash index
service logstash start
ps aux |grep logstash
root 6365 1.1 38.5 1319172 140912 ? SNl 14:22 1:27 /usr/bin/java-Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC-XX:+UseConcMarkSweepGC -Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l/var/log/logstash/logstash.log

3)，安装elasicsearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.noarch.rpm rpm -ivh elasticsearch-1.3.4.noarch.rpm Preparing... ###########################################[100%] 1:elasticsearch ########################################### [100%]### NOT starting on installation, pleaseexecute the following statements to configure elasticsearch to start automaticallyusing chkconfig sudo /sbin/chkconfig --add elasticsearch### You can start elasticsearch byexecuting sudo service elasticsearch start chkconfig --add elasticsearchchkconfig elasticsearch onservice elasticsearch start

4),安装httpd和kibana3yum -y install httpdwget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.1.tar.gz tar kibana-3.1.1.tar.gz –C/var/www/htmlmv /var/www/html/kibana-3.1.1 /var/www/html/kibana 修改默认面板为logstash.jsongrep default_route/var/www/html/kibana/config.js
default_route :'/dashboard/file/logstash.json',启动httpd服务器，在web浏览器上访问
http://192.168.6.135/kibana/#/dashboard/file/default.json





本文出自 “jerrymin” 博客，请务必保留此出处http://jerrymin.blog.51cto.com/3002256/1565819