elk 日志分析系统Logstash+ElasticSearch+Kibana4

elkelasticsealogstashKibana

目录(?)[+]

elk 日志分析系统

Logstash+ElasticSearch+Kibana4
logstash 管理日志和事件的工具
ElasticSearch 搜索
Kibana4 功能强大的数据显示客户端
redis 缓存

安装包

logstash-1.4.2-1_2c0f5a1.noarch.rpm
elasticsearch-1.4.4.noarch.rpm
logstash-contrib-1.4.2-1_efd53ef.noarch.rpm
kibana-4.0.1-linux-x64.tar.gz

安装jdk

openjdk或者Oracle的jdk都可以。 

这里用openjdk

<code class="hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">yum <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span> java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

安装redis

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">yum install redis
/etc/init<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.d</span>/redis start
查看redis中的键值
redis-<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cli</span> keys <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'*'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

安装测试ElasticSearch

<code class="hljs java has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">rpm -ivh elasticsearch-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.4</span>.noarch.rpm
确认<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9200</span>端口监听有正常返回值即可:
curl -X GET http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//localhost:9200</span>

[root<span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@iZ</span>28ywqw7nhZ ~]# curl -X GET http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//localhost:9200</span>
{
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"status"</span> : <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Damballah"</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"cluster_name"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"elasticsearch"</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"version"</span> : {
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"number"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"1.4.4"</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_hash"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"c88f77ffc81301dfa9dfd81ca2232f09588bd512"</
4000
span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_timestamp"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"2015-02-19T13:05:36Z"</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_snapshot"</span> : <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>,
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"lucene_version"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"4.10.3"</span>
},
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"tagline"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You Know, for Search"</span>
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li></ul>

安装logstash

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">rpm -ivh logstash-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>_2c0f5a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.noarch</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rpm</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

logstash配置

最简单的是接受一个输入，然后将在输出出来:

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/logstash <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { stdout {} }'</span>
helo
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">03</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">19</span>T09:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38.161</span>+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0000</span> iZ28ywqw7nhZ helo</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

类似的还有:

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/logstash <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { stdout { codec => rubydebug } }'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

但是上面两个并没有很大的实际意义，我们可以将数据插入到elasticsearch中，然后用kibana显示出来。
首先要确保elasticsearch启动，9200监听。
然后插入数据:

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/opt/logstash/bin/logstash  <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { elasticsearch { host => localhost } }'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

然后可以用kibana 查看 

或者访问 

curl ‘http://localhost:9200/_search?pretty’ 来查看数据

下载kibana，运行

kibana就是一个java包。

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar <span class="hljs-attribute" style="box-sizing: border-box;">-xf</span> kibana<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-attribute" style="box-sizing: border-box;">-linux</span><span class="hljs-attribute" style="box-sizing: border-box;">-x64</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gz
bin/kibana <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">></span> kibana<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">log</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">&</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

里面配置index patterns ，用默认的就行了。

后记

基本上elk都是连着用的，连官网都是同一个官方网站，比如说elasticsearch一般是从logstash上面取数据的。kibana 默认又是从elasticsearch上面取数据。三者结合的非常好。 

更多功能正在探索中。elk 很强大的数据收集查询分析开源解决方案。
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash
http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html
Logstash 最佳实践
Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. 

init: tty (/dev/tty4) main process killed by KILL signal
list all indexs 

curl ‘localhost:9200/_cat/indices?v’
delete
curl -XDELETE ‘localhost:9200/customer?pretty’ 

curl ‘localhost:9200/_cat/indices?v’
http://www.elastic.co/guide/en/elasticsearch/reference/1.x/_index_and_query_a_document.html