ElasticSearch, Kibana, and Fluentd 的简单部署实例

参考：http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd

要求：JDK 1.6及以上，ruby 1.9.2及以上(最后附安装过程mac os)


一、安装ElasticSearch：
1、下载：http://www.elasticsearch.org/download/，本人下载的是：elasticsearch-0.90.3.tar.gz 包

$ tar zxvf elasticsearch-0.90.0.RC2.tar.gz
$ cd elasticsearch-0.90.0.RC2/

Once installation is complete, start ElasticSearch.

$ ./bin/elasticsearch -f

2、安装Kibana(http://kibana.org/intro.html)

Extract your archive
Open KibanaConfig.rb in your favorite editor
Set Elasticsearch = "localhost:9200" to your ElasticSearch server

Run gem install bundler
Run bundle install

运行

$ bundle exec ruby kibana.rb

3、fluentd的安装，参照：http://docs.fluentd.org/categories/installation

4、elasticsearch plugin 的安装：https://github.com/uken/fluent-plugin-elasticsearch(运行时如有问题，请查阅：https://gist.github.com/y-matsuwitter/4951605)
如下：
# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql
# vi /etc/td-agent/td-agent.conf
file1.txt
# service td-agent restart
Starting td-agent: 2013-02-14 18:14:40 +0900: fluent/supervisor.rb:187:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'mysql'. Run 'gem search -rd fluent-plugin' to find plugins"
[FAILED]
file2.txt
# /usr/lib64/fluent/ruby/bin/gem environment
~~~~~中略~~~~~~~~~~~~~
- GEM PATHS:
- /usr/local/rvm/gems/ruby-1.9.3-p327
- /usr/local/rvm/gems/ruby-1.9.3-p327@global
~~~~~中略~~~~~~~~~~~~~
file3.txt
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems
file4.txt

GEM_HOME="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"
GEM_PATH="/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/"
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql


-----------------------------------------------------
三个节点：192.168.1.4，192.168.1.5，192.168.1.6；

fluentd的配置，

192.168.1.4
********************************************************************

<source>
type http
port 8888
</source>

<source>
type forward
port 24224
bind 0.0.0.0
</source>

<source>
type tail
format /^ *(?<level>[^ ]* )(?<type>[^ ]* )(?<time>[^ ]* [^ ]* )(?<message>.*)$/
path /data/cassandra/log/system.log
pos_file /root/logserver/fluentd/logpos/system.pos
tag 192.168.1.4
</source>

<match 192.**>
type copy
<store>
type file
path /root/logserver/fluentd/log/cassandra
</store>
<store>
type elasticsearch
logstash_format true
flush_interval 10s
include_tag_key true
tag_key host
</store>
</match>

<match debug.**>
type stdout
</match>


192.168.1.5/6
********************************************************************
<source>
type http
port 8888
</source>
<source>
type tail
format /^ *(?<level>[^ ]* )(?<type>[^ ]* )(?<time>[^ ]* [^ ]* )(?<message>.*)$/
path /data/cassandra/log/system.log
pos_file /root/logserver/fluentd/logpos/system.pos
tag 192.168.1.5
</source>
<match debug.**>
type stdout
</match>
<match 192.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 10s
phi_threshold 8
hard_timeout 60s
<server>
name icecrown
host 192.168.1.4
port 24224
weight 60
</server>
<secondary>
type file
path /root/logserver/fluentd/forward-failed
</secondary>
</match>
********************************************************************
--------------ok-------------------------------------


安装相关资料
ruby install:
1、rvm:的安装（http://stackoverflow.com/questions/11677771/rvm-command-not-found-mac-ox）

curl -L https://get.rvm.io | bash -s -- --version latest

$ source ~/.rvm/scripts/rvm

then

$ type rvm | head -n 1

If the output is:

rvm is a function

You may need to add "source ~/.rvm/scripts/rvm" to your ~/.bash_profile file
或者（this worked for me):

rm -rf ~/.rvm
curl -L https://get.rvm.io | bash -s stable

use rvm install ruby(http://misheska.com/blog/2013/06/16/using-rvm-to-manage-multiple-versions-of-ruby/)

$ rvm install 1.9.3

You may get an error message saying “There was an error while trying to resolve rubygems version for ‘latest’. Halting the installation.” Just run the install again like so to fix the issue:

$ rvm reinstall 1.9.3

Verify the RVM install by running the following commands:

$ rvm -h
$ rvm list
$ rvm use 1.9.3
$ rvm rubygems latest

To ensure that the newer Ruby 1.9.3 is used by default instead of the system 1.8.7 version, run the following command:

$ rvm use 1.9.3 --default

linux 上的安装：http://tecadmin.net/how-to-install-ruby-2-0-0-on-centos-6-using-rvm/#