Kibana + ElasticSearch + Logstash + Redis on RHEL 6
2013-08-03 14:21
435 查看
系统各部分应用介绍:
Kibana:开源web展现
Elasticsearch:开源的搜索引擎框架logstash部分,可进行多数据集群,提高效率,从redis中读取数据,并转发到Kibana
Logstash:系统log收集,转载的工具,同时集成各类日志插件,对日志查询和分析的效率显著提高
Logstash shipper:收集log 并将log转发给redis 存储
Logstash indexer:从redis中读取数据并转发给elasticsearch
Redis:是db,logstash shipper将log转发到redis数据库中存储
部署:
1、jdk:
path:/usr/local/jdk7
cat /etc/profile
export JAVA_HOME=/usr/local/jdk7
export PATH=$JAVA_HOME/bin:$PATH
export REDIS_HOME=/usr/local/redis-2.6.12
export ES_HOME=/usr/local/elasticsearch
export ES_CLASSPATH=$ES_HOME/config
2、ElasticSearch
vim 6379.conf
redis {
host => "192.168.0.235"
port => "6379"
type => "redis-input"
data_type => "list"
key => "logstash"
format => "json_event"
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "192.168.0.235"
port => "9300"
cluster => "elasticsearch"
}
}
file {
type => "nginx"
path => ["/usr/local/nginx/logs/*.log"]
exclude => ["*.gz"]
tags => ["nginx"]
}
}
output {
stdout { debug => true debug_format => "json"}
redis {
host => "192.168.0.235"
data_type => "list"
key => "logstash"
}
}
Kibana:开源web展现
Elasticsearch:开源的搜索引擎框架logstash部分,可进行多数据集群,提高效率,从redis中读取数据,并转发到Kibana
Logstash:系统log收集,转载的工具,同时集成各类日志插件,对日志查询和分析的效率显著提高
Logstash shipper:收集log 并将log转发给redis 存储
Logstash indexer:从redis中读取数据并转发给elasticsearch
Redis:是db,logstash shipper将log转发到redis数据库中存储
部署:
1、jdk:
path:/usr/local/jdk7
cat /etc/profile
export JAVA_HOME=/usr/local/jdk7
export PATH=$JAVA_HOME/bin:$PATH
export REDIS_HOME=/usr/local/redis-2.6.12
export ES_HOME=/usr/local/elasticsearch
export ES_CLASSPATH=$ES_HOME/config
2、ElasticSearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz tar xvf elasticsearch-0.20.2.tar.gz mv elasticsearch-0.20.2 elasticsearch
cd /usr/local/elasticsearch/config
vim elasticsearch.ymlcluster.name: elasticsearch
node.name: "litong"
path.conf: /usr/local/elasticsearch
path.data: /usr/local/elasticsearch/data
path.work: /usr/local/elasticsearch/tmp
path.logs: /usr/local/elasticsearch/logs
bootstrap.mlockall: true
mkdir -p /usr/local/elasticsearch/data /usr/local/elasticsearch/tmp /usr/local/elasticsearch/logs
3、Configure Java Service Wrapper
Get the service wrapper
wget http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip unzip master.zip mv elasticsearch-servicewrapper-master/service/ /usr/local/elasticsearch/bin/ rm -rf elasticsearch-servicewrapper-master/
vim service/elasticsearch.conf
set.default.ES_HOME=/usr/local/elasticsearch set.default.ES_HEAP_SIZE=1024 #memory wrapper.java.additional.10=-Des.max-open-files=true wrapper.logfile.maxsize=5m wrapper.logfile.maxfiles=5
service:/etc/init.d/elasticsearch
bin/service/elasticsearch install
service elasticsearch start
4、ElasticSearch Head
bin/plugin -install mobz/elasticsearch-head
5、RedisInstall Redis server
wget http://redis.googlecode.com/files/redis-2.6.12.tar.gz tar xzf redis-2.6.12.tar.gz mv redis-2.6.12 /usr/local/redis cd /usr/local/redis make
make install5、Configure Redis – ‘cp redis.conf 6379.conf’
vim 6379.conf
daemonize yes pidfile /var/run/redis/redis_6379.pid port 6379 timeout 300 tcp-keepalive 60 logfile /var/log/redis/redis_6379.logAdd REDIS home to root user’s ‘.bash_profile’
# Redis export REDIS_HOME=/usr/local/redisCopy Redis init script
cp utils/redis_init_script /etc/init.d/redis_6379Configure Redis init script
# chkconfig: - 85 15 # description: Redis is a persistent key-value database # processname: redis REDISPORT=6379 EXEC=/usr/local/redis/src/redis-server CLIEXEC=/usr/local/redis/src/redis-cli PIDFILE=/var/run/redis/redis_6379.pid CONF="/usr/local/redis/6379.conf"Activate Redis service
mkdir /var/run/redis /var/log/redis cd /etc/init.d chkconfig --add redisStart
service redis start
6、Logstash
mkdir /usr/local/logstash cd /usr/local/logstash wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar
Indexer configuration – indexer.conf:input {
redis {
host => "192.168.0.235"
port => "6379"
type => "redis-input"
data_type => "list"
key => "logstash"
format => "json_event"
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "192.168.0.235"
port => "9300"
cluster => "elasticsearch"
}
}
Shipper configuration – shipper.conf:input {
file {
type => "nginx"
path => ["/usr/local/nginx/logs/*.log"]
exclude => ["*.gz"]
tags => ["nginx"]
}
}
output {
stdout { debug => true debug_format => "json"}
redis {
host => "192.168.0.235"
data_type => "list"
key => "logstash"
}
}
java -jar logstash-1.1.9-monolithic.jar agent -f indexer.conf & java -jar logstash-1.1.9-monolithic.jar agent -f shipper.conf &7、KibanaSetup Ruby
yum install ruby ruby-devel ruby-ri ruby-rdoc rubygems
wget http://production.cf.rubygems.org/rubygems/rubygems-2.0.3.zip unzip rubygems-2.0.3.zip ruby rubygems-2.0.3/setup.rbGet Kibana
wget https://github.com/rashidkpc/Kibana/archive/v0.2.0.zip unzip v0.2.0.zip cd Kibana-0.2.0 gem install bundler bundle install
Configure KibanaConfig.rb:
Elasticsearch = "192.168.0.235:9200" KibanaPort = 80 KibanaHost = '192.168.0.235'Run Kibana
bundle exec ruby kibana.rb
相关文章推荐
- ELK日志平台搭建logstash+elasticsearch+redis+Kibana
- kibana+logstash+elasticsearch+redis 续(1)
- 使用kibana+logstash+elasticsearch+redis搭建分布式日志收集、分析、查询系统。
- How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7
- 使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务
- 使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务
- ELK(elasticsearch+logstash+kibana)+redis实现nginx 日志的分析
- Centos 6.5 安装nginx日志分析系统 elasticsearch + logstash + redis + kibana
- ElasticSearch+LogStash+Kibana+Redis日志服务的高可用方案
- ElasticSearch+LogStash+Kibana+Redis日志服务的高可用方案
- 搭建ELK(ElasticSearch+Logstash+Kibana)日志分析系统(五) 通过redis扩展logstash
- 开源分布式搜索平台ELK(Elasticsearch+Logstash+Kibana)+Redis+Syslog-ng实现日志实时搜索
- 笔记: 试用Kibana+Logstash+Elasticsearch+Redis
- CENTOS6.5安装日志分析ELK elasticsearch + logstash + redis + kibana
- 安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台
- Logstash+Redis+Elasticsearch+Kibana+Nginx搭建日志分析系统
- Logstash+Redis+Elasticsearch+Kibana+Nginx搭建日志分析系统
- Logstash+Redis+Elasticsearch+Kibana+Nginx搭建日志分析系统
- ELK学习3_使用redis+logstash+elasticsearch+kibana快速搭建日志平台
- elasticsearch+logstash+redis+kibana+logback整合