ELK-filebeat+kafka日志收集

环境

centos6.9

ELK5.6

所有节点都是单点非集群

filebeat：10.99.2.16

elk：10.99.2.17

kafka：10.99.2.23

官方文档

软件下载

ELK5.6.7-rpm

java1.8-rpm

es安装

yum安装java环境和es：

yum install elasticsearch-5.6.7.rpm jre-8u161-linux-x64.rpm -y
yum安装的需要普通用户开启
useradd elk
cp -r /etc/elasticsearch /usr/share/elasticsearch/config
chown -R elk:elk /usr/share/elasticsearch
chown -R elk:elk /etc/elasticsearch

配置文件

1、在/etc/security/limits.conf中添加：
3.           soft   nproc        655350
4.           hard   nproc        655350
2、在/etc/security/limits.d/90-nproc.conf中修改：
5.          soft    nproc     4096
root       soft    nproc     unlimited
3、修改es配置文件jvm.options和elasticsearch.yml
# jvm.options：
-Xms2g
-Xmx2g
# elasticsearch.yml：
network.host: 0.0.0.0
http.port: 9200
bootstrap.system_call_filter: false

启动

su elk      切换用户
/usr/share/elasticsearch/bin/elasticsearch
加上"-d"是后台运行，不加可以看到无法启动的报错信息

logstash安装

yum安装

yum install -y logstash-5.6.7.rpm

配置文件 /etc/logstash/logstash.yml

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d

配置文件 /etc/logstash/conf.d/logstash_agent.conf

input {
kafka {
bootstrap_servers => "10.99.2.30:9092"
topics => "service-pv"
#   group_id => "logstash-lf"
codec => "json"
}
}
filter {
json {
source => "message"
remove_field => "message"
}
}
output {
elasticsearch {
hosts => "10.99.2.17:9200"
}
stdout { codec => rubydebug }
}

启动

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_agent.conf

kibana安装

yum安装

yum install kibana-5.6.5-x86_64.rpm

配置文件 /etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

启动

/usr/share/kibana/bin/kibana

kafka

kafka下载地址

也需要java环境，安装：

tar -zxf  kafka_2.11-1.0.0.tgz
cd kafka_2.11-1.0.0
启动zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
启动kafka
bin/kafka-server-start.sh config/server.properties

1、创建topic
$bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

2、查看创建的topic
$ bin/kafka-topics.sh -list -zookeeper localhost:2181
test

3、生产消息测试
$bin/kafka-console-producer.sh --broker-list 192.168.12.105:9092 --topic test
this is test          #输入后回车

4、消费消息测试
$bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
this is test

filebeat

filebeat下载地址

下载安装

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.7-x86_64.rpm yum install -y filebeat-5.6.7-x86_64.rpm

修改配置文件 /etc/filebeat/filebeat.yml

- input_type: log
paths:
- /usr/local/lnmp/tengine/logs/access_json.log
# 要采集的日志文件
output.kafka:
hosts: ["10.99.2.30:9092"]
enable: true
topic: service-pv
# 输出到kafka

测试配置文件

filebeat.sh -configtest -e

启动

/etc/init.d/filebeat start