【centos】安装ELK之插件topbeat

部署环境：

centos 6.X

jdk 1.7

elasticsearch 2.3.1 https://www.elastic.co/downloads/elasticsearch
logstash 2.3.1 https://www.elastic.co/downloads/logstash
Kibana 4.5.0 https://www.elastic.co/downloads/kibana
topbeat 1.2.1 https://www.elastic.co/downloads/beats/topbeat
下载资源：

　　topbeat :https://www.elastic.co/downloads/beats/topbeat 比如下载的是tar.gz包；根据自己习惯，可以下载rpm，或者zip 都可以。

配置topbeat.yml文件：

################### Topbeat Configuration Example #########################

############################# Input ############################################
input:
# In seconds, defines how often to read server statistics
period: 10

# Regular expression to match the processes that are monitored
# By default, all the processes are monitored
procs: [".*"]

# Statistics to collect (all enabled by default)
stats:
# per system statistics, by default is true
system: true

# per process statistics, by default is true
process: true

# file system information, by default is true
filesystem: true

# cpu usage per core, by default is false
cpu_per_core: false

###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

### Elasticsearch as output
elasticsearch:
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200 # hosts: ["localhost:9200"]
hosts: ["192.168.87.8:9200"]

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "admin"
#password: "s3cr3t"

# Number of workers per Elasticsearch host.
#worker: 1

# Optional index name. The default is "topbeat" and generates
# [topbeat-]YYYY.MM.DD keys.
#index: "topbeat"

# A template is used to set the mapping in Elasticsearch
# By default template loading is disabled and no template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones
#template:

# Template name. By default the template name is topbeat.
#name: "topbeat"

# Path to template file
#path: "topbeat.template.json"

# Overwrite existing template
#overwrite: false

# Optional HTTP Path
#path: "/elasticsearch"

# Proxy server url
#proxy_url: http://proxy:3128 
# The number of times a particular Elasticsearch index operation is attempted. If
# the indexing operation doesn't succeed after this many retries, the events are
# dropped. The default is 3.
#max_retries: 3

# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50

# Configure http request timeout before failing an request to Elasticsearch.
#timeout: 90

# The number of seconds to wait for new events between two bulk API index requests.
# If `bulk_max_size` is reached before this interval expires, addition bulk index
# requests are made.
#flush_interval: 1

# Boolean that sets if the topology is kept in Elasticsearch. The default is
# false. This option makes sense only for Packetbeat.
#save_topology: false

# The time to live in seconds for the topology information that is stored in
# Elasticsearch. The default is 15 seconds.
#topology_expire: 15

# tls configuration. By default is off.
#tls:
# List of root certificates for HTTPS server verifications
#certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for TLS client authentication
#certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#certificate_key: "/etc/pki/client/cert.key"

# Controls whether the client verifies server certificates and host name.
# If insecure is set to true, all server host names and certificates will be
# accepted. In this mode TLS based connections are susceptible to
# man-in-the-middle attacks. Use only for testing.
#insecure: true

# Configure cipher suites to be used for TLS connections
#cipher_suites: []

# Configure curve types for ECDHE based cipher suites
#curve_types: []

# Configure minimum TLS version allowed for connection to logstash
#min_version: 1.0

# Configure maximum TLS version allowed for connection to logstash
#max_version: 1.2

### Logstash as output
#logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]

# Number of workers per Logstash host.
#worker: 1

# Set gzip compression level.
#compression_level: 3

# Optional load balance the events between the Logstash hosts
#loadbalance: true

# Optional index name. The default index name depends on the each beat.
# For Packetbeat, the default is set to packetbeat, for Topbeat
# top topbeat and for Filebeat to filebeat.
#index: topbeat

# Optional TLS. By default is off.
#tls:
# List of root certificates for HTTPS server verifications
#certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for TLS client authentication
#certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#certificate_key: "/etc/pki/client/cert.key"

# Controls whether the client verifies server certificates and host name.
# If insecure is set to true, all server host names and certificates will be
# accepted. In this mode TLS based connections are susceptible to
# man-in-the-middle attacks. Use only for testing.
#insecure: true

# Configure cipher suites to be used for TLS connections
#cipher_suites: []

# Configure curve types for ECDHE based cipher suites
#curve_types: []

### File as output
#file:
# Path to the directory where to save the generated files. The option is mandatory.
#path: "/tmp/topbeat"

# Name of the generated files. The default is `topbeat` and it generates files: `topbeat`, `topbeat.1`, `topbeat.2`, etc.
#filename: topbeat

# Maximum size in kilobytes of each file. When this size is reached, the files are
# rotated. The default value is 10 MB.
#rotate_every_kb: 10000

# Maximum number of files under path. When this number of files is reached, the
# oldest file is deleted and the rest are shifted from last to first. The default
# is 7 files.
#number_of_files: 7

### Console output
# console:
# Pretty print json event
#pretty: false

############################# Shipper #########################################

shipper:
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
# If this options is not defined, the hostname is used.
#name:

# The tags of the shipper are included in their own field with each
# transaction published. Tags make it easy to group servers by different
# logical properties.
#tags: ["service-X", "web-tier"]

# Uncomment the following if you want to ignore transactions created
# by the server on which the shipper is installed. This option is useful
# to remove duplicates if shippers are installed on multiple servers.
#ignore_outgoing: true

# How often (in seconds) shippers are publishing their IPs to the topology map.
# The default is 10 seconds.
#refresh_topology_freq: 10

# Expiration time (in seconds) of the IPs published by a shipper to the topology map.
# All the IPs will be deleted afterwards. Note, that the value must be higher than
# refresh_topology_freq. The default is 15 seconds.
#topology_expire: 15

# Internal queue size for single events in processing pipeline
#queue_size: 1000

# Configure local GeoIP database support.
# If no paths are not configured geoip is disabled.
#geoip:
#paths:
#  - "/usr/share/GeoIP/GeoLiteCity.dat"
#  - "/usr/local/var/GeoIP/GeoLiteCity.dat"

############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

# Send all logging output to syslog. On Windows default is false, otherwise
# default is true.
#to_syslog: true

# Write all logging output to files. Beats automatically rotate files if rotateeverybytes
# limit is reached.
#to_files: false

# To enable logging to files, to_files option has to be set to true
files:
# The directory where the log files will written to.
#path: /var/log/mybeat

# The name of the files where the logs are written to.
#name: mybeat

# Configure log file size limit. If limit is reached, log file will be
# automatically rotated
rotateeverybytes: 10485760 # = 10MB

# Number of rotated log files to keep. Oldest files will be deleted first.
#keepfiles: 7

# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are beat, publish, service
# Multiple selectors can be chained.
#selectors: [ ]

# Sets log level. The default log level is error.
# Available log levels are: critical, error, warning, info, debug
#level: error

View Code
其中，

　　

input:
# In seconds, defines how often to read server statistics
# 间隔多少秒统计服务器数据
period: 10

# Regular expression to match the processes that are monitored
# By default, all the processes are monitored
　# 此处可以过滤不需要的进程，采集全部进行 使用*号
procs: [".*"]

# Statistics to collect (all enabled by default)
stats:
# per system statistics, by default is true
system: true

# per process statistics, by default is true
process: true

# file system information, by default is true
filesystem: true

# cpu usage per core, by default is false
　  # 每个进程占用的CPU，默认未打开统计

cpu_per_core: false

如果修改了topbeat.yml文件，需要重启一下topbeat服务的；

[root@td-cigtest-1 topbeat]# ps -ef|grep topbeat
root     55861 52838  0 10:14 pts/1    00:00:52 ./topbeat -c topbeat.yml
root     61753 52838  0 12:05 pts/1    00:00:00 grep topbeat
[root@td-cigtest-1 topbeat]# kill 55861
[root@td-cigtest-1 topbeat]# vim topbeat.yml
[root@td-cigtest-1 topbeat]# ./topbeat -c topbeat.yml &
[1] 61878
[root@td-cigtest-1 topbeat]# ps -ef|grep topbeat
root     61878 52838  0 12:07 pts/1    00:00:00 ./topbeat -c topbeat.yml
root     61894 52838  0 12:07 pts/1    00:00:00 grep topbeat
[root@td-cigtest-1 topbeat]#

然后，新配置就生效了。

上传ElasticSearch数据模板（topbeat.template.json为你自己topbeat中的文件地址，官方给的是/etc/..... ）

curl -XPUT 'http://192.168.87.8:9200/_template/topbeat' -d@/usr/local/topbeat/topbeat.template.json

2016-4-15补充：

　　template设定也有多种方法。最简单的就是和存储数据一样POST上去。长期的办法，就是写成json文件放在配置路径里。其中，default配置放在/etc/elasticsearch/下，其他配置放在/etc/elasticsearch/templates/下。

我个人的配置，是放在了/usr/local/elasticsearch/templates/下；然后重启ES服务就可以生效了。

Q：ES没有安装成服务，查了一下资料说要用一个可以安装elasticsearch-servicewrapper插件。暂时没有尝试成功，这个插件有点问题，不再支持ES2.X版本了。

启动topbeat服务：

[root@td-cigtest-1 topbeat]# ls
topbeat  topbeat.template.json  topbeat.yml
[root@td-cigtest-1 topbeat]# pwd
/usr/local/topbeat
[root@td-cigtest-1 topbeat]# curl -XPUT 'http://192.168.87.8:9200/_template/topbeat' -d@/usr/local/topbeat/topbeat.template.json
{"acknowledged":true}[root@td-cigtest-1 topbeat]# ./topbeat -c topbeat.yml&
[2] 9298
[root@td-cigtest-1 topbeat]#

测试是否正常：

curl -XGET 'http://192.168.87.8:9200/topbeat-*/_search?pretty'

这种方式，是直接把数据提交到了ES中心，如果想先提交到logstash的话，参考https://www.elastic.co/guide/en/beats/topbeat/current/config-topbeat-logstash.html

参考：
https://www.elastic.co/guide/en/beats/topbeat/current/topbeat-template.html