Elasticsearch和mysql数据同步（logstash）

1、版本介绍

Elasticsearch：
https://www.elastic.co/products/elasticsearch
版本：2.4.0

Logstash：
https://www.elastic.co/products/logstash
版本：2.4.0
所需要的安装文件，到官网下载即可。

还需要对应的数据库JDBC，这里使用的是mysql-connector-java-5.1.39.jar

Elasticsearch配置请参照之前的博客，不在这里介绍了。

2、Logstash安装配置

Logstash直接下载后解压即可，主要是配置文件的内容编写。

安装logstash-input-jdbc的Logstash插件，用来进行mysql、oracle等的数据同步。

[zsz@VS-zsz logstash-2.4.0]$ bin/plugin install logstash-input-jdbc

The use of bin/plugin is deprecated and will be removed in a feature release. Please use bin/logstash-plugin.

Validating logstash-input-jdbc

Installing logstash-input-jdbc

Installation successful

配置文件（自行指定文件名，这里命名为logstash-mysql.conf ）：
[zsz@VS-zsz conf]$ vi logstash-mysql.conf
input {

jdbc {

jdbc_driver_library => "/usr/local/logstash-2.4.0/mysql-connector-java-5.1.39.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_connection_string => "jdbc:mysql://192.168.****:3306/******?characterEncoding=UTF-8&useSSL=false"

jdbc_user => "*****"

jdbc_password => "*********"

statement => "SELECT * FROM news limit 0,1"

jdbc_paging_enabled => "true"

jdbc_page_size => "50000"

schedule => "* * * * *"

}

}

filter {

json {

source => "message"

remove_field => ["message"]

}

}

output {

stdout {

codec => rubydebug

}

elasticsearch {

hosts => "192.168.****"

index => "myindex"

}

}

启动Logstash
[zsz@VS-zsz conf]$ /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/conf/logstash-mysql.conf
Settings: Default pipeline workers: 2

Pipeline main started

{

"pk" => 8,

"user_pk" => 11,

"media_pk" => 5,

"topic_pk" => 0

}

{

"pk" => 8,

"user_pk" => 11,

"media_pk" => 5,

"topic_pk" => 0

}
........

这个进程会一直执行下去，因为设置的schedule => "* * * * *"（每分钟执行一次），如果想结束进程需要kill掉进程。

查看elasticsearch是否同步数据成功
[root@VS-zsz conf]# curl '192.168.31.79:9200/_cat/indices?v'

health status index pri rep docs.count docs.deleted store.size pri.store.size

green open test 5 1 0 0 1.5kb 795b

green open myindex 5 1 494 0 924.7kb 457.6kb
[root@VS-zsz conf]# curl '192.168.31.78:9200/_cat/indices?v'

health status index pri rep docs.count docs.deleted store.size pri.store.size

green open test 5 1 0 0 1.5kb 795b

green open myindex 5 1 494 0 925kb 457.8kb
[root@VS-zsz conf]# curl '192.168.31.79:9200/_cat/indices?v'

health status index pri rep docs.count docs.deleted store.size pri.store.size

green open test 5 1 0 0 1.5kb 795b

green open myindex 5 1 494 0 925kb 457.8kb

说明数据成功导入，而且在设置了定时任务的情况下， myindex索引的容量不断增加。

5、常见错误：
（1）Pipeline aborted due to error {:exception=>"LogStash::ConfigurationError", :backtrace=>..................stopping pipeline {:id=>"main"}

原因：logstash-mysql.conf 文件配置错误，对于>=2.*的版本， elasticsearch 的参数名应为hosts，如果设置为host则会报错。此处应该是可以配置多个数据源。