go-mysql-elasticsearch实现mysql 与elasticsearch实时同步深入详解
2016-06-27 21:36
1061 查看
引言:
go-mysql-elasticsearch 是国内作者开发的一款插件。测试表明:该插件优点:能实现同步增、删、改、查操作。不足之处(待完善的地方):1、仍处理开发、相对不稳定阶段;
2、没有日志,不便于排查问题及查看同步结果。
本文深入详解了插件的安装、使用、增删改查同步测试。
1. go-mysql-elasticsearch 插件安装
步骤1:安装go
yum install go步骤2:安装godep
go get github.com/tools/godep步骤3:获取go-mysql-elastisearch插件
go get github.com/siddontang/go-mysql-elasticsearch步骤4:安装go-mysql-elastisearch插件
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearchmake
2.go-mysql-elasticsearch 插件使用
2.1修改配置文件
[root@5b9dbaaa148a etc]# cat river.toml # MySQL address, user and password # user must have replication privilege in MySQL. my_addr = "192.168.1.1:3306" my_user = "root" my_pass = "password@!" # Elasticsearch address es_addr = "192.168.1.1:9200" # Path to store data, like master.info, and dump MySQL data data_dir = "./var" # Inner Http status address stat_addr = "192.168.1.1:12800" # pseudo server id like a slave server_id = 1 # mysql or mariadb flavor = "mysql" # mysqldump execution path mysqldump = "mysqldump" # MySQL data source [[source]] schema = "test" # Only below tables will be synced into Elasticsearch. # "test_river_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023 # I don't think it is necessary to sync all tables in a database. tables = ["cc"] # Below is for special rule mapping #[[rule]] #schema = "test" #table = "cc" #index = "go_river" #type = "go_rivert" # title is MySQL test_river field name, es_title is the customized name in Elasticsearch # [rule.field] # This will map column title to elastic search my_title # title="es_title" # This will map column tags to elastic search my_tags and use array type # tags="my_tags,list" # This will map column keywords to elastic search keywords and use array type #keywords=",list" # wildcard table rule, the wildcard table must be in source tables [[rule]] schema = "test" table = "cc" index = "gocc" type = "gocc_t" # title is MySQL test_river field name, es_title is the customized name in Elasticsearch [[rule.fields]] mysql = "mysql101" elastic = "es_mysql101"
2.2执行同步操作
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch./bin/go-mysql-elasticsearch -config=./etc/river.toml
3. go-mysql-elasticsearch 插件同步测试结果
3.1插入Insert操作实时同步验证(验证ok)
3.1.1Mysql端插入操作
mysql> insert into cc(id,name) values(12, ‘test12’);Query OK, 1 row affected (0.06 sec)
3.1.2Mysql执行insert后查询结果
mysql> select * from cc where id =12;+—-+——–+——–+———————+
| id | name | status | modified_at |
+—-+——–+——–+———————+
| 12 | test12 | ok | 2016-06-24 02:27:29 |
+—-+——–+——–+———————+
1 row in set (0.02 sec)
3.1.3ES端能查询到新增的value字段。
[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d ' > {"query": > {"term": > {"id":12}}}' { "took" : 402, "timed_out" : false, "_shards" : { "total" : 8, "successful" : 8, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "gocc", "_type" : "gocc_t", "_id" : "12", "_score" : 1.0, "_source" : { "id" : 12, "modified_at" : "2016-06-24T02:27:29+01:00", "name" : "test12", "status" : "ok" } } ] } }
3.2修改Update操作实时同步验证(验证ok)
3.2.1mysql执行更新操作
mysql> update cc set name = 'test12_001' where id = 12; Query OK, 1 row affected (0.05 sec) Rows matched: 1 Changed: 1 Warnings: 0
3.2.2mysql执行修改后查询
Mysql查询修改后结果:mysql> select * from cc where id = 12; +----+------------+--------+---------------------+ | id | name | status | modified_at | +----+------------+--------+---------------------+ | 12 | test12_001 | ok | 2016-06-24 02:27:29 | +----+------------+--------+---------------------+ 1 row in set (0.00 sec)
3.2.3 ES查询修改结果
[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d ' {"query": {"term": {"id":12}}}' { "took" : 59, "timed_out" : false, "_shards" : { "total" : 8, "successful" : 8, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "gocc", "_type" : "gocc_t", "_id" : "12", "_score" : 1.0, "_source" : { "id" : 12, "modified_at" : "2016-06-24T02:27:29+01:00", "name" : "test12_001", "status" : "ok" } } ] } }
3.3删除操作实时同步验证
3.3.1Mysql执行删除操作
mysql> delete from cc where id = 12; Query OK, 1 row affected (0.04 sec)
3.3.2删除后查询表
mysql> select * from cc; +----+--------------------+--------+---------------------+ | id | name | status | modified_at | +----+--------------------+--------+---------------------+ | 1 | laoyang360 | ok | 0000-00-00 00:00:00 | | 2 | test002 | ok | 2016-06-23 06:16:42 | | 3 | dlllaoyang360 | ok | 0000-00-00 00:00:00 | | 11 | test11 | ok | 2016-06-24 02:09:15 | | 5 | jdbc_test_update08 | ok | 0000-00-00 00:00:00 | | 7 | test7 | ok | 0000-00-00 00:00:00 | | 8 | test008 | ok | 0000-00-00 00:00:00 | | 9 | test009 | ok | 0000-00-00 00:00:00 | | 10 | test10 | ok | 2016-06-24 02:08:14 | +----+--------------------+--------+---------------------+ 9 rows in set (0.02 sec)
3.3.3ES查询删除后结果
[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d ' {"query": {"term": {"id":12}}}' { "took" : 40, "timed_out" : false, "_shards" : { "total" : 8, "successful" : 8, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } }
4小结
验证发现:(1)go-mysql-elasticsearch插件可以实现同步insert、update、delete操作。
(2)可视化做的不好,没有打印日志。
(3)go-mysql-elasticsearch尚不大稳定,出现过无法同步成功的情况,但没有报错。不便于排查。
——————————————————————————————————
更多ES相关实战干货经验分享,请扫描下方【铭毅天下】微信公众号二维码关注。
(每周至少更新一篇!)
和你一起,死磕Elasticsearch!
——————————————————————————————————
作者:铭毅天下
转载请标明出处,原文地址:http://blog.csdn.net/laoyang360/article/details/51771483
如果感觉本文对您有帮助,请点击‘顶’支持一下,您的支持是我坚持写作最大的动力,谢谢!
相关文章推荐
- postgresql----COPY之表与文件之间的拷贝
- orcale装完sqldevelop启动不了
- SQLiteOpenHelper 类
- 14. ubuntu 16.04 LTS 安装sqldeveloper
- MoNyog 5.72 mysql监控工具的介绍
- 分布式监控系统Zabbix-3.0.3-完整安装记录(2)-添加mysql监控
- MySQL server has gone away
- MySQL的多存储引擎架构
- SQL语句中的 truncate 、delete与drop区别
- MySQL的多存储引擎架构
- 蛋疼的mysql_ping()以及MYSQL_OPT_RECONNECT
- 在线建立或重做mysql主从复制架构方法(传统模式和GTID模式)
- sql语句----drop、truncate和delete 精屁解析--终于知道ID为啥有时候不连续了
- PL/SQL developer里面查dblink会要commit
- Oracle GoldenGate for Sql Server连接ODBC失败的处理方法
- PL/SQL Developer报错Dynamic Performance Tables not accessible
- SQL Server 2005架构
- SQL Server 2005架构
- hive内部表、外部表
- ubuntu环境-Django下快速安装和配置MySQL