从MongoDB同步数据到ElasticSearch 的五种方式
2017-06-22 11:14
851 查看
Elastic
search(ES) is a pop-star for recording and analyzing data, andMongodb is
a famous NoSQL database for storing and querying data. With our web infrastructure improving, how can we export data from mongodb to ES for searching or analyzing purposes? There are 5 possible solutions recommended for your choice.
We can use Mongoosastic module
for storing-in-both-sides purpose when we use Nodejs as a
web server
Container. When one document needs to be stored, Mongoosastic can
commit the changes to both mongo and ES. As the chart below:
Here is the reference link: Mongoosastic.
The advantage is that data can be stored in both mongo and ES simultaneously, and the downside is that overhead may be caused in CUD operation efficiency. And inconsistent data might be generated when one type of the db store failed.
And the server framework is not flexible enough for db migrating.
Transporter tool
is a good choice to synchronize data once you want to export mongo data to another ES server. Transporter also
can export data from or to other type of data store. Reference link is: Transporter.
It's
important to know that the transporter synchronizing only once. When the job is done, the transporter comes to its end.
There is a plugin for ES named "elasticsearch-river-mongodb", and was widely used in ES 1.x, but now river mechanism for ES 2.x is deprecated. Reference link is elasticsearch-river-mongodb.
We can
take advantage of buffering , inputting, outputting and filtering abilities from logstash by adding
a mongo input and ES output plugin to get this job done.
JDBC input plugin is one of the choices, but it needs JDBC driver support. As I know there is no well-supported-free JDBC driver for mongo. Some trial versions can be found in Unity or Simba.
Reference link is : JDBC
Plugin for Logstash
mongo-connector is
a real-time sync service as a package ofPython. It creates a pipeline from
a mongodb cluster to one or more target systems. It needs mongo to run in replica-set mode, sync data in mongo to the target then tails the mongo oplog. It needs a package named "elastic2_doc_manager" to write data to ES. Process chart below:
Reference link is : github or python.
To recapitulate it, it is a must to remember: mongo replica set, an opened port and IP for ES, using elastic2_doc_manager if you
use ES
2.x.
At present, I am not yet ready with any official support in Beats.
it will be in the future.
So that's all, 5 ways to mongo-ES-sync.
注:本文转自https://www.linkedin.com/pulse/5-way-sync-data-from-mongodb-es-kai-hao
search(ES) is a pop-star for recording and analyzing data, andMongodb is
a famous NoSQL database for storing and querying data. With our web infrastructure improving, how can we export data from mongodb to ES for searching or analyzing purposes? There are 5 possible solutions recommended for your choice.
1.synchronized by web server
We can use Mongoosastic modulefor storing-in-both-sides purpose when we use Nodejs as a
web server
Container. When one document needs to be stored, Mongoosastic can
commit the changes to both mongo and ES. As the chart below:
Here is the reference link: Mongoosastic.
The advantage is that data can be stored in both mongo and ES simultaneously, and the downside is that overhead may be caused in CUD operation efficiency. And inconsistent data might be generated when one type of the db store failed.
And the server framework is not flexible enough for db migrating.
2.Manually loading data from Mongo to ES
Transporter toolis a good choice to synchronize data once you want to export mongo data to another ES server. Transporter also
can export data from or to other type of data store. Reference link is: Transporter.
It's
important to know that the transporter synchronizing only once. When the job is done, the transporter comes to its end.
3. Plugin for ES
There is a plugin for ES named "elasticsearch-river-mongodb", and was widely used in ES 1.x, but now river mechanism for ES 2.x is deprecated. Reference link is elasticsearch-river-mongodb.
4.
JDBC input plugin for logstash
We cantake advantage of buffering , inputting, outputting and filtering abilities from logstash by adding
a mongo input and ES output plugin to get this job done.
JDBC input plugin is one of the choices, but it needs JDBC driver support. As I know there is no well-supported-free JDBC driver for mongo. Some trial versions can be found in Unity or Simba.
Plugin for Logstash
5. Mongo-ES connector
mongo-connector isa real-time sync service as a package ofPython. It creates a pipeline from
a mongodb cluster to one or more target systems. It needs mongo to run in replica-set mode, sync data in mongo to the target then tails the mongo oplog. It needs a package named "elastic2_doc_manager" to write data to ES. Process chart below:
Reference link is : github or python.
To recapitulate it, it is a must to remember: mongo replica set, an opened port and IP for ES, using elastic2_doc_manager if you
use ES
2.x.
At present, I am not yet ready with any official support in Beats.
it will be in the future.
So that's all, 5 ways to mongo-ES-sync.
注:本文转自https://www.linkedin.com/pulse/5-way-sync-data-from-mongodb-es-kai-hao
相关文章推荐
- 五种方式将数据从MongoDB同步到ElasticSearch
- Elasticsearch中的索引数据Index同步到Mongodb中的集合collection中
- elasticsearch与mongodb分布式集群环境下数据同步
- ES-MongoDB学习1_elasticsearch与mongodb分布式集群环境下数据同步以及遇到的相应问题解答
- Linux安装ElasticSearch与MongoDB分布式集群环境下数据同步
- elasticsearch与mongodb分布式集群环境下数据同步以及遇到的相应问题解答
- Mongodb数据库导入数据到elasticsearch,mongo-connector实现mongodb与elastic数据同步
- ES-MongoDB学习3_MongoDB数据同步到Elasticsearch中
- MongoDB 数据自动同步到 ElasticSearch
- Elasticsearch与MongoDB 数据同步及分布式集群搭建 (一)
- Elasticsearch与MongoDB 数据同步及分布式集群搭建(一)
- elasticsearch与mongodb分布式集群环境下数据同步
- logstash-out-mongodb实现elasticsearch到Mongodb的数据同步
- 分布式搜索elasticsearch java API 之(七)------与MongoDB同步数据
- 10008---Elasticsearch与mongodb分布式集群环境下数据同步
- Elasticsearch与MongoDB 数据同步及分布式集群搭建 (二)
- MongoDB与Elasticsearch达到数据同步的一点尝试
- python实现MongoDB数据同步到ES(Elasticsearch)
- 基于netcore实现mongodb和ElasticSearch之间的数据实时同步的工具(Mongo2Es)