mongodb数据同步
2015-12-16 15:42
447 查看
需求:
1.源数据库新增一条记录,目标库同时新增一条记录;
2.源数据库修改一条记录,目标库同时修改该条记录;
示例用到三个Kettle组件
下面详细说下每个组件的配置
Source:
本示例连接的是Mongodb数据库,四个字段,ID默认为主键,_id会系统自动生成暂时先不管。
本节点的详细说明,可见官网:http://wiki.pentaho.com/display/EAI/MongoDB+Input
值映射:
本步在本示例作用不大,只是为了测试效果。按照截图上进行配置即可
MongoDbOutput:
关键是这一步的配置
官网上对这个tab页的解释是这样的:
The MongoDb output step provides a number of options that control what and how data is written to the target Mongo document collection. By default, data is inserted into the target collection. If the specified collection doesn't exist, it will be created before
data is inserted. Selecting the Truncate option will delete any existing data in the target collection before inserting begins. Unless unique indexes are being used (see section on indexing below) then Mongo DB will allow duplicate records
to be inserted. Mongo DB allows for fast bulk insert operations - the batch size can be configured using the Batch insert size field. If no value is supplied here, then the default size of 100 rows is used.
Selecting the Upsert option changes the write mode from insert to upsert (i.e. update if a match is found, otherwise insert a new record). Information on defining how records are matched can be found in the next section. Standard upsert replaces
a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. Modifier update enables modifier ($ operators) to be used to mutate individual fields within matching documents. This type of update
is fast and involves minimal network traffic; it also has the ability to update all matching documents, rather than just the first, if the Multi-update option is enabled
个人理解就是默认都不选择时往目标集合添加数据,如果指定的集合不存在则先创建集合然后插入数据;
Truncate:插入数据之前先把集合里的所有数据删除。(相当于数据删除同步);
Update:修改数据,存在则修改,不存在不操作。
upsert:找到匹配项则修改,找不到匹配项则添加。
勾选上红色圈着的选项之后,源数据修改、添加了,在目标库里都会有相应的操作。不过还要设置下面的一步
ID为主键match field for update时一定要选择Y否则运行时出错。
同步过程最主要的就是上边列出的几步设置,当然如果想要再设置更强大的功能,可详细去研究官网的API
1.源数据库新增一条记录,目标库同时新增一条记录;
2.源数据库修改一条记录,目标库同时修改该条记录;
示例用到三个Kettle组件
下面详细说下每个组件的配置
Source:
本示例连接的是Mongodb数据库,四个字段,ID默认为主键,_id会系统自动生成暂时先不管。
本节点的详细说明,可见官网:http://wiki.pentaho.com/display/EAI/MongoDB+Input
值映射:
本步在本示例作用不大,只是为了测试效果。按照截图上进行配置即可
MongoDbOutput:
关键是这一步的配置
官网上对这个tab页的解释是这样的:
2.2 Selecting the write mode
The MongoDb output step provides a number of options that control what and how data is written to the target Mongo document collection. By default, data is inserted into the target collection. If the specified collection doesn't exist, it will be created beforedata is inserted. Selecting the Truncate option will delete any existing data in the target collection before inserting begins. Unless unique indexes are being used (see section on indexing below) then Mongo DB will allow duplicate records
to be inserted. Mongo DB allows for fast bulk insert operations - the batch size can be configured using the Batch insert size field. If no value is supplied here, then the default size of 100 rows is used.
Selecting the Upsert option changes the write mode from insert to upsert (i.e. update if a match is found, otherwise insert a new record). Information on defining how records are matched can be found in the next section. Standard upsert replaces
a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. Modifier update enables modifier ($ operators) to be used to mutate individual fields within matching documents. This type of update
is fast and involves minimal network traffic; it also has the ability to update all matching documents, rather than just the first, if the Multi-update option is enabled
个人理解就是默认都不选择时往目标集合添加数据,如果指定的集合不存在则先创建集合然后插入数据;
Truncate:插入数据之前先把集合里的所有数据删除。(相当于数据删除同步);
Update:修改数据,存在则修改,不存在不操作。
upsert:找到匹配项则修改,找不到匹配项则添加。
勾选上红色圈着的选项之后,源数据修改、添加了,在目标库里都会有相应的操作。不过还要设置下面的一步
ID为主键match field for update时一定要选择Y否则运行时出错。
同步过程最主要的就是上边列出的几步设置,当然如果想要再设置更强大的功能,可详细去研究官网的API
相关文章推荐
- 分享微信开发Html5轻游戏中的几个坑
- 如何在 Fedora 上安装 MongoDB 服务器
- PHP添加yaf xhprof mongodb 同理
- mongodb安装
- 如何在 Ubuntu 上安装 MongoDB
- perl操作MongoDB报错undefined symbol: HeUTF8解决方法
- C#中使用1.7版本驱动操作MongoDB简单例子
- MongoDB系列教程(四):设置用户访问权限
- php实现的mongodb操作类实例
- 解决mongodb在ubuntu下启动失败,提示couldn‘t remove fs lock errno:9 Bad file descriptor的错误
- 在PostgreSQL的基础上创建一个MongoDB的副本的教程
- java操作mongodb示例分享
- php对mongodb的扩展(初出茅庐)
- 作为PHP程序员应该了解MongoDB的五件事
- 基于MySQL到MongoDB简易对照表的详解
- MongoDB入门教程之C#驱动操作实例
- MongoDB为用户设置访问权限
- MongoDB db.serverStatus()输出内容中文注释
- MongoDB的一些常用查询方法
- mongodb与mysql命令详细对比