使用 Tungsten Replicator4 实现 mysql -- > mongodb 数据同步

摘要: tungsten-replicator 是基于binlog的开源同步工具，其大概原理是 从mysql 端抓取 binlog日志，写到自己的 thl (transaction history log)中，再由 thl 发送到 nosql（mongo） 主机上thl 中，nosql 主机上的 thl 再写到 nosql 中。

tungsten-replicator 技术交流群：376068310

一 规划主机：

192.168.67.128 #mysql主机 (事先安装好，mysql5.6 )
192.168.67.129 #mongodb主机 （事先安装好，mongodb3.0.6）

二 前置条件：（128和129）

1 创建tungsten用户

[root@localhost ~] useradd tungsten
[root@localhost ~] passwd tungsten

2 将tungsten账户添加到 mysql 用户组里

128主机：
[root@localhost ~] sudo usermod -G mysql tungsten

129主机
[root@localhost ~] sudo usermod -G mongod tungsten

3 修改sudoers权限，并且让 tungsten 用户使用 sudo 命令时免密码

[root@localhost ~] chmod 740 /etc/sudoers
[root@localhost ~] vi /etc/sudoers

添加如下一行：
tungsten        ALL=(ALL)       NOPASSWD: ALL

4 切换到 tungsten 用户下

[root@localhost ~] su - tungsten

5 切换到 tungsten 用户后，创建密钥对。默认在/home/tungsten/.ssh 目录下

[tungsten@localhost ~] ssh-keygen -t rsa

6 修改 limit 文件

[tungsten@localhost ~] vi /etc/security/limits.conf

添加下面一行：
* - nofile 65535

注：这种方式需要重启才能生效，使用如下方式修改ulimit，则可以即时生效：

[tungsten@localhost ~] ulimit -n65535

7 检查修改是否成功

[tungsten@localhost ~] ulimit -n
65535

8 配置hosts

[tungsten@localhost ~] sudo vi /etc/hosts
127.0.0.1 localhost
192.168.67.128 host1
192.168.67.129 host2

9 测试主机

[tungsten@localhost ~] uname -n
host1

10 测试ip地址

[tungsten@localhost ~] hostname --ip-address

11 防火墙设置

[tungsten@localhost ~] iptables -A INPUT -i lo -m state --state NEW -j ACCEPT

12 把生成的公钥和私钥拷贝到 tungsten用户的主目录下 。默认就在用户主目录

[tungsten@localhost ~] cp .ssh/id_rsa.pub .ssh/id_rsa /home/tungsten/.ssh

13 把公钥放到 .ssh/authorized_keys文件中，128 的公钥放到 129，129 的公钥放到 128

[tungsten@localhost ~] cat .ssh/id_rsa.pub >> .ssh/authorized_keys

14 修改权限

[tungsten@localhost ~] chmod 700 ~/.ssh
[tungsten@localhost ~] chmod 600 ~/.ssh/*

15 测试主机间ssh

[tungsten@localhost ~] ssh tungsten@host2

16 修改mysql配置，注意，binlog-fomat 必须 是 row 格式的，如果该mysql 是 slave，则需要在 my.cnf 里 加上 log-slave-updates=TRUE，否则，slave 的变化信息都在 relay-relay-bin.0000*.log 文件中

[tungsten@localhost ~] vi /etc/my.cnf

server-id = 128
open_files_limit = 65535
log-bin = mysql-bin
sync_binlog = 1
max_allowed_packet = 52m
default-storage-engine = InnoDB
innodb_flush_log_at_trx_commit = 2
binlog-format = row
binlog-row-image = full

17 登录mysql，创建用于 tungsten 复制的 mysql 用户

[tungsten@localhost ~] /usr/bin/mysql

mysql> grant all privileges on *.* to tungsten_alpha@'%' identified by 'ts' with grant option;
mysql> flush privileges;

三 安装服务：

1 创建 tungsten-replication 软件主目录

[tungsten@localhost ~] sudo mkdir -p /opt/continuent/software
[tungsten@localhost ~] sudo chown -R tungsten /opt/continuent
[tungsten@localhost ~] sudo chmod 700 /opt/continuent

2 把下载好的 tungsten-replicator-4.0.0-2667425.tar.gz 包上传到 software 目录

[tungsten@localhost ~] cd /opt/continuent/software
[tungsten@host1 software] ll
[tungsten@host1 software] tungsten-replicator-4.0.0-2667425.tar.gz
[tungsten@host1 software] tar -zxvf tungsten-replicator-4.0.0-2667425.tar.gz
[tungsten@host1 software] cd tungsten-replicator-4.0.0-2667425
[tungsten@host1 tungsten-replicator-4.0.0-2667425]

128主机（mysql）上：

[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tools/tpm install alpha \
--info \
--master=192.168.67.128 \
--datasource-mysql-conf=/etc/my.cnf \
--enable-heterogenous-master=true \
--replication-user=tungsten_alpha \
--replication-password=ts \
--install-directory=/opt/continuent/alpha \
--start-and-report

129主机（mongo）上：

[tungsten@host2 tungsten-replicator-4.0.0-2667425]./tools/tpm install alpha \
--info \
--datasource-type=mongodb \
--master=192.168.67.128 \
--members=192.168.67.129 \
--enable-heterogenous-slave=true \
--topology=master-slave \
--install-directory=/opt/continuent/alpha \
--start-and-report

3 如果两个主机的 state 都是 online 说明 两个主机的 tungsten-replicator 都正常运行了，通过下面的命令查看 tungsten 状态

[tungsten@host2 tungsten-replicator-4.0.0-2667425] ./tungsten-replicator/bin/trepctl status

Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000006:0000000000014751;-1
appliedLastSeqno       : 26
appliedLatency         : 54.232
autoRecoveryEnabled    : false
autoRecoveryTotal      : 0
channels               : 1
clusterName            : huobi
currentEventId         : mysql-bin.000006:0000000000014751
currentTimeMillis      : 1446648043958
dataServerHost         : 192.168.67.128
extensions             :
host                   : 192.168.67.128
latestEpochNumber      : 2
masterConnectUri       : thl://localhost:/
masterListenUri        : thl://192.168.67.128:2112/
maximumStoredSeqNo     : 26
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : jdbc:mysql:thin://192.168.67.128:3306/tungsten_alpah
relativeLatency        : 749.958
resourcePrecedence     : 99
rmiPort                : 10000
role                   : master
seqnoType              : java.lang.Long
serviceName            : huobi
serviceType            : local
simpleServiceName      : huobi
siteName               : default
sourceId               : 192.168.67.128
state                  : ONLINE
timeInStateSeconds     : 845.783
timezone               : GMT
transitioningTo        :
uptimeSeconds          : 849.608
useSSLConnection       : false
version                : Tungsten Replicator 5.0.0
Finished status command...

4 上一步我们通过 状态命令可以看到 128 tungsten 默认抓取 mysql 的 tungsten_alpha 库，该库的命名规则是 tungsten_服务名，比如，我们 install 的时候发布的服务名字为 alpha，那么安装完成后，mysql 中会自动创建一个 名称 为 tungsten_alpha的库

5 因为业务需要，我们需要同步自己已有的库，通过下面命令来指定具体的库

[tungsten@host2 tungsten-replicator-4.0.0-2667425] ./tools/tpm update alpha --dataservice-schema=huobi

注：alpha 是install时指定的 dataservice 名称

6 又因为业务需要，我们可能只同步指定的库的 某些表的数据，非不是全部表，做如下配置：

ON Mysql (128主机上)

[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tools/tpm update huobi \
--repl-svc-extractor-filters=replicate \
--property=replicator.filter.replicate.do=huobi.table1,huobi.table2,huobi.table3

注：mysql端，即数据抓取端，使用 repl-svc-extractor-filters 过滤器

On Mongo(129主机上)

[tungsten@host2 tungsten-replicator-4.0.0-2667425] ./tools/tpm update huobi \
--repl-svc-applier-filters=replicate \
--property=replicator.filter.replicate.do=huobi.table1,huobi.table2,huobi.table3

注：mongo端，即数据重入端，使用 repl-svc-applier-filters 过滤器

7 关于tungsten 工具的使用可以通过命令查看

[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tools/tpm -h

Usage: tpm help [commands,config-file,template-file] [general-options] [command-options]
...........
...........
----------------------------------------------------------------------------------------------------
Commands:
configure                     Update the data service settings in the global configuration
delete-service                Delete a replication service from this host
diag                          Package diagnostic information from the cluster to the current machine
diff                          Output the manual changes that have been made to configuration files.
dump                          Display the 'tpm configure' commands required to recreate the configuration
firewall                      Print firewall information for the listed dataservices
help                          Display a list of all commands available in tpm
install                       Install Tungsten with the current configuration and any options specified at runtime.
mysql                         Open a terminal to the DBMS
promote                       Take a previously prepared directory and make it active.  You should run the prepare command prior to running promote.
query                         Get information about the configuration, topology and version of this directory
reset                         Reset the cluster on each host
reset-thl                     Reset the THL on each host
update
Updates an existing installation of Tungsten. If not arguments are specified, the local configuration is used to install. If you specify --user, --hosts and --directory; this command will get the
current configuration from each host and continue.
validate                      Validate Tungsten with the current configuration and any options specified at runtime before installing.
validate-update               Validate before updating an existing Tungsten installation

8 在使用tungsten同步数据时，如果因为tungsten-replicator服务挂掉，那么tungsten服务重启的时候回从挂断点的地方继续开始同步。而针对master的tungsten服务，如果想指定binlog的位点，可以如下：

[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tungsten-replicator/bin/trepctl offline
[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tungsten-replicator/bin/trepctl online -from-event mysql-bin.000006:1029

9 在mysql主机上的tungsten服务中，如果想查看THL中的mysql的binlog文件的位点同步到哪里了，则可以使用如下命令：

[tungsten@host1 tungsten-replicator-4.0.0-2667425] ./tungsten-replicator/bin/thl list

........................................
SEQ# = 207 / FRAG# = 0 (last frag)
- TIME = 2015-11-08 08:43:00.0
- EPOCH# = 195
- EVENTID = mysql-bin.000006:0000000000092292;-1
- SOURCEID = 192.168.67.128
- METADATA = [mysql_server_id=141;dbms_type=mysql;tz_aware=true;strings=utf8;service=alpha;shard=huobi]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [foreign_key_checks = 1, unique_checks = 1, time_zone = '+00:00']
- SQL(0) =
- ACTION = UPDATE
- SCHEMA = alpha
- TABLE = order
- ROW# = 0
- COL(1: id) = 115347
- COL(2: user_id) = 410
- COL(3: order_amount) = 10000000000
- COL(4: order_price) = 110000000000
- COL(5: order_time) = 1446972178
- COL(6: processed_amount) = 0
- COL(7: last_processed_time) = 1446972178
- COL(8: fee_rate) = 0
- COL(9: type) = 1
- COL(10: status) = 3
- COL(11: is_sys) = 18
- KEY(1: id) = 115347