oracle迁移到mysql分库分表方案之——ogg(goldengate)
2018-06-28 11:15
435 查看
之前文章主要介绍了oracle 迁移到mysql,主要是原表原结构迁移,但是实际运维中会发现,到mysql以后需要分库和分表的拆分操作,这个时候,用ogg来做,也是很强大好用的。
主要结合ogg的2个参数
参数1:filter
Use a FILTER clause to select rows based on a numeric value by using basic operators or one or more Oracle GoldenGate column-conversion functions.
NOTE To filter a column based on a string, use one of the Oracle GoldenGate string
functions or use a WHERE clause.
Syntax TABLE ,
, FILTER (
[, ON INSERT | ON UPDATE| ON DELETE]
[, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE]
, );
Or...
Syntax MAP
, TARGET
,
, FILTER (
[, ON INSERT | ON UPDATE| ON DELETE]
[, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE]
[, RAISEERROR ]
, );
Valid FILTER clause elements are the following:
An Oracle GoldenGate column-conversion function. These functions are built into
Oracle GoldenGate so that you can perform tests, manipulate data, retrieve values,
and so forth. For more information about Oracle GoldenGate conversion functions, see
“Testing and transforming data” on page 158.
Numbers
Columns that contain numbers
Functions that return numbers
Arithmetic operators:
(plus)
(minus)
(multiply)
/ (divide)
\ (remainder)
Comparison operators:
(greater than)
= (greater than or equal)
< (less than)
<= (less than or equal)
= (equal)
<> (not equal)
Results derived from comparisons can be zero (indicating FALSE) or non-zero (indicating TRUE).
Parentheses (for grouping results in the expression)
Conjunction operators: AND, OR
下面是官方给出的几个例子:
Example 1 The following calls the @COMPUTE function to extract records in which the price multiplied by the amount exceeds 10,000.
MAP SALES.TCUSTORD, TARGET SALES.TORD,
FILTER (@COMPUTE (PRODUCT_PRICE*PRODUCT_AMOUNT) > 10000);
Example 2 The following uses the @STREQ function to extract records where a string is equal to ’JOE’.This example assumes that the USEANSISQLQUOTES parameter is used in the GLOBALS parameter file to apply SQL-92 rules for single and double quote marks.
TABLE ACCT.TCUSTORD, FILTER (@STREQ ("Name", ’joe’) > 0);
Example 3 The following selects records in which the amount column is greater than 50 and executes the filter on updates and deletes.
TABLE ACT.TCUSTORD, FILTER (ON UPDATE, ON DELETE, AMOUNT > 50);
Example 4 You can use the @RANGE function to divide the processing workload among multiple FILTER clauses, using separate TABLE or MAP statements. For example, the following splits the replication workload into two ranges (between two Replicat processes) based on the ID column of the source acct table.
Note that object names are case-sensitive in this case. (Replicat group 1 parameter file)
MAP "sales"."acct", TARGET "sales"."acct", FILTER (@RANGE (1, 2, ID));
(Replicat group 2 parameter file)
MAP "sales"."acct", TARGET "sales"."acct", FILTER (@RANGE (2, 2, ID));
参数2:COMPUTE
Use the @COMPUTE function to return the value of an arithmetic expression to a target column. The value returned from the function is in the form of a string.
You can omit the @COMPUTE phrase when returning the value of an arithmetic expression to another Oracle GoldenGate function, as in:
@STRNUM ((AMOUNT1 + AMOUNT2), LEFT)
The preceding returns the same result as:
@STRNUM ((@COMPUTE (AMOUNT1 + AMOUNT2), LEFT)
Arithmetic expressions can be combinations of the following elements.
Numbers
The names of columns that contain numbers
Functions that return numbers
Arithmetic operators:
(plus)
(minus)
(multiply)
/ (divide)
\ (remainder)
Comparison operators:
(greater than)
= (greater than or equal)
< (less than)
<= (less than or equal)
= (equal)
<> (not equal)
Results that are derived from comparisons can be zero (indicating FALSE) or non-zero (indicating TRUE).
Parentheses (for grouping results in the expression)
The conjunction operators AND, OR. Oracle GoldenGate only evaluates the necessary part of a conjunction expression. Once a statement is FALSE, the rest of the expression is ignored. This can be valuable when evaluating fields that may be missing or null. For example, if the value of COL1 is 25 and the value of COL2 is 10, then the following are possible:
@COMPUTE (COL1 > 0 AND COL2 < 3) returns 0.
@COMPUTE (COL1 < 0 AND COL2 < 3) returns 0. COL2 < 3 is never evaluated.
@COMPUTE ((COL1 + COL2)/5) returns 7.
Syntax
@COMPUTE (expression)
expression
A valid arithmetic expression. The numeric value plus the precision cannot be greater than 17 digits. If this limit is exceeded, @COMPUTE returns an error similar to the following.
2013-08-01 01:54:22 ERROR OGG-01334 Error mapping data from column to column in function COMPUTE.
Examples
Example 1
AMOUNT_TOTAL = @COMPUTE (AMT + AMT2)
Example 2
AMOUNT_TOTAL = @IF (AMT >= 0, AMT 100, 0)
Example 3
ANNUAL_SALARY = @COMPUTE (MONTHLY_SALARY 12)
2个参数的使用方法上面介绍了,下面进行分库分表
根据某业务id(sale_prod_id)进行分库分表—— 此id非主键
源端:scott.sale_date表
目标端:多库多表:
d_sale0.sale_date
d_sale1.sale_date
d_sale2.sale_date
d_sale3.sale_date
d_sale4.sale_date
d_sale5.sale_date
抽取投递进程参考上一文章,主要改变的就是应用进程的map
map scott.sale_date,target d_sale0.sale_date,FILTER (@compute( sale_prod_id \ 5)=0);
map scott.sale_date,target d_sale1.sale_date,FILTER (@compute( sale_prod_id \ 5)=1);
map scott.sale_date,target d_sale2.sale_date,FILTER (@compute( sale_prod_id \ 5)=2);
map scott.sale_date,target d_sale3.sale_date,FILTER (@compute( sale_prod_id \ 5)=3);
map scott.sale_date,target d_sale4.sale_date,FILTER (@compute( sale_prod_id \ 5)=4);
初始化未发现任何问题,实时同步发现异常dml不同步。
最终解决方案在源端:add trandata scott.sale_date COLS(sale_prod_id)----其中sale_prod_id就是分表的字段
大致原因是由于ogg同步主要以主键或者唯一键为同步基础,而此案例分表键并非主键。所以同步的时候无法以此分表键进行数据分表。
主要结合ogg的2个参数
参数1:filter
Use a FILTER clause to select rows based on a numeric value by using basic operators or one or more Oracle GoldenGate column-conversion functions.
NOTE To filter a column based on a string, use one of the Oracle GoldenGate string
functions or use a WHERE clause.
Syntax TABLE ,
, FILTER (
[, ON INSERT | ON UPDATE| ON DELETE]
[, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE]
, );
Or...
Syntax MAP
, TARGET
,
, FILTER (
[, ON INSERT | ON UPDATE| ON DELETE]
[, IGNORE INSERT | IGNORE UPDATE | IGNORE DELETE]
[, RAISEERROR ]
, );
Valid FILTER clause elements are the following:
An Oracle GoldenGate column-conversion function. These functions are built into
Oracle GoldenGate so that you can perform tests, manipulate data, retrieve values,
and so forth. For more information about Oracle GoldenGate conversion functions, see
“Testing and transforming data” on page 158.
Numbers
Columns that contain numbers
Functions that return numbers
Arithmetic operators:
(plus)
(minus)
(multiply)
/ (divide)
\ (remainder)
Comparison operators:
(greater than)
= (greater than or equal)
< (less than)
<= (less than or equal)
= (equal)
<> (not equal)
Results derived from comparisons can be zero (indicating FALSE) or non-zero (indicating TRUE).
Parentheses (for grouping results in the expression)
Conjunction operators: AND, OR
下面是官方给出的几个例子:
Example 1 The following calls the @COMPUTE function to extract records in which the price multiplied by the amount exceeds 10,000.
MAP SALES.TCUSTORD, TARGET SALES.TORD,
FILTER (@COMPUTE (PRODUCT_PRICE*PRODUCT_AMOUNT) > 10000);
Example 2 The following uses the @STREQ function to extract records where a string is equal to ’JOE’.This example assumes that the USEANSISQLQUOTES parameter is used in the GLOBALS parameter file to apply SQL-92 rules for single and double quote marks.
TABLE ACCT.TCUSTORD, FILTER (@STREQ ("Name", ’joe’) > 0);
Example 3 The following selects records in which the amount column is greater than 50 and executes the filter on updates and deletes.
TABLE ACT.TCUSTORD, FILTER (ON UPDATE, ON DELETE, AMOUNT > 50);
Example 4 You can use the @RANGE function to divide the processing workload among multiple FILTER clauses, using separate TABLE or MAP statements. For example, the following splits the replication workload into two ranges (between two Replicat processes) based on the ID column of the source acct table.
Note that object names are case-sensitive in this case. (Replicat group 1 parameter file)
MAP "sales"."acct", TARGET "sales"."acct", FILTER (@RANGE (1, 2, ID));
(Replicat group 2 parameter file)
MAP "sales"."acct", TARGET "sales"."acct", FILTER (@RANGE (2, 2, ID));
参数2:COMPUTE
Use the @COMPUTE function to return the value of an arithmetic expression to a target column. The value returned from the function is in the form of a string.
You can omit the @COMPUTE phrase when returning the value of an arithmetic expression to another Oracle GoldenGate function, as in:
@STRNUM ((AMOUNT1 + AMOUNT2), LEFT)
The preceding returns the same result as:
@STRNUM ((@COMPUTE (AMOUNT1 + AMOUNT2), LEFT)
Arithmetic expressions can be combinations of the following elements.
Numbers
The names of columns that contain numbers
Functions that return numbers
Arithmetic operators:
(plus)
(minus)
(multiply)
/ (divide)
\ (remainder)
Comparison operators:
(greater than)
= (greater than or equal)
< (less than)
<= (less than or equal)
= (equal)
<> (not equal)
Results that are derived from comparisons can be zero (indicating FALSE) or non-zero (indicating TRUE).
Parentheses (for grouping results in the expression)
The conjunction operators AND, OR. Oracle GoldenGate only evaluates the necessary part of a conjunction expression. Once a statement is FALSE, the rest of the expression is ignored. This can be valuable when evaluating fields that may be missing or null. For example, if the value of COL1 is 25 and the value of COL2 is 10, then the following are possible:
@COMPUTE (COL1 > 0 AND COL2 < 3) returns 0.
@COMPUTE (COL1 < 0 AND COL2 < 3) returns 0. COL2 < 3 is never evaluated.
@COMPUTE ((COL1 + COL2)/5) returns 7.
Syntax
@COMPUTE (expression)
expression
A valid arithmetic expression. The numeric value plus the precision cannot be greater than 17 digits. If this limit is exceeded, @COMPUTE returns an error similar to the following.
2013-08-01 01:54:22 ERROR OGG-01334 Error mapping data from column to column in function COMPUTE.
Examples
Example 1
AMOUNT_TOTAL = @COMPUTE (AMT + AMT2)
Example 2
AMOUNT_TOTAL = @IF (AMT >= 0, AMT 100, 0)
Example 3
ANNUAL_SALARY = @COMPUTE (MONTHLY_SALARY 12)
2个参数的使用方法上面介绍了,下面进行分库分表
根据某业务id(sale_prod_id)进行分库分表—— 此id非主键
源端:scott.sale_date表
目标端:多库多表:
d_sale0.sale_date
d_sale1.sale_date
d_sale2.sale_date
d_sale3.sale_date
d_sale4.sale_date
d_sale5.sale_date
抽取投递进程参考上一文章,主要改变的就是应用进程的map
map scott.sale_date,target d_sale0.sale_date,FILTER (@compute( sale_prod_id \ 5)=0);
map scott.sale_date,target d_sale1.sale_date,FILTER (@compute( sale_prod_id \ 5)=1);
map scott.sale_date,target d_sale2.sale_date,FILTER (@compute( sale_prod_id \ 5)=2);
map scott.sale_date,target d_sale3.sale_date,FILTER (@compute( sale_prod_id \ 5)=3);
map scott.sale_date,target d_sale4.sale_date,FILTER (@compute( sale_prod_id \ 5)=4);
初始化未发现任何问题,实时同步发现异常dml不同步。
最终解决方案在源端:add trandata scott.sale_date COLS(sale_prod_id)----其中sale_prod_id就是分表的字段
大致原因是由于ogg同步主要以主键或者唯一键为同步基础,而此案例分表键并非主键。所以同步的时候无法以此分表键进行数据分表。
相关文章推荐
- Oracle迁移到MySQL方案之——ogg(goldengate)
- 【翻译自mos文章】怎么检查Oracle GoldenGate(OGG)的checkpoint file 是否被locked?
- 推测Oracle GoldenGate(OGG)目的端 rep进程Checkpoint Lag的计算方法
- Oracle Goldengate数据库复制与容灾项目实施与维护_OGG实战培训
- Oracle Goldengate OGG 11g与各操作系统及数据库版本的兼容列表
- 从SQL Server 向Oracle 8迁移的技术实现方案
- Oracle goldengate的OGG-01004 OGG-1296错误
- 从SQL SERVER向ORACLE的迁移方案[转] PartII
- 【GoldenGate】使用OGG,两个Oracle库之间单向同步数据
- Oracle GoldenGate (以下简称ogg)在异种移植os同一种db之间的数据同步。
- 从SQL SERVER 向ORACLE 8迁移的技术实现方案
- 从SQLSERVER向ORACLE8迁移的技术实现方案 PartIV [转]
- 对一段Oracle GoldenGate (OGG) 传输进程日志(.rpt文件)的解释
- Oracle GoldenGate的思想天空--感悟OGG的本质
- 【OGG】GoldenGate For Oracle实施文档 --超详细总结版
- 改动Oracle GoldenGate(ogg)各个进程的读检查点和写检查点
- Oracle GoldenGate学习之windows下ogg单向复制配置
- Oracle goldengate ogg00869 ogg01431 ogg1003
- Oracle--ogg(goldengate)
- oracle goldengate(ogg)windows运行ggsci.exe报错找不到OCI.dll