用spring做框架,实现通用数据库倒表工具

在数据采集中经常会遇到表数据复制加工, 该工具就是为了通过配置可以实现大部分的库表采集任务:/

使用场景:

1. 把数据库DB1中A表的一些数据复制数据库DB2中的B表

2. 其中A表结构不一定与B表中结构完全一样,因为有可能只要倒A表中的部分字段倒B表中

3. 对B表可能还要一些加工,如果更新一些字段,删除历史数据等,这些任务之间有先后顺序

抽象对象:

对象分成;数据库采集器, 采集任务,采集步骤, 采集器下可能会有多个的采集任务,单个采集任务又可能又多个的采集步骤组成.所以采集器跟采集任务,采集步骤之间建立是有先后关系,采集任务之间,采集步骤之间也是有先后关系的,由于这个关系联想到spring配置中的bean可以指定之间的依赖关系.

实现:

1.配置文件设计:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//Spring//DTD Bean//EN"
"http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<!-- ================================================================ -->
<!-- source database connection -->
<!-- ================================================================ -->
<bean id="SrcDataBaseConnection" class="com.happysoft.tools.db.SqlServerDbConnection">
<property name="hostIp">
<value>localhost</value>
</property>
<property name="dbInstance">
<value>SMS</value>
</property>
<property name="userName">
<value>sa</value>
</property>
<property name="passWord">
<value>111111</value>
</property>
<property name="dbType">
<value>sqlserver</value>
</property>
<property name="factory">
<value>microsoft</value>
</property>
<property name="driverType">
<value>jdbc</value>
</property>
<property name="port">
<value>1433</value>
</property>
</bean>
<!-- ================================================================ -->
<!-- destination database connection -->
<!-- ================================================================ -->
<bean id="DestDataBaseConnection" class="com.happysoft.tools.db.SqlServerDbConnection">
<property name="hostIp">
<value>localhost</value>
</property>
<property name="dbInstance">
<value>SMS</value>
</property>
<property name="userName">
<value>sa</value>
</property>
<property name="passWord">
<value>111111</value>
</property>
<property name="dbType">
<value>sqlserver</value>
</property>
<property name="factory">
<value>microsoft</value>
</property>
<property name="driverType">
<value>jdbc</value>
</property>
<property name="port">
<value>1433</value>
</property>
</bean>

<!-- ================================================================ -->
<!-- Collector -->
<!-- ================================================================ -->
<bean id="MainCollector" class="com.happysoft.tools.dbcollector.CollectorSupport">
<property name="desc">
<value>main collector</value>
</property>
<property name="collects">
<list>
<ref bean="ZteSiteBlockTask"/>
</list>
</property>
</bean>
<!-- ================================================================ -->
<!-- 断站采集任务 -->
<!-- ================================================================ -->
<bean id="ZteSiteBlockTask" class="com.happysoft.tools.dbcollector.task.CollectTaskSupport">
<property name="desc">
<value>zte site block task</value>
</property>
<property name="steps">
<list>
<ref bean="CleanOldDataStep"/>
<ref bean="PumpDataStep"/>
<ref bean="UpdateDataStep"/>
</list>
</property>
</bean>
<!-- ================================================================ -->
<!-- 断站采集步骤 -->
<!-- ================================================================ -->
<bean id="CleanOldDataStep" class="com.happysoft.tools.dbcollector.step.UpdateCollectStep">
<property name="desc">
<value>clean old data setp</value>
</property>
<property name="type">
<value>del</value>
</property>
<property name="sql">
<value>delete from sec_user_info where userid=10001</value>
</property>
<property name="dataConnection">
<ref bean="DestDataBaseConnection"/>
</property>
</bean>
<bean id="PumpDataStep" class="com.happysoft.tools.dbcollector.step.PumpCollectStep">
<property name="desc">
<value>pump data setp</value>
</property>
<property name="type">
<value>pump</value>
</property>
<property name="srcFieldQuery">
<value>select userid, loginname, password, user_level, company, email from sec_user_info</value>
</property>
<property name="destFieldMapping">
<value>(userid, loginname, password, user_level, company, email)</value>
</property>
<property name="destTable">
<value>sec_user_info</value>
</property>
<property name="srcDataConnection">
<ref bean="DestDataBaseConnection"/>
</property>
<property name="destDataConnection">
<ref bean="DestDataBaseConnection"/>
</property>
</bean>
<bean id="UpdateDataStep" class="com.happysoft.tools.dbcollector.step.UpdateCollectStep">
<property name="desc">
<value>Update data</value>
</property>
<property name="type">
<value>update</value>
</property>
<property name="sql">
<value>update sec_user_info set loginname='fjunicom' where userid=10002</value>
</property>
<property name="dataConnection">
<ref bean="DestDataBaseConnection"/>
</property>
</bean>
</beans>

以上配置的采集器主要功能是,删除本地数据库旧的数据,从源数据库倒数据倒目的数据库,然后更新一些字段.

其中SrcDataBaseConnection,DestDataBaseConnection为源数据库连接,目标数据库连接配置,后续的采集步骤将引用该bean

MainCollector为采集器配置,该采集器只有一个采集任务

ZteSiteBlockTask为采集任务配置,该任务下有三个采集步骤, CleanOldDataStep,PumpDataStep,UpdateDataStep 分别为清理历史数据,从源数据库导入数据到目的数据库,更新目标数据库中其他字段.

将采集步骤分成两大类一个是update类,另外一类是倒表类,这样可以适应不同的需求.