由一次long SQL调优引发de血案

背景：有一个定时任务，在每天的凌晨执行，任务是备份一张表的数据到另一张表，数据量大约200W。操作很简单，从source表里select数据，然后insert到target表里。由于数据量较大，所以从一开始就注定了要走上优化的不归路。
Step 1
如果使用单线程，一边读一边存，程序要跑多久，不得而知，估计只有傻X才会这么写。所以第一条思路为分批次取数据，然后丢到多线程里去存。分批次取数据类似数据分页，每取一页数据就丢到一个线程里。单线程分批次取数据，多线程存数据。取数据的SQL语句如下：
[code=sql;toolbar:false">SELECT ID,AccountID,AccountType,AccountSubject,Credit,Debit,STATUS,IsBuffer,AccountDirection,Balance,FrozenBalance,ADDTIME,UpdateTime
        FROM Account
        WHERE AccountType = 2
        ORDER BY ID
        LIMIT #offSet#, #batchSize#; public void execute() {
        //获取线程数，默认30
        int threadNum = 30;
        String strThreadNum = LionConfigUtils.getProperty("ts-monitor-job.dailyJob.accountBalanceDailyCheckerThreadNum",
                "");
        if (isNumeric(strThreadNum)) {
            threadNum = Integer.parseInt(strThreadNum);
        }
        monitorLogger.info(String.format("BATCH_SIZE:%s, Thread number:%s", BATCH_SIZE, threadNum));

        ExecutorService service = Executors.newFixedThreadPool(threadNum);
        //所有账号总数
        int accountCount = accountDao.findAllMerchantAccountCount();
        int latchCount = accountCount % BATCH_SIZE == 0 ? accountCount / BATCH_SIZE : (accountCount / BATCH_SIZE + 1);
        CountDownLatch latch = new CountDownLatch(latchCount);
        Date bizDate = DateUtils.addDate(DateUtils.removeTime(new Date()), -1);
        Date lastBizDate = DateUtils.addDate(bizDate, -1);
        //起始偏移
        int offset = 0;
        //上一页中最大的ID
        int lastBiggestId = 0;
        List<AccountData> accountDataList = null;
        while (offset < accountCount) {
            accountDataList = accountDao.findAllMerchantAccount(offset, BATCH_SIZE, lastBiggestId);
            if(accountDataList == null){
                break;
            }
            //取出当前批次中最大的id供下次取数据使用，SQL分页查询优化
            lastBiggestId = accountDataList.get(accountDataList.size() - 1).getId();
            //偏移后移，取下一页
            offset += BATCH_SIZE;
            service.submit(new DoAccountMonitorThread(accountDataList, bizDate, lastBizDate,monitorAccountBalanceDao, accountEntryDao, latch));
        }
        try {
            latch.await();
        } catch (InterruptedException e) {
            monitorLogger.error("CountDownLatch.await() error:" + e);
        }
    }<insert id="batchInsertDailyAccountBalance" parameterClass="java.util.Map">
        <![CDATA[
        INSERT INTO TS_DailyAccountBalance
        (BizDate,
        AccountID,
        AccountSubject,
        CreditAmount,
        DebitAmount,
        AccountAmount,
        YesterdayBalance,
        CurrentBalance,
        VarianceAmount,
        Memo,
        Status,
        AddTime,
        UpdateTime)
        VALUES
         ]]>
        <iterate property="monitorAccountBalanceDataList" conjunction=",">
            <![CDATA[(
            #monitorAccountBalanceDataList[].bizDate#,
            #monitorAccountBalanceDataList[].accountId#,
            #monitorAccountBalanceDataList[].accountSubject#,
            #monitorAccountBalanceDataList[].creditAmount#,
            #monitorAccountBalanceDataList[].debitAmount#,
            #monitorAccountBalanceDataList[].accountAmount#,
            #monitorAccountBalanceDataList[].yesterdayBalance#,
            #monitorAccountBalanceDataList[].currentBalance#,
            #monitorAccountBalanceDataList[].varianceAmount#,
            #monitorAccountBalanceDataList[].memo#,
            #monitorAccountBalanceDataList[].status#,
            NOW(),
            NOW()
            )
        ]]>
        </iterate>
    </insert>