您的位置:首页 > 其它

TBSchedule淘宝开源定时任务调度框架(附客户端源码demo)

2016-11-19 14:49 281 查看
淘宝开源定时任务调度框架

下载
TBSchedule
源码svn地址:code.taobao.org/p/tbschedule/src/trunk/

内容包括两部分
TBSchedule
源码及
开发依赖包
tbschedule
、调度控制台
ScheduleConsole.war(
配置监控查看运行状况)


 


TBSchedule
介绍:


1)
调度机的高可用有保障
,多调度机向注册中心注册后,共享调度任务,且同一调度任务仅由一台调度机执行调度,当前调度机异常宕机后,其余的调度机会接上。

2)
执行机的高可用有保障
,多执行机向注册中心注册后,配置执行机单线程(多机总线程为1)执行任务,调度机会随机启动一台执行机执行,当前执行异常机宕机后,调度机会会新调度一台执行机。

3)
执行机的并行高效保障
,配置执行机多线程且划分多任务子项后,各任务子项均衡分配到所有执行机,各执行机均执行,多线程数据一致性协调由任务项参数区分。

4)
弹性扩展失效转移保障
,运行中的执行机宕机,或新增执行机,调度机将在下次任务执行前重新分配任务项,不影响正常执行机任务(崩溃的执行机当前任务处理失效);运行中的调度机宕机或动态新增调度机,不影响执行机当前任务,调度机宕机后动态切换。

TBSchedule特性【网摘】
1)TBSchedule的目的是让一种批量任务或者不断变化的任务,能够被动态的分配到多个主机的JVM中,不同的线程组中并行执行。所有的任务能够被不重复,不遗漏的快速理。
2)调度的Manager可以动态的随意增加和停止。
3)可以通过JMX控制调度服务的创建和停止。
4)可以指定调度的时间区间。
 

 几款定时任务框架对比【网摘】
1)Quartz:Java事实上的定时任务标准。但Quartz关注点在于定时任务而非数据,并无一套根据数据处理而定制化的流程。虽然Quartz可以基于数据库实现作业的高可用,但缺少分布式并行执行作业的功能。
2)Crontab:Linux系统级的定时任务执行器,缺乏分布式和集中管理功能。
3)elastic-job:当当网最近开源项目,功能跟TBSchedule几乎一样(批斗TBSchedule文档缺失严重),一台服务器只能开启一个任务实例,基于Ip不基于IpPort,单机难调试集群功能。
4)TBSchedule:淘宝早期开源,稳定性可以保证。
 

 

Tbs执行流程图(来源于交流群)

 

Tbs客户端运行实例

 

1、   zookeeper集群搭建(windows环境下),单击模拟集群

创建ZookeeperLab文件夹,下面创建三个文件server1、server2、server3模拟三个zk端。

每个服务端下三个文件夹,data、logs、下载的zk版本解压

三个zkserver下区别

(1)   logs为空即可

(2)   data下放一个myid,没有扩展名哦,server1-data下myid写1,server2下写2,server3下写3,代表各自serverid,后面配置里会用到。

 

 

 

 

     (3)修改或增加 datadir,datalogdir,为具体路径,注意server1/2/3写正确,clienport=2181/2182/2183 每个server下端口不能重复哦,底部增加三个server的端口信息,server.A=127.0.0.1:B:C,A=myid里写的数字,B=集群成员的信息交换端口,C=在leader挂掉时专门用来进行选举leader所用

(4)运行zk集群

分别运行server1/2/3下的zkServer.cmd,运行server1、server2的时候会出错误信息,不用管它,是因为集群没有全部启动的原因,启动server3后正常通信。

 

2、   创建java job demo,并添加spring依赖

Myeclipse新建maven项目(提前部署好maven环境,此内容略,可参考https://my.oschina.net/softwarechina/blog/219633

3、Task设计(网摘)

1、假设场景:任务需要将订单表tbOrder中制单日期在20141201--20141208內(共8天)的数据同步到备份tbOrder_copy表,其中每2天分为一个任务组并行同步(每次提取500条),关于任务组的划分和TbSchedule中TaskItem的相关概念请先参考wiki,后续也会有部分解释。

2、数据环境使用MySQL数据库,创建tbOrder和tbOrder_Copy数据表,结构相同,同时在tbOrder事先生成好测试数据,建议每天的数据量在1000条以上。

     

    

 3、DataSyncABean

packageTask;

 

 

importjava.sql.Connection;

importjava.sql.PreparedStatement;

importjava.sql.ResultSet;

importjava.util.ArrayList;

import java.util.Comparator;

importjava.util.List;

 

importorg.apache.log4j.Logger;

 

importutil.ConnectionUtil;

importcom.taobao.pamirs.schedule.IScheduleTaskDealSingle;

importcom.taobao.pamirs.schedule.TaskItemDefine;

 

 

publicclass DataSyncABean implements IScheduleTaskDealSingle<OrderInfo> {

 

      

      

        private static final Logger LOG =Logger.getLogger(DataSyncABean.class); 

       public List<OrderInfo>selectTasks(String taskParameter, String ownSign, 

            int taskItemNum,List<TaskItemDefine> queryCondition, 

            int eachFetchDataNum) throwsException { 

 

             

 

        if (queryCondition.size() == 0) { 

            return null; 

        } 

 

        StringBuffer condition = newStringBuffer(); 

        for (int i = 0; i <queryCondition.size(); i++) { 

            if (i > 0) { 

               condition.append(","); 

            } 

           condition.append(queryCondition.get(i).getTaskItemId()); 

        } 

       

              String sql = "select * fromtbOrder " + "where " 

                + " BillNumber not in(select BillNumber from tbOrder_copy) " 

                + " and RIGHT(BuildDate,1)in (" + condition + ") " + "limit " 

                + eachFetchDataNum; 

              LOG.info(sql);

        Connection connection = null;

        PreparedStatement pStatement = null;

        ResultSet rs = null;

        List<OrderInfo> Orderlist = null;

        try {

            connection =ConnectionUtil.getConnection();

            pStatement =connection.prepareStatement(sql);

            rs = pStatement.executeQuery();

            Orderlist = newArrayList<OrderInfo>();

            while (rs.next()) {

                   OrderInfoorder = new OrderInfo();

                   order.BillNumber= rs.getString("BillNumber"); 

                  order.BuildDate =rs.getString("BuildDate"); 

                  order.Customer =rs.getString("Customer"); 

                  order.GoodsName =rs.getString("GoodsName"); 

                  order.Amount =rs.getFloat("Amount"); 

                  order.SaleMoney =rs.getFloat("SaleMoney"); 

                   Orderlist.add(order);

            }

          

        } catch (Exception e) {

            e.printStackTrace();

            return null;

        } finally {

            ConnectionUtil.closeResultSet(rs);

           ConnectionUtil.closePreparedStatement(pStatement);

           ConnectionUtil.closeConnection(connection);

        } 

        return Orderlist;

    } 

 

    public Comparator<OrderInfo>getComparator() { 

 

        return null; 

    } 

 

    public boolean execute(OrderInfo task,String ownSign) throws Exception { 

            String sql = "insert into tbOrder_copyvalues('" + task.BillNumber 

                       + "','" +task.BuildDate + "','" + task.Customer + "','" 

                       + task.GoodsName +"'," + task.Amount + "," + task.SaleMoney 

                       + ")"; 

         Connection connection = null;

         PreparedStatement pStatement = null;

         try {

             connection =ConnectionUtil.getConnection();

             pStatement =connection.prepareStatement(sql);

             int count =pStatement.executeUpdate();

             return count > 0 ? true :false;

         } catch (Exception e) {

             e.printStackTrace();

             return false;

         } finally {

            ConnectionUtil.closePreparedStatement(pStatement);

            ConnectionUtil.closeConnection(connection);

         }

    } 

    public void aaa()

    {

           Stringa="df";

           a="d";

}

}

3、      OrderInfo

package Task;
 
public
class
OrderInfo {
 
    public String getBillNumber(){
        returnBillNumber;
    }
    public
void
setBillNumber(StringbillNumber) {
        BillNumber = billNumber;
    }
    public String getBuildDate() {
        returnBuildDate;
    }
    public
void
setBuildDate(StringbuildDate) {
        BuildDate = buildDate;
    }
    public String getCustomer() {
        returnCustomer;
    }
    public
void
setCustomer(Stringcustomer) {
        Customer = customer;
    }
    public String getGoodsName() {
        returnGoodsName;
    }
    public
void
setGoodsName(StringgoodsName) {
        GoodsName = goodsName;
    }
    public
float
getAmount() {
        returnAmount;
    }
    public
void
setAmount(float amount) {
        Amount = amount;
    }
    public
float
getSaleMoney() {
        returnSaleMoney;
    }
    public
void
setSaleMoney(float saleMoney) {
        SaleMoney = saleMoney;
    }
    public StringBillNumber;
    public StringBuildDate;
    public StringCustomer;
    public StringGoodsName;
    public
float
Amount;
    public
float
SaleMoney;
}
4、完整pom

<projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>test1</groupId>
 <artifactId>test1</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>war</packaging>
 <name>test1</name>
 <description/>
 <properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 </properties>
 <dependencies>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>bean-validator</artifactId>
     <version>3.0-JBoss-4.0.2</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.annotation</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.ejb</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.enterprise.deploy</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.jms</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.management.j2ee</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.eclipse.persistence</groupId>
     <artifactId>javax.persistence</artifactId>
     <version>2.0.0</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.resource</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.security.auth.message</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.security.jacc</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.servlet</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.servlet.jsp</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.servlet.jsp.jstl</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish</groupId>
     <artifactId>javax.transaction</artifactId>
     <version>3.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>javax.xml.bind</groupId>
     <artifactId>jaxb-api-osgi</artifactId>
     <version>2.2.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>javax.ws.rs</groupId>
     <artifactId>jsr311-api</artifactId>
     <version>1.1.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish.web</groupId>
     <artifactId>jstl-impl</artifactId>
     <version>1.2</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>javax.mail</groupId>
     <artifactId>mail</artifactId>
     <version>1.4.3</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>javax.xml</groupId>
     <artifactId>webservices-api-osgi</artifactId>
     <version>2.0.1</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.jboss.weld</groupId>
     <artifactId>weld-osgi-bundle</artifactId>
     <version>1.0.1-SP3</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish.web</groupId>
     <artifactId>javax.servlet.jsp.jstl</artifactId>
     <version>1.2.1</version>
   </dependency>
   <dependency>
     <groupId>javax.servlet.jsp.jstl</groupId>
     <artifactId>javax.servlet.jsp.jstl-api</artifactId>
     <version>1.2.1</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-core</artifactId>
     <version>2.5.6</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-beans</artifactId>
     <version>2.5.6</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-context</artifactId>
     <version>2.5.6</version>
   </dependency>
   <dependency>
     <groupId>commons-attributes</groupId>
     <artifactId>commons-attributes-api</artifactId>
     <version>2.1</version>
   </dependency>
   <dependency>
     <groupId>commons-attributes</groupId>
     <artifactId>commons-attributes-compiler</artifactId>
     <version>2.1</version>
   </dependency>
   <dependency>
     <groupId>log4j</groupId>
     <artifactId>log4j</artifactId>
     <version>1.2.14</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-acl</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-cas-client</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-core</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-core-tiger</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-ntlm</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-openid</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-portlet</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-taglibs</artifactId>
     <version>2.0.5.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-portlet</artifactId>
     <version>2.0.5</version>
   </dependency>
   <dependency>
     <groupId>org.samba.jcifs</groupId>
     <artifactId>jcifs</artifactId>
     <version>1.2.19</version>
   </dependency>
   <dependency>
     <groupId>org.jasig.cas</groupId>
     <artifactId>cas-client-core</artifactId>
     <version>3.1.5</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.ldap</groupId>
     <artifactId>spring-ldap-core</artifactId>
     <version>1.3.0.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.webflow</groupId>
     <artifactId>spring-webflow</artifactId>
     <version>2.0.8.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.webflow</groupId>
     <artifactId>spring-binding</artifactId>
     <version>2.0.8.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.webflow</groupId>
     <artifactId>spring-js</artifactId>
     <version>2.0.8.RELEASE</version>
   </dependency>
   <dependency>
     <groupId>ognl</groupId>
     <artifactId>ognl</artifactId>
     <version>2.6.9</version>
   </dependency>
   <dependency>
     <groupId>org.springframework.javaconfig</groupId>
     <artifactId>spring-javaconfig</artifactId>
     <version>1.0.0.m4</version>
   </dependency>
   
   
   <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.4.9</version>
        </dependency>
       
<dependency>
            <groupId>com.taobao.pamirs.schedule</groupId>
            <artifactId>tbschedule</artifactId>
            <version>3.3.3.2</version>
        </dependency>
       
 
    <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.34</version>
        </dependency>
       
<dependency> 
   <groupId>org.slf4j</groupId> 
   <artifactId>slf4j-api</artifactId> 
   <version>1.6.6</version> 
</dependency> 
<dependency> 
   <groupId>org.slf4j</groupId> 
   <artifactId>slf4j-log4j12</artifactId> 
   <version>1.6.6</version> 
</dependency> 
<dependency> 
   <groupId>log4j</groupId> 
   <artifactId>log4j</artifactId> 
   <version>1.2.16</version> 
</dependency> 
 
 </dependencies>
 <build>
   <plugins>
     <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
     </plugin>
     <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.2</version>
        <configuration>
          <version>3.0</version>
          <failOnMissingWebXml>false</failOnMissingWebXml>
        </configuration>
     </plugin>
     <plugin>
        <groupId>org.apache.tomcat.maven</groupId>
        <artifactId>tomcat7-maven-plugin</artifactId>
        <version>2.1</version>
        <executions>
          <execution>
            <id>tomcat-deploy</id>
            <phase>deploy</phase>
            <goals>
              <goal>deploy</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <url>http://localhost:8081/manager/text</url>
          <username>tomcat</username>
          <password>tomcat</password>
          <server>tomcat7</server>
          <path>/test1</path>
          <ignorePackaging>true</ignorePackaging>
        </configuration>
     </plugin>
   </plugins>
 </build>
</project>

5、TaskCenter启动

public
class
TaskCenter {
     private
staticfinal
Logger LOG= Logger.getLogger(testlog4j.class); 
    public
static void
main(String[] args)throws Exception { 
         
        // 初始化Spring 
        ApplicationContext ctx = newFileSystemXmlApplicationContext( 
                "src\\main\\java\\applicationContext.xml"); 
 
        // 初始化调度工厂 
        TBScheduleManagerFactoryscheduleManagerFactory =
new
TBScheduleManagerFactory(); 
 
        Properties p = new Properties(); 
        p.put("zkConnectString","127.0.0.1:2181"); 
        p.put("rootPath",
"/taobao-pamirs-schedule/test1"); 
        p.put("zkSessionTimeout","60000"); 
        p.put("userName",
"admin"); 
        p.put("password",
"123456"); 
        p.put("isCheckParentPath","true"); 
 
       scheduleManagerFactory.setApplicationContext(ctx); 
        System.out.println("---------------------------------");  
        LOG.info("---------------maven test------------------"); 
       
 
        scheduleManagerFactory.init(p);  
 
        System.out.println("hello");      
   } 
}
运行结果

以上内容word文档下载:http://download.csdn.net/detail/mingtianmeihao/9688421

Demo源码下载地址:http://download.csdn.net/detail/mingtianmeihao/9687371

文章来源:唐山人才网

 
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
相关文章推荐