自动化二期CQC（TAOBAO TOAST框架二次开发）---支持自定义测试环境

一.背景

小组内决定自动化二期，添加前端调用，展示运行结果。

一期是由Ant构建，过程：1.Compile Java Junit Cases->2.JUnit target->3.JUnit Report target->4.Report Copy to Tomcat target->5.Email notify Result Link target；

通过不同的build.xml指定不同功能的class， 使用shell编写脚本，选择测试环境，再选择相应的模块；shell会先Svn拉下最新的用例工程代码，重置用户，根据用户选择的环境修改用例工程中配置文件内容，再根据选择的模块去运行不同的build.xml。

在github寻找自动化测试开源框架，只有TOAST和我们现有的工程能很好的整合。

比如：

百度的ITEST（https://github.com/BaiduQA/itest）： 就跟我们的自动化一期的类似，也是执行ant，得到结果，也没用到前端调用；

网易的Dagger（https://github.com/NetEase/Dagger）： a light, robust Web UI auto test framework ， 使用了Selenium（浏览器兼容性测试，ThoughWorks）和TestNg，也并不适合我们。

TOAST（https://github.com/alibaba/toast ) ：是由taobao etao测试团队开发的一套自动化测试框架，支持单元测试，功能测试，集成测试；由web端，Controller端，Agent端三部分组成，web端由PHP YII框架（MVC）编写的前端，Controller端和Agent端都是C++编写的，通过gcc编译生成二进制执行脚本，用户通过web端自定义测试内容，当用户点击运行某一测试任务，web端将需要执行的内容写入协议文件，/tmp/toast/Run-Created_***_*_*******.ini；controller端实时扫描目录/tmp/toast，
如果有新文件（Run-Created_***_*_*******.ini）将文件内容读取，解析出执行命令，传递给命令中对应机器（TestBox）的agent，对应的agent去执行相应的脚本，以功能测试为例：前端设置相应的SVN 地址（测试工程地址），对应的测试类，点击运行；controller端监控到新的协议文件：

2015-04-03 16:26:01 [INFO]: (./toastcontroller,,7770,0)There is command file: /tmp/toast/Run_Create_141_0_1428049561.ini
2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)requestStr:
{"RunID":"141","Commands":[{"CommandID":"141","TestBox":"10.210.230.26","TestCommand":"python \/data1\/toast\/plugin\/case_run\/run_case -t mvn -u https:\/\/svn1.intra.sina.com.cn\/weibo_QA\/TEST\/work\/AutoTestContent -c 6 -f LikesGroupTest;","Timeout":"127","Sudoer":"root"}],"TestType":"Regress"}

第一行为controller监控到的文件（web端写入），第二行到末尾为请求的内容，RunID为这次运行的id:141，TestBox为指定的测试机器，TestCommand在测试机上执行的命令：python \/data1\/toast\/plugin\/case_run\/run_case -t mvn -u https:\/\/svn1.intra.sina.com.cn\/weibo_QA\/TEST\/work\/AutoTestContent
-c 6 -f LikesGroupTest;","Timeout":"127","Sudoer":"root"， 执行python脚本run_case，参数-t：运行测试用例的方式，-u: 测试代码的svn地址，-c: 用例ID, -f: 测试类名。

如下，controller端将命令发送给agent端(controller和agent建立了一个TCP链接，通过该链接controller发送命令，接收命令执行结果，agent每3分钟向controller端发送heart beat信息，实现上heart beat与机器信息结合，如cpu, 内存，网络利用率等发送给controller端)。

2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)Send command to agent id 141, type 1, timeout 127
2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)command: python /data1/toast/plugin/case_run/run_case -t mvn -u https://svn1.intra.sina.com.cn/weibo_QA/TEST/work/AutoTestContent -c 6 -f LikesGroupTest;

agent端接收到controller端命令：

2015-04-03 16:26:01 [DEBUG]: (,,4585,0)Receive command id 141, type 1, timeout 127
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)account length 4, command length 144
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)account: root command: python /data1/toast/plugin/case_run/run_case -t mvn -u https://svn1.intra.sina.com.cn/weibo_QA/TEST/work/AutoTestContent -c 6 -f LikesGroupTest;
2015-04-03 16:26:01 [INFO]: (,,4585,0)Send command 141 starting message

2015-04-03 16:26:01 [DEBUG]: (,,4585,0)Command 141, processing processid: 28292, ttyfd: 6
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)GetCommandOutput fd 6
2015-04-03 16:26:28 [INFO]: (,,4585,0)Send command 141 result: 2, return code: 0 result string:
2015-04-03 16:26:28 [DEBUG]: (,,4585,0)Command 141 , result status is: 0

agent端执行用例结束后，向controller端发送结束信息，controller端接收到运行结束信息，并通过API：/toast/run/updaterun通知web端：状态为status为200，RunId:141运行结束。

2015-04-03 16:26:28 [INFO]: (./toastcontroller,,7770,0)Task 141 is run completed
2015-04-03 16:26:28 [DEBUG]: (./toastcontroller,,7770,0)URL: http://10.13.1.139/toast/run/updaterun 2015-04-03 16:26:28 [DEBUG]: (./toastcontroller,,7770,0)Content: id=141&status=200&return_value=0&desc_info=
2015-04-03 16:26:28 [INFO]: (./toastcontroller,,7770,0)Curl result: Receive update command run info with id#141 info: {"id":"141","status":"200","return_value":"0","desc_info":""} source is (10.13.1.139)

controller端将agent执行用例时的 stdout 写入/tmp/toast_output文件，以这次RunId为名的log文件：141.log；前端php根据141.log，通过正则表达式（输出结果中信息较固定）进行解析运行结果。

以上就是一个完成的执行流程。

TOAST能很好的结合我们现有的测试工程，进行调度和展示结果，而测试的运行速度完全依赖于我们自身测试工程的JUnit优化（多线程执行，同时支持类和方法级别的并发），但是TOAST已有2年未更新，现有的部分功能代码并没有实现，如解析stdout结果，前端解析文件（php）并没有正常解析，源码中把所有case都赋予失败，这个是未完成的功能； 还有我们需要指定测试的环境，这个是新的需求，原有TOAST不支持。

二. 功能测试支持自定义测试环境

agent端执行用例，实际上是执行python脚本run_case，新增-e(--env) 参数，指定测试环境。

#/usr/bin/python2.6
# Filename: run_case
# -*- coding: utf-8 -*-

#
#   Copyright (C) 2007-2013 Alibaba Group Holding Limited
#
#   This program is free software;you can redistribute it and/or modify
#   it under the terms of the GUN General Public License version 2 as
#   published by the Free Software Foundation.
#

import sys
import getopt
import os
TOOL = ["mmt", "mvn"]

def usage():
'''
@summary: run_case manual.
'''
print "Usage: run_case -t tool -u url -c case_id [-f] ...\n"
print "Misc:\n",
print "	-h, --help	Print this for help, then exit.\n"
print "Operation:\n",
print "	-t, --tool	The test tool to run the test case.\n",
print "	-u, --url	The URL for checking out the test case, support SVN only now.\n \
For maven project, this is maven project base svn url",
print "	-c, --caseid	The caseid which comes from TOAST, and it will be printed.\n \
For maven project, this is the test calss will be run.",
print "	-f, --function	The function which will be run, some test tools no need it(such as MMT).\n"
print " -e, --env  test environment.\n"
print "Example:\n",
print "	run_case -t mmt -u http://xxx.xxx.xxx/svn/case1.php -c 1\n"

def get_options(argv):
'''
@summary: get options and check options.
'''
message = "Use run_case --help to get usage information.\n"
options = {
"tool": "",
"url": "",
"function": "",
"env":"",
"caseid": ""
}
try:
opts, args = getopt.getopt(argv[1:], "ht:u:c:e:f:", ["help", "tool=", "url=", "caseid=", "env=","function="])
for o, a in opts:
if o in ("-h", "--help"):
usage()
exit(0)
if o in ("-t", "--tool"):
if a in TOOL:
options["tool"] = a
else:
print "The test tool is required, and support " + ", ".join(TOOL) + " only. " + message
exit(1)
if o in ("-u", "--url"):
if "" != a:
options["url"] = a
else:
print "The URL is required. " + message
exit(1)
if o in ("-c", "--caseid"):
if "" != a:
options["caseid"] = a
else:
print "The caseid is required. " + message
exit(1)
if o in("-e", "--env"):
if "" != a:
options["env"] = a
else:
print "The env is required. " + message
exit(1)
if o in ("-f", "--function"):
if "" != a:
options["function"] = a
else:
print "The function is required. " + message
exit(1)
return options
except getopt.GetoptError:
print message
exit(1)

if __name__ == "__main__":
if 1 == len(sys.argv):
usage()
exit(0)
else:
options = get_options(sys.argv)
try:
if options["tool"] == 'mvn':
#from './tool/runmvn' import run_mvn_case
sys.path.append(os.path.join(os.path.dirname(__file__),"./tool"))
from runmvn import run_mvn_case
cfg_file = os.path.splitext(os.path.abspath(__file__))[0] + ".conf"
print cfg_file + options["caseid"] + options["url"]
runner = run_mvn_case(cfg_file, options["function"], options["url"], options["caseid"], options["env"])
try:
runner.get_code()
print 'code has checked out'
if "" != options["env"]:
runner.set_env(options["env"])
runner.run_a_case(options["function"])
except Exception, ex:
print Exception,":",ex
print traceback.format_exc()
finally:
runner.cleanup()
sys.exit(0)
else:
toollib = __import__("tool." + options["tool"], globals(), locals(), ['Mmt'])
Tool = getattr(toollib, options["tool"].capitalize())
tool = Tool(options["url"], options["caseid"])
tool.execute()
sys.exit(0)
except ImportError, e:
print "Import tool module error."
print e
exit(1)

新增set_env(env)函数，修改配置文件内容，替换成自定义环境：

#!/usr/bin/python
# call mvn command run mvn case
# 1. checkout code in the svn
# 2. mvn test to run the specify case
# Infact it's just a mvn wapper

#
#   Copyright (C) 2007-2013 Alibaba Group Holding Limited
#
#   This program is free software;you can redistribute it and/or modify
#   it under the terms of the GUN General Public License version 2 as
#   published by the Free Software Foundation.
#

import ConfigParser
import string, os, sys
import subprocess
import uuid
import shutil
class run_mvn_case:
def __init__(self):
self.options       = {}
self.CONFILE       = ""
self.casetorun     = ""
self.mvnprojectsvn = ""
self.configer      =NULL
self.local_path = ""
self.env = ""
def __init__(self, cfg_file, casetorun, mvnprojectsvn, caseid, env):
self.CONFILE = cfg_file
self.casetorun = casetorun
self.mvnprojectsvn=mvnprojectsvn
self.configer = ConfigParser.ConfigParser()
self.configer.read(self.CONFILE)
self.local_path = "/tmp/" + str(uuid.uuid4())
self.id = caseid
self.env = env
self.cleanup()
def get_code(self):
svn_account = self.configer.get('svn', 'account')
svn_password = self.configer.get('svn', 'password')
svn_command = "svn co " + "--username " + svn_account + " --password " + svn_password + " --no-auth-cache " + " --non-interactive " + self.mvnprojectsvn + " " + self.local_path
print svn_command
pipe = subprocess.Popen(svn_command, bufsize=4096, shell=True, stderr=subprocess.STDOUT, stdout = subprocess.PIPE, close_fds=True)
while True:
line = pipe.stdout.readline(4096)
if not line:
break
sys.stdout.write(line)
return pipe.wait()
#reset environment
def set_env(self, env):
print 'change environment'
filepath = self.local_path + "/src/test/java/global.properties"
host, port = env.split(':', 1)
with open(filepath, 'w') as f:
f.write("host=" + host + "\nport=" + port + "\nsource=2975945008\n")
with open(filepath, 'r') as fr:
print fr.read()
def run_a_case(self, case):
print 'start to run case'
command = "cd " + self.local_path + "; mvn -Dtest=" + case + " test"
print command
pipe = subprocess.Popen(command, bufsize=4096, shell=True, stderr=subprocess.STDOUT, stdout = subprocess.PIPE, close_fds=True)
while True:
line = pipe.stdout.readline()
if not line:
break
sys.stdout.write(line)
print "CASE ID: " + self.id + "\n"
return pipe.wait()

def cleanup(self):
if os.path.exists(self.local_path):
shutil.rmtree(self.local_path)

def usage():
print "run mvn\n" \
"-h --help print this help message\n" \
"-c --class test calass want to run\n" \
"-u --svnurl the maven project base svn url\n"

if __name__ == '__main__':
import getopt

if len(sys.argv) < 2:
usage()
sys.exit(1)
try:
opts,args = getopt.getopt(sys.argv[1:], "hc:u:", ["help", "class=", "svnurl="])
except getopt.GetoptError as err:
print str(err)
usage()
system.exit(2)
runclass = ""
svnurl = ""
for o, a in opts:
if o in("-h", "--help"):
usage()
sys.exit()
elif o in ("-c", "--class"):
runclass = a
elif o in("-u", "--svnurl"):
svnurl = a
else:
assert False, "unhandled option"
cfg_file = os.path.splitext(os.path.abspath(__file__))[0] + ".conf"
print cfg_file
print runclass
print svnurl
runner = run_mvn_case(cfg_file, runclass, svnurl)
try:
runner.get_code()
print 'code has checked out'
runner.run_a_case(runclass)
except Exception, ex:
print Exception,":",ex
print traceback.format_exc()
finally:
runner.cleanup()
sys.exit(0)

该脚本流程如下：1.在/tmp目录下生成一个临时文件（由uuid.uuid4()生成，32字节字符串）->2.svn co拉测试用例工程放在step1生成的临时文件中->3.判断是否有参数-e，如果有修改配置文件中内容为相应环境，如果没有直接默认跑线上->4.进入step1创建的目录，执行controller发来的命令， mvn -Dtest=LikeObjectRpcTest test，执行相应的测试用例（要求测试工程为maven并安装surefire插件）。
该脚本执行的同时，agent端会将该stdout通过socket传给controller端，controller端写入stdout文件夹中，以RunId命名，eg:141.log；web端读取stdout文件中的141.log进行结果分析，php使用全局正则表达式进行匹配，用例总数，失败数，skipped数。

解析用例执行结果（原有TOAST，功能并未正常实现，将用例全赋为 $caseInfo->result = CaseInfo::RESULT_FAILED(web端JUnitMvnParser.php)；并根据该文件中正则表达式，反推taobao的JUnit case设计时，会在每个case中，System.out.println()出每个用例的信息，步骤和用例图示（按照统一的规则输出），类似于：

System.out.println("Results : casename(caseinfo)Tests run:");
System.out.println("[step case=0 number=\"10000001\"]步骤1[/step]");
System.out.println("[img case=0]*.img[/img]");

这个值得学习，但是如果JUnit支持用例重试机制，stdout就会有重复，解析就会出现重复的现象，当然可以包一层，滤重的过程，暂不输出用例信息）。

在实际执行用例时，标准的stdout并没有打印出成功和skipped用例的内容，支持自定义stdout全部用例信息：自动化二期（TAOBAO
TOAST框架二次开发）---支持结果展示。

InfoQ上有一篇文章值得一读，支付宝AQC：支付宝分层与端到端回归平台建设实践，可惜没开源，文章中的理念还是值得学习。