Zabbix监控Tengine 后端服务器健康检查

一 应用场景描述:

前端使用Tengine作为负载均衡器，需要对监控Tengine到后端服务器的健康状况，利用Tengine提供的接口可以获取每一个后端主机的状态，是up还是down等。

二 编写Zabbix监控脚本
Tengine的ngx_http_upstream_check_module 模块提供后端监控检查功能。可以使用html,csv,json三种格式查看后端主机状态，这里我们使用json格式便于编写脚本

/status?format=html
/status?format=csv
/status?format=json

{"servers": {"total": 1,"generation": 3,"server": [{"index": 0, "upstream": "backend", "name": "106.187.48.116:80", "status": "up", "rise": 58, "fall": 0, "type": "http", "port": 80}]}}

tengine_status.py

#!/usr/bin/env /usr/bin/python
'''
curl 127.0.0.1/up_status?format=json

'''
import json
import optparse
import socket
import urllib2
import subprocess
import tempfile
import os
import logging

logging.basicConfig(filename='/opt/logs/zabbix/tengine_zabbix.log', level=logging.WARNING, format='%(asctime)s %(levelname)s: %(message)s')
#logging.basicConfig(filename='/tmp/tengine_zabbix.log', level=logging.INFO, format='%(asctime)s %(levelname)s: %(message)s')

class Tengine(object):
'''Class for Tengine Management API'''

def __init__(self,host_name='',conf='/opt/app/zabbix/conf/zabbix_agentd.conf', senderhostname=None):
self.host_name = host_name or socket.gethostname()
self.conf = conf or '/opt/app/zabbix/conf/zabbix_agentd.conf'
self.senderhostname = senderhostname if senderhostname else host_name

def call_api(self,tmpfile=None,write='no'):
##### change url here
url = 'http://127.0.0.1/up_status?format=json'
logging.debug('Get tengine upstream status')
up_status=json.loads(urllib2.urlopen(url).read())
upstreams = []
for upserver in up_status['servers']['server']:
logging.debug("Discovered upserver " + upserver['upstream'] + upserver['name'] + str(upserver['index']))
element = {'{#UPSTREAM}': upserver['upstream'],
'{#UPNAME}': upserver['name'],
'{#UPINDEX}': upserver['index']
}
upstreams.append(element)
if write is 'yes' and tmpfile is not None:
for item in [ 'status','rise','fall','type' ]:
key = '"tengine.upstream_status[{0},{1},{2},{3}]"'.format(upserver['index'], upserver['upstream'], upserver['name'],item)
value = upserver[item]
#print key + ":" + str(value)
logging.debug("SENDER_DATA: - %s %s" % (key,value))
tmpfile.write("- %s %s\n" % (key, value))
#print upstreams
return upstreams

def check_data(self):
return_code = 0
#### use tempfile module to create a file on memory, will not be deleted when it is closed , because 'delete' argument is set to False
rdatafile = tempfile.NamedTemporaryFile(delete=False)
self.call_api(rdatafile,write='yes')
rdatafile.close()
return_code = self._send_status_data(rdatafile)
#### os.unlink is used to remove a file
os.unlink(rdatafile.name)
return return_code

def _send_status_data(self, tmpfile):
'''Send the status data to Zabbix.'''
'''Get key value from temp file. '''
args = '/opt/app/zabbix/sbin/zabbix_sender -c {0} -i {1}'
if self.senderhostname:
args = args + " -s " + self.senderhostname
return_code = 0
process = subprocess.Popen(args.format(self.conf, tmpfile.name),
shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = process.communicate()
logging.debug("Finished sending data")
return_code = process.wait()
logging.info("Found return code of " + str(return_code))
if return_code != 0:
logging.warning(out)
logging.warning(err)
else:
logging.debug(err)
logging.debug(out)
return return_code

def main():
choices=['list_upservers','upstream_status']
parser = optparse.OptionParser()
parser.add_option('--hostname', help='Tengine server hostname',
default=socket.gethostname())
parser.add_option('--conf', default='/opt/app/zabbix/conf/zabbix_agentd.conf')
parser.add_option('--senderhostname', default='', help='Allows including a sender parameter on calls to zabbix_sender')
parser.add_option('--check', type='choice',choices=choices,help='Type of check')
(options, args) = parser.parse_args()
logging.debug("Started trying to process data")

if not options.check:
parser.error('At least one check should be specified')
logging.debug("Started trying to process data")

api = Tengine(host_name=options.hostname,conf=options.conf, senderhostname=options.senderhostname)

if options.check == 'list_upservers':
print json.dumps({'data': api.call_api(tmpfile=None,write='no')},indent=4,separators=(',',':'))
elif options.check == 'upstream_status':
print api.check_data()

if __name__ == '__main__':
main()

tengine_status.conf

UserParameter=tengine.discovery_upstream,/usr/bin/python /opt/app/zabbix/sbin/tengine_status.py  --check='list_upservers'
UserParameter=tengine.upstream_status[*],/usr/bin/python /opt/app/zabbix/sbin/tengine_status.py  --check='upstream_status'

脚本中url根据自己的情况进行修改，还有脚本中使用zabbix_sender进行监控数据的发送工作，zabbix_sender发送数据时需要指定的主机名和Zabbix上被监控主机的主机名相同才会发送成功。
如果 --check选项的参数是list_upservers 则只会列出各个upstream后端主机，对应的zabbix key是tengine.discovery_upstream

{
"data":[
{
"{#UPNAME}":"172.28.16.140:80",
"{#UPSTREAM}":"test",
"{#UPINDEX}":0
},
{
"{#UPNAME}":"172.28.16.143:80",
"{#UPSTREAM}":"test",
"{#UPINDEX}":1
},
{
"{#UPNAME}":"172.28.16.144:80",
"{#UPSTREAM}":"test",
"{#UPINDEX}":2
},
{
"{#UPNAME}":"172.28.16.158:80",
"{#UPSTREAM}":"test",
"{#UPINDEX}":3
}
]
}

如果 --check参数是upstream_status则会通过zabbix_sender将一个临时文件rdatafile中的key，value对发送到zabbix proxy或者zabbix server。这里需要注意一下，zabbix_sender是不会主动发送数据过去的，需要有操作触发zabbix_sender才会发送，也就是

/usr/bin/python /opt/app/zabbix/sbin/tengine_status.py  --check='upstream_status'

这个执行需要定时任务执行或者通过zabbix agent或者zabbix agent(active)触发执行

脚本中的各种文件路径根据自己情况修改。
在模板中使用一个类型为Zabbix agent(active)的key来定期执行这个脚本，然后其他的监控数据就通过zabbix_sender上报给zabbix proxy或者zabbix server
tengine.upstream_status[send_data]

tengine.upstream_status[{#UPINDEX},{#UPSTREAM},{#UPNAME},fall]

tengine.upstream_status[{#UPINDEX},{#UPSTREAM},{#UPNAME},rise]

tengine.upstream_status[{#UPINDEX},{#UPSTREAM},{#UPNAME},status]

tengine.upstream_status[{#UPINDEX},{#UPSTREAM},{#UPNAME},type]

这几个key的类型都是Zabbix trapper

添加一个报警，当后端主机的status值为down时就报警
{Template Tengine Status:tengine.upstream_status[{#UPINDEX},{#UPSTREAM},{#UPNAME},status].str(down)}=1

脚本测试的时候，最好是在zabbix proxy后者zabbix server上使用zabbix_get 来测试
zabbix_get -s 172.28.16.139 -k tengine.discovery_upstream
返回数据正常则测试成功

zabbix_get -s 172.28.16.139 -k tengine.upstream_status[send_data]

返回值为0则发送数据成功，可以在zabbix页面查看相应的数据
返回值为非0则发送数据失败，需要检查下脚本中的url，zabbix_sender,conf配置文件路径，日志文件权限，主机名是否匹配等

三 制作Zabbix监控模板
模板参见附件

参考文档：
http://tengine.taobao.org/
http://tengine.taobao.org/document_cn/http_upstream_check_cn.html

附件：http://down.51cto.com/data/2368448