Redis Sentinel高可用架构

Redis的高可用架构现在越来越多了，可以见得Redis的发展是有多么的迅速，现在不少公司都用上了Redis，所以Redis高可用也显得尤其重要，现在Redis的高可用架构有比如keepalived+redis，redis cluster，twemproxy，codis，下面我们主要针对Redis Sentinel高可用架构展开学习。

Redis Sentinel主要功能有以下几点：

不时地监控redis是否按照预期良好地运行;

如果发现某个redis节点运行出现状况，能够通知另外一个进程(例如它的客户端);

能够进行自动切换。当一个master节点不可用时，能够选举出master的多个slave(如果有超过一个slave的话)中的一个来作为新的master,其它的slave节点会将它所追随的master的地址改为被提升为master的slave的新地址。

Sentinel是一个监视器，它可以根据被监视实例的身份和状态来判断应该执行何种动作。Sentinel是如何发现其他Sentinel的呢？Sentinel会通过命令连接向被监视的主从服务器发送HELLO信息，该消息包含Sentinel的IP、端口号、ID等内容，以此来向其他Sentinel宣告自己的存在。与此同时，Sentinel会通过订阅连接接收其他Sentinel的HELLO信息，以此来发现监视同一个主服务器的其他Sentinel。

#!/usr/bin/python
#coding:utf8

import sys
import time
import smtplib
import logging
from email.mime.text import MIMEText
from email.message import Message
from email.header import Header

alarm_mail =['1111111111@qq.com']

def main():

failover_time=time.strftime("%Y-%m-%d %H:%M:%S")

logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(filename)s[line:%(lineno)d] %(levelname)s %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
filename='/data/service/redis/failover.log',
filemode='a')

console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)

mail_host='smtp.163.com'
mail_port=25
mail_user=''
mail_pass=''
mail_send_from = ''

def send_mail(to_list,sub,content):
me=mail_send_from
msg = MIMEText(content, _subtype='html', _charset='utf-8')
msg['Subject'] = Header(sub,'utf-8')
msg['From'] = Header(me,'utf-8')
msg['To'] = ";".join(to_list)
try:
smtp = smtplib.SMTP()
smtp.connect(mail_host,mail_port)
smtp.login(mail_user,mail_pass)
smtp.sendmail(me,to_list, msg.as_string())
smtp.close()
return True
except Exception as error:
logging.error("邮件发送失败: %s" % (error))
return False

try:
master_name = sys.argv[1]
role = sys.argv[2]
from_ip = sys.argv[4]
from_port = sys.argv[5]
to_ip = sys.argv[6]
to_port = sys.argv[7]
except Exception as error:
logging.error('从 Sentinel 获取参数错误: %s ' % (error))
sys.exit(1)

sub='redis %s faiover' % (master_name)
nodify_message = "%s %s is failover end. sentinel find redis master %s:%s is down. failover to slave %s:%s" % (failover_time,master_name,from_ip,from_port,to_ip,to_port)

if role == 'leader':
logging.info(nodify_message)
send_mail(alarm_mail,sub,nodify_message)

if __name__ == "__main__":
main()

四、下面启动Sentinel服务，启动方式有两种：

方式一：

redis-sentinel /path/to/sentinel.conf

方式二：

redis-server /path/to/sentinel.conf --sentinel

我习惯用第一种方法，分别在三台Sentinel服务器进行启动：

第一台Sentinel_1启动log：

[root@Sentinel_1 sentinel]# redis-sentinel /data/service/redis/sentinel/conf/26379.conf
[root@Sentinel_1 sentinel]# tail -f log/sentinel.log
|    `-._`-._        _.-'_.-'    |
`-._    `-._`-.__.-'_.-'    _.-'
`-._    `-.__.-'    _.-'
`-._        _.-'
`-.__.-'

5153:X 07 Mar 22:37:16.290 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
5153:X 07 Mar 22:37:16.290 # Sentinel runid is 21e629e6d2b26682e660258787d5fb995010e6c8
5153:X 07 Mar 22:37:16.290 # +monitor master master-6379 192.168.10.131 6379 quorum 2
5153:X 07 Mar 22:37:17.330 * +slave slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379
5153:X 07 Mar 22:38:29.406 * +sentinel sentinel 192.168.10.129:26379 192.168.10.129 26379 @ master-6379 192.168.10.131 6379
5153:X 07 Mar 22:38:45.024 * +sentinel sentinel 192.168.10.130:26379 192.168.10.130 26379 @ master-6379 192.168.10.131 6379

第二台Sentinel_2启动log:

[root@Sentinel_2 sentinel]# redis-sentinel /data/service/redis/sentinel/conf/26379.conf
[root@Sentinel_2 sentinel]# tail -f log/sentinel.log
`-._    `-._`-.__.-'_.-'    _.-'
`-._    `-.__.-'    _.-'
`-._        _.-'
`-.__.-'

4647:X 07 Mar 22:38:27.570 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
4647:X 07 Mar 22:38:27.570 # Sentinel runid is f391228f430177d881464e908c683bfc73d61c24
4647:X 07 Mar 22:38:27.571 # +monitor master master-6379 192.168.10.131 6379 quorum 2
4647:X 07 Mar 22:38:28.582 * +slave slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379
4647:X 07 Mar 22:38:29.218 * +sentinel sentinel 192.168.10.128:26379 192.168.10.128 26379 @ master-6379 192.168.10.131 6379
4647:X 07 Mar 22:38:45.200 * +sentinel sentinel 192.168.10.130:26379 192.168.10.130 26379 @ master-6379 192.168.10.131 6379

第三台Sentinel_3启动log:

[root@Sentinel_3 sentinel]# redis-sentinel /data/service/redis/sentinel/conf/26379.conf
[root@Sentinel_3 sentinel]# tail -f log/sentinel.log
`-._    `-.__.-'    _.-'
`-._        _.-'
`-.__.-'

2115:X 07 Mar 22:38:43.161 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2115:X 07 Mar 22:38:43.161 # Sentinel runid is 7fbee9138d4e5c1e2def7bbc4f888cef04d95677
2115:X 07 Mar 22:38:43.161 # +monitor master master-6379 192.168.10.131 6379 quorum 2
2115:X 07 Mar 22:38:44.167 * +slave slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379
2115:X 07 Mar 22:38:44.818 * +sentinel sentinel 192.168.10.129:26379 192.168.10.129 26379 @ master-6379 192.168.10.131 6379
2115:X 07 Mar 22:38:44.851 * +sentinel sentinel 192.168.10.128:26379 192.168.10.128 26379 @ master-6379 192.168.10.131 6379

可以看到Sentinel整个集群都开始工作了，我们可以随便登录一台Sentinel看下现在监视的状态：

[root@Sentinel_1 sentinel]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=master-6379,status=ok,address=192.168.10.131:6379,slaves=1,sentinels=3
127.0.0.1:26379>

可以看到状态是status=ok，slaves=1有一个从节点。

五、Redis down机测试

测试一、停掉Redis_Master，看Sentinel会不会把存活的Slave节点提升为Master节点

[root@Redis_Master redis]# sh redis stop
Stopping ...
Waiting for Redis to shutdown ...
Redis stopped
[root@Redis_Master redis]#

1、随便查看一台Sentinel的log，tail -f log/sentinel.log：

5153:X 07 Mar 22:48:20.986 # +sdown master master-6379 192.168.10.131 6379
5153:X 07 Mar 22:48:21.047 # +odown master master-6379 192.168.10.131 6379 #quorum 2/2
5153:X 07 Mar 22:48:21.049 # +new-epoch 1
5153:X 07 Mar 22:48:21.050 # +try-failover master master-6379 192.168.10.131 6379
5153:X 07 Mar 22:48:21.053 # +vote-for-leader 21e629e6d2b26682e660258787d5fb995010e6c8 1
5153:X 07 Mar 22:48:21.057 # 192.168.10.130:26379 voted for 7fbee9138d4e5c1e2def7bbc4f888cef04d95677 1
5153:X 07 Mar 22:48:21.062 # 192.168.10.129:26379 voted for 7fbee9138d4e5c1e2def7bbc4f888cef04d95677 1
5153:X 07 Mar 22:48:22.441 # +config-update-from sentinel 192.168.10.130:26379 192.168.10.130 26379 @ master-6379 192.168.10.131 6379
5153:X 07 Mar 22:48:22.442 # +switch-master master-6379 192.168.10.131 6379 192.168.10.132 6379
5153:X 07 Mar 22:48:22.443 * +slave slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379
5153:X 07 Mar 22:48:37.496 # +sdown slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379

2、再查看Redis_Slave的log:

2437:S 07 Mar 22:48:18.023 * Connecting to MASTER 192.168.10.131:6379
2437:S 07 Mar 22:48:18.026 * MASTER <-> SLAVE sync started
2437:S 07 Mar 22:48:18.029 # Error condition on socket for SYNC: Connection refused
2437:S 07 Mar 22:48:19.050 * Connecting to MASTER 192.168.10.131:6379
2437:S 07 Mar 22:48:19.053 * MASTER <-> SLAVE sync started
2437:S 07 Mar 22:48:19.055 # Error condition on socket for SYNC: Connection refused
2437:S 07 Mar 22:48:20.074 * Connecting to MASTER 192.168.10.131:6379
2437:S 07 Mar 22:48:20.077 * MASTER <-> SLAVE sync started
2437:S 07 Mar 22:48:20.079 # Error condition on socket for SYNC: Connection refused
2437:M 07 Mar 22:48:20.724 * Discarding previously cached master state.
2437:M 07 Mar 22:48:20.725 * MASTER MODE enabled (user request from 'id=7 addr=192.168.10.130:60991 fd=11 name=sentinel-7fbee913-cmd age=577 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=rw cmd=exec')
2437:M 07 Mar 22:48:20.745 # CONFIG REWRITE executed with success.
2437:M 07 Mar 22:48:20.796 * 1 changes in 900 seconds. Saving...
2437:M 07 Mar 22:48:20.870 * Background saving started by pid 2442
2442:C 07 Mar 22:48:20.915 * DB saved on disk
2442:C 07 Mar 22:48:20.915 * RDB: 4 MB of memory used by copy-on-write
2437:M 07 Mar 22:48:20.974 * Background saving terminated with success

3、现在再登录Sentinel查看现在的主节点是谁：

[root@Sentinel_1 sentinel]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=master-6379,status=ok,address=192.168.10.132:6379,slaves=1,sentinels=3
127.0.0.1:26379>

可以看到，新的Master已经变成192.168.10.132了。切换后的邮件通知：



4、把down机的redis启动后，会自动添加为slave角色：

[root@Redis_Master redis]# sh redis start
Starting Redis server...
[root@Redis_Master redis]# tail -f logs/redis_6379.log
`-._    `-._`-.__.-'_.-'    _.-'
`-._    `-.__.-'    _.-'
`-._        _.-'
`-.__.-'

2050:M 07 Mar 22:55:21.357 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2050:M 07 Mar 22:55:21.357 # Server started, Redis version 3.0.7
2050:M 07 Mar 22:55:21.357 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2050:M 07 Mar 22:55:21.357 * DB loaded from disk: 0.000 seconds
2050:M 07 Mar 22:55:21.357 * The server is now ready to accept connections on port 6379
2050:S 07 Mar 22:55:31.393 * SLAVE OF 192.168.10.132:6379 enabled (user request from 'id=4 addr=192.168.10.129:50326 fd=8 name=sentinel-f391228f-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=rw cmd=exec')
2050:S 07 Mar 22:55:31.397 # CONFIG REWRITE executed with success.
2050:S 07 Mar 22:55:31.596 * Connecting to MASTER 192.168.10.132:6379
2050:S 07 Mar 22:55:31.597 * MASTER <-> SLAVE sync started
2050:S 07 Mar 22:55:31.597 * Non blocking connect for SYNC fired the event.
2050:S 07 Mar 22:55:31.598 * Master replied to PING, replication can continue...
2050:S 07 Mar 22:55:31.600 * Partial resynchronization not possible (no cached master)
2050:S 07 Mar 22:55:31.634 * Full resync from master: 234202729a196fd6523e41bcb7e29d9866c905c6:1
2050:S 07 Mar 22:55:31.648 * MASTER <-> SLAVE sync: receiving 34 bytes from master
2050:S 07 Mar 22:55:31.649 * MASTER <-> SLAVE sync: Flushing old data
2050:S 07 Mar 22:55:31.649 * MASTER <-> SLAVE sync: Loading DB in memory
2050:S 07 Mar 22:55:31.649 * MASTER <-> SLAVE sync: Finished with success

5、查看Sentinel log，可以看到slave被加进来，并成为Slave的角色了:

4647:X 07 Mar 22:55:31.787 * +convert-to-slave slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379

测试二、把新的Redis_Master(192.168.10.132，原来的slave)停掉，看是否把新的Slave（192.168.10.131，原来的master）提升为主：

1、执行redis stop操作

[root@Redis_Slave redis]# sh redis stop
Stopping ...
Waiting for Redis to shutdown ...
Redis stopped

2、查看Sentinel log：

5153:X 07 Mar 23:01:54.895 # +try-failover master master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:54.898 # +vote-for-leader 21e629e6d2b26682e660258787d5fb995010e6c8 2
5153:X 07 Mar 23:01:54.908 # 192.168.10.129:26379 voted for f391228f430177d881464e908c683bfc73d61c24 2
5153:X 07 Mar 23:01:54.913 # 192.168.10.130:26379 voted for 21e629e6d2b26682e660258787d5fb995010e6c8 2
5153:X 07 Mar 23:01:54.968 # +elected-leader master master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:54.968 # +failover-state-select-slave master master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:55.027 # +selected-slave slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:55.027 * +failover-state-send-slaveof-noone slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:55.085 * +failover-state-wait-promotion slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:55.912 # +promoted-slave slave 192.168.10.131:6379 192.168.10.131 6379 @ master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:55.915 # +failover-state-reconf-slaves master master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:56.009 # +failover-end master master-6379 192.168.10.132 6379
5153:X 07 Mar 23:01:56.010 # +switch-master master-6379 192.168.10.132 6379 192.168.10.131 6379
5153:X 07 Mar 23:01:56.010 * +slave slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379
5153:X 07 Mar 23:02:11.066 # +sdown slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379

3、再查看新Redis_Master的log，可以看到状态从SLave转回了Master:

[root@Redis_Master redis]# tail -f logs/redis_6379.log
2050:S 07 Mar 23:01:52.367 # Error condition on socket for SYNC: Connection refused
2050:S 07 Mar 23:01:53.382 * Connecting to MASTER 192.168.10.132:6379
2050:S 07 Mar 23:01:53.384 * MASTER <-> SLAVE sync started
2050:S 07 Mar 23:01:53.384 # Error condition on socket for SYNC: Connection refused
2050:S 07 Mar 23:01:54.404 * Connecting to MASTER 192.168.10.132:6379
2050:S 07 Mar 23:01:54.405 * MASTER <-> SLAVE sync started
2050:S 07 Mar 23:01:54.406 # Error condition on socket for SYNC: Connection refused
2050:M 07 Mar 23:01:54.868 * Discarding previously cached master state.
2050:M 07 Mar 23:01:54.868 * MASTER MODE enabled (user request from 'id=8 addr=192.168.10.128:37585 fd=6 name=sentinel-21e629e6-cmd age=383 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=rw cmd=exec')
2050:M 07 Mar 23:01:54.870 # CONFIG REWRITE executed with success.

4、再查看Sentinel的监视信息，可以看到新的Redis_Master已经是192.168.10.131了：

[root@Sentinel_1 sentinel]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=master-6379,status=ok,address=192.168.10.131:6379,slaves=1,sentinels=3
127.0.0.1:26379>

故障转移后的邮件报警如下：



5、把down机的Redis启动，Sentinel 又会把它加进来来，作为Slave的角色：

[root@Redis_Slave redis]# sh redis start
Starting Redis server...
[root@Redis_Slave redis]#

查看Sentinel log：

2115:X 07 Mar 23:18:49.494 * +convert-to-slave slave 192.168.10.132:6379 192.168.10.132 6379 @ master-6379 192.168.10.131 6379

再查看Redis-Master log：

[root@Redis_Master redis]# tail -f logs/redis_6379.log
2050:S 07 Mar 23:01:52.367 # Error condition on socket for SYNC: Connection refused
2050:S 07 Mar 23:01:53.382 * Connecting to MASTER 192.168.10.132:6379
2050:S 07 Mar 23:01:53.384 * MASTER <-> SLAVE sync started
2050:S 07 Mar 23:01:53.384 # Error condition on socket for SYNC: Connection refused
2050:S 07 Mar 23:01:54.404 * Connecting to MASTER 192.168.10.132:6379
2050:S 07 Mar 23:01:54.405 * MASTER <-> SLAVE sync started
2050:S 07 Mar 23:01:54.406 # Error condition on socket for SYNC: Connection refused
2050:M 07 Mar 23:01:54.868 * Discarding previously cached master state.
2050:M 07 Mar 23:01:54.868 * MASTER MODE enabled (user request from 'id=8 addr=192.168.10.128:37585 fd=6 name=sentinel-21e629e6-cmd age=383 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=rw cmd=exec')
2050:M 07 Mar 23:01:54.870 # CONFIG REWRITE executed with success.
2050:M 07 Mar 23:10:22.009 * 1 changes in 900 seconds. Saving...
2050:M 07 Mar 23:10:22.126 * Background saving started by pid 2084
2084:C 07 Mar 23:10:22.167 * DB saved on disk
2084:C 07 Mar 23:10:22.167 * RDB: 4 MB of memory used by copy-on-write
2050:M 07 Mar 23:10:22.229 * Background saving terminated with success
2050:M 07 Mar 23:18:49.389 * Slave 192.168.10.132:6379 asks for synchronization
2050:M 07 Mar 23:18:49.389 * Full resync requested by slave 192.168.10.132:6379
2050:M 07 Mar 23:18:49.389 * Starting BGSAVE for SYNC with target: disk
2050:M 07 Mar 23:18:49.417 * Background saving started by pid 2085
2085:C 07 Mar 23:18:49.428 * DB saved on disk
2085:C 07 Mar 23:18:49.429 * RDB: 4 MB of memory used by copy-on-write
2050:M 07 Mar 23:18:49.479 * Background saving terminated with success
2050:M 07 Mar 23:18:49.479 * Synchronization with slave 192.168.10.132:6379 succeeded

再查看Redis_Slave的log：

2514:S 07 Mar 23:18:48.859 # CONFIG REWRITE executed with success.
2514:S 07 Mar 23:18:49.049 * Connecting to MASTER 192.168.10.131:6379
2514:S 07 Mar 23:18:49.053 * MASTER <-> SLAVE sync started
2514:S 07 Mar 23:18:49.055 * Non blocking connect for SYNC fired the event.
2514:S 07 Mar 23:18:49.059 * Master replied to PING, replication can continue...
2514:S 07 Mar 23:18:49.065 * Partial resynchronization not possible (no cached master)
2514:S 07 Mar 23:18:49.099 * Full resync from master: 3e1fbd2ec6f57b3362687051ab1bb6edf1d2ee27:1
2514:S 07 Mar 23:18:49.157 * MASTER <-> SLAVE sync: receiving 34 bytes from master
2514:S 07 Mar 23:18:49.157 * MASTER <-> SLAVE sync: Flushing old data
2514:S 07 Mar 23:18:49.157 * MASTER <-> SLAVE sync: Loading DB in memory
2514:S 07 Mar 23:18:49.157 * MASTER <-> SLAVE sync: Finished with success

可以看到Redis主从同步还是正常运行的。更多的测试就留给同学们了^o^

总结：

一、Redis-Sentinel是Redis官方推荐的高可用性(HA)解决方案，还是比较可靠的，推荐大家在生产环境部署并使用

二、Redis-Sentinel可以自定义故障转移脚本，这还是比较人性化的，可以结合shell脚本或者Python脚本

三、现在Redis高可用架构非常多，但各有优劣，需要说的是，如果要上Redis高可用架构，需要反复测试。

参考资料：

http://segmentfault.com/a/1190000002680804

http://segmentfault.com/a/1190000002685515

http://redis.io/topics/sentinel-clients

https://pypi.python.org/pypi/redis/

http://www.cnblogs.com/gomysql/p/5040847.html