通过Keepalived实现Redis Failover自动故障切换

环境介绍：
Master: 192.168.0.1
Slave: 192.168.0.2
Virtural IP Address (VIP): 192.168.0.253

设计思路：
当 Master 与 Slave 均运作正常时, Master负责服务，Slave负责Standby；
当 Master 挂掉，Slave 正时, Slave接管服务，同时关闭主从复制功能；
当 Master 恢复正常，则从Slave同步数据，同步数据之后关闭主从复制功能，恢复Master身份，于此同时Slave等待Master同步数据完成之后，恢复Slave身份。
然后依次循环。

需要注意的是，这样做需要在Master与Slave上都开启本地化策略，否则在互相自动切换的过程中，未开启本地化的一方会将另一方的数据清空，造成数据完全丢失。

下面，是具体的实施步骤：

在Master和Slave上安装Keepalived
$ yum install keepalived

修改Master和Slave的/etc/hosts文件
$ vim /etc/hosts

127.0.0.1 localhost

192.168.0.1 redis

192.168.0.2 redis-slave

默认安装完成keepalived之后是没有配置文件的，因此我们需要手动创建：

首先，在Master上创建如下配置文件：
$ vim /etc/keepalived/keepalived.conf

vrrp_script chk_redis {

script "/etc/keepalived/scripts/redis_check.sh" ###监控脚本

interval 2 ###监控时间

}

vrrp_instance VI_1 {

state MASTER ###设置为MASTER

interface eth0 ###监控网卡

virtual_router_id 51

priority 101 ###权重值

authentication {

auth_type PASS ###加密

auth_pass redis ###密码

}

track_script {

chk_redis ###执行上面定义的chk_redis

}

virtual_ipaddress {

192.168.0.253 ###VIP

}

notify_master /etc/keepalived/scripts/redis_master.sh

notify_backup /etc/keepalived/scripts/redis_backup.sh

notify_fault /etc/keepalived/scripts/redis_fault.sh

notify_stop /etc/keepalived/scripts/redis_stop.sh

}

然后，在Slave上创建如下配置文件：
$ vim /etc/keepalived/keepalived.conf

vrrp_script chk_redis {

script "/etc/keepalived/scripts/redis_check.sh" ###监控脚本

interval 2 ###监控时间

}

vrrp_instance VI_1 {

state BACKUP ###设置为BACKUP

interface eth0 ###监控网卡

virtual_router_id 51

priority 100 ###比MASTRE权重值低

authentication {

auth_type PASS

auth_pass redis ###密码与MASTRE相同

}

track_script {

chk_redis ###执行上面定义的chk_redis

}

virtual_ipaddress {

192.168.0.253 ###VIP

}

notify_master /etc/keepalived/scripts/redis_master.sh

notify_backup /etc/keepalived/scripts/redis_backup.sh

notify_fault /etc/keepalived/scripts/redis_fault.sh

notify_stop /etc/keepalived/scripts/redis_stop.sh

}

在Master和Slave上创建监控Redis的脚本
$ mkdir /etc/keepalived/scripts
$ vim /etc/keepalived/scripts/redis_check.sh

#!/bin/bash

0ALIVE=`/opt/redis/bin/redis-cli PING`

if [ "$ALIVE" == "PONG" ]; then

echo $ALIVE

exit 0

else

echo $ALIVE

exit 1

fi

编写以下负责运作的关键脚本：
notify_master /etc/keepalived/scripts/redis_master.sh
notify_backup /etc/keepalived/scripts/redis_backup.sh
notify_fault /etc/keepalived/scripts/redis_fault.sh
notify_stop /etc/keepalived/scripts/redis_stop.sh

因为Keepalived在转换状态时会依照状态来呼叫：
当进入Master状态时会呼叫notify_master
当进入Backup状态时会呼叫notify_backup
当发现异常情况时进入Fault状态呼叫notify_fault
当Keepalived程序终止时则呼叫notify_stop

首先，在Redis Master上创建notity_master与notify_backup脚本：
$ vim /etc/keepalived/scripts/redis_master.sh

#!/bin/bash

REDISCLI="/opt/redis/bin/redis-cli"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF 192.168.0.2 6379 >> $LOGFILE 2>&1

sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

$ vim /etc/keepalived/scripts/redis_backup.sh

#!/bin/bash

REDISCLI="/opt/redis/bin/redis-cli"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[backup]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF 192.168.0.2 6379 >> $LOGFILE 2>&1

接着，在Redis Slave上创建notity_master与notify_backup脚本：

$ vim /etc/keepalived/scripts/redis_master.sh

#!/bin/bash

REDISCLI="/opt/redis/bin/redis-cli"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF 192.168.0.1 6379 >> $LOGFILE 2>&1

sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

$ vim /etc/keepalived/scripts/redis_backup.sh

#!/bin/bash

REDISCLI="/opt/redis/bin/redis-cli"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[backup]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF 192.168.0.1 6379 >> $LOGFILE 2>&1

然后在Master与Slave创建如下相同的脚本：
$ vim /etc/keepalived/scripts/redis_fault.sh

#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo "[fault]" >> $LOGFILE

date >> $LOGFILE

$ vim /etc/keepalived/scripts/redis_stop.sh

#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo "[stop]" >> $LOGFILE

date >> $LOGFILE

给脚本都加上可执行权限：
$ chmod +x /etc/keepalived/scripts/*.sh

脚本创建完成以后，我们开始按照如下流程进行测试：
1.启动Master上的Redis
$ /etc/init.d/redis start

2.启动Slave上的Redis
$ /etc/init.d/redis start

3.启动Master上的Keepalived
$ /etc/init.d/keepalived start

4.启动Slave上的Keepalived
$ /etc/init.d/keepalived start

5.尝试通过VIP连接Redis:
$ redis-cli -h 192.168.0.253 INFO

连接成功，Slave也连接上来了。
role:master
slave0:192.168.0.2,6379,online

6.尝试插入一些数据：
$ redis-cli -h 192.168.0.253 SET Hello Redis
OK

从VIP读取数据
$ redis-cli -h 192.168.0.253 GET Hello
"Redis"

从Master读取数据
$ redis-cli -h 192.168.0.1 GET Hello
"Redis"

从Slave读取数据
$ redis-cli -h 192.168.0.2 GET Hello
"Redis"

下面，模拟故障产生：
将Master上的Redis进程杀死：
$ killall -9 redis-server

查看Master上的Keepalived日志
$ tail -f /var/log/keepalived-redis-state.log
[fault]
Thu Sep 27 08:29:01 CST 2012

同时Slave上的日志显示：
$ tail -f /var/log/keepalived-redis-state.log
[master]
Fri Sep 28 14:14:09 CST 2012
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK

然后我们可以发现，Slave已经接管服务，并且担任Master的角色了。
$ redis-cli -h 192.168.0.253 INFO
$ redis-cli -h 192.168.0.2 INFO
role:master

然后我们恢复Master的Redis进程
$ /etc/init.d/redis start

查看Master上的Keepalived日志
$ tail -f /var/log/keepalived-redis-state.log
[master]
Thu Sep 27 08:31:33 CST 2012
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK

同时Slave上的日志显示：
$ tail -f /var/log/keepalived-redis-state.log
[backup]
Fri Sep 28 14:16:37 CST 2012
Being slave....
Run SLAVEOF cmd ...
OK

可以发现目前的Master已经再次恢复了Master的角色，故障切换以及自动恢复都成功了。