HA 集群软件 keepalived 详解

下面的内容根据上课笔记总结而来
1、HA集群中的相关术语
1.1 节点（node）
运行keepalived进程的一个独立主机，称为节点，节点是ＨＡ的核心组成部分，每个节点上运行着操作系统和ｋｅｅｐａｌｉｖｅｄ软件服务，在ｋｅｅｐａｌｉｖｅｄ集群中，节点有主次之分，分别称为主节点和备用节点，每个节点拥有唯一的主机名，并且拥有属于自己的一组资源，例如　　磁盘，文件系统，网络地址和应用服务等，主节点一般运行着一个或多个应用服务，而备用节点一般处于监控状态
１.2 资源（resource）
资源是一个节点可以控制的实体，并且当节点发生故障时，这些资源能够被其他节点接管，keepalived中，可以当做资源的实体有
磁盘分区 文件系统
VIP地址
应用程序服务
NFS文件系统
3.事件（event）
也就是集群中可能发生的事情，例如节点系统故障，网络连通故障，网卡驱动，应用程序故障等，这些事情都会导致节点的资源发生转移，HA的测试也是基于这些事情来进行的
4.动作（action）
事件发生时HA的响应方式，动作是由shell脚本控制的，例如当某个节点发送故障后，备份节点将通过事先设定好的执行脚本进行服务的关闭或启动，进而接管故障节点的资源
2、keepalived简介
keepalived是LInux下一个轻量级的高可用解决方案，他与heartbeat 实现的功能类似，都可以实现服务或网络的高可用，但又有区别，heartbeat是一个专业的，功能完善的高可用软件，它提供了HA软件所需的基本功能，比如心跳检测和资源接管，监测集群中的系统服务，在集群节点间转移vip的，heartbeat功能强大，但是部署和使用相对比较麻烦，与heartbeat相比，keepalived主要通过vrrp协议来实现高可用功能，部署和使用非常简单，所有配置只需一个配置文件即可
2.1 keepalived的用途
keepalived起初视为lvs设计的，专门用来监控集群系统中各个服务节点的状态，他根据layer 3,4交换机制检查每个服务节点的状态，如果某个服务节点出现异常，或工作出现故障，keepalived将检测到，并将出现故障的服务节点从集群系统中剔除，而在故障节点恢复正常后，keepalived又可以自动将服务节点重新加入到服务器集群中，而这些工作全部都自动完成，不需要人工干预，需要人工干预的只是修复出现故障的服务节点
keepalived后来又引入了vrrp，他的出现就是解决单点故障，实现高可用功能，因此keepalived一方面具有服务器状态监测和故障隔离功能，另一方面也具有HA功能，对于vrrp协议不明白的可以看我之前写过的hsrp及vrrp

2.2 keepalived的体系架构
keepalived是一个高度模块化的软件，结构简单，但扩展性很强，下面是keepalived体系架构图


从图中可以看出，keepalived从整体上分为两层，用户空间层和内核空间层
内核空间层包含ipvs和NETLINK两个模块，ipvs是keepalived引入的一个第三方模块，通过ipvs可以实现基于IP的负载均衡集群，ipvs默认包含在lvs软件中

NETLINK模块主要用于实现一些高级路由框架和一些相关的网络功能，完成用户空间层netlink reflector模块发来的各种网络请求
用户空间层位于内核层之上，keepalived所有功能都在这里实现
3、keepalived的安装 ( 操作系统centos6.7)

[root@centos01 tools]# wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
[root@centos01 tools]# tar xf keepalived-1.2.12.tar.gz
[root@centos01 tools]# cd keepalived-1.2.12
[root@centos01 keepalived-1.2.12]# ./configure --sysconf=/etc --with-kernel-dir=/usr/src/kernels/2.6.32-573.el6.x86_64/


[root@centos01 keepalived-1.2.12]# make && make install




[root@centos01 keepalived-1.2.12]# ln -s /usr/local/sbin/keepalived /sbin
[root@centos01 keepalived-1.2.12]# chkconfig --add keepalived
[root@centos01 keepalived-1.2.12]# chkconfig --level 345 keepalived on

说明：
--sysconf 指定了keepalived配置文件的安装路径，即路径为 /etc/keepalived/keepalived.conf

--with-kernel-dir 是个很重要的参数，这个参数并不是要把keepalived编译进内核，而是指定使用内核源代码中的头文件，即include目录，只有在使用lvs是才需要用到这个参数，其他时候是不需要的

4.keepalived基础功能应用实例
作为一个高可用集群软件，keepalived没有heartbeat专业的高可用集群软件功能强大，它不能实现集群资源的托管，也不能实现对集群中运行服务的监控，但这并不妨碍keepalived的易用性，他提供了
vrrp_script nofify_master nofify_backup等多个功能模块，通过这些模块也可以实现对集群自的托管以及集群服务的监控
4.1keepalived集成HA功能演示
在默认情况下，keepalived可以实现对系统死机，网络异常及keepalived本身进行监控，但这些还是不够的 ，因为集群中运行的服务也随时可能出现问题，因此还需要对集群运行服务的状态进行监控，当服务出现问题时也进行主备切换，keepalived作为一个优秀的高可用集群软件也考虑到了这一点，它提供了一个vrrp_script模块专门用来对集群服务资源进行监控
keepalived高可用集群环境部署说明 vip地址172.16.80.100





主节点配置文件如下
[root@centos01 keepalived]# cat keepalived.conf

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id http_master
}
vrrp_script check_httpd {
script "killall -0 httpd"
interval 2
}
vrrp_instance HA_1 {
state MASTER
interface eth0
virtual_router_id 80
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
notify_master "/etc/keepalived/master.sh"
notify_backup "/etc/keepalived/backup.sh"
notify_fault "/etc/keepalived/fault.sh"
track_script {
check_httpd
}
virtual_ipaddress {
172.16.80.100
}
}

备用节点配置如下
[root@centos02 keepalived]# cat keepalived.conf

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id http_backup
}
vrrp_script check_httpd {
script "killall -0 httpd"
interval 2
}
vrrp_instance HA_1 {

state BACKUP
interface eth0
virtual_router_id 80
priority 85
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
notify_master "/etc/keepalived/master.sh"
notify_backup "/etc/keepalived/backup.sh"
notify_fault "/etc/keepalived/fault.sh"
track_script {
check_httpd
}
virtual_ipaddress {
172.16.80.100
}
}

其中 master.sh 文件内容如下
[root@centos01 keepalived]# cat master.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-http-state.log
echo "[Master]">>$LOGFILE
date >> $LOGFILE

backup.sh内容如下
[root@centos01 keepalived]# cat backup.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-http-state.log
echo "[Backup]">>$LOGFILE
date >> $LOGFILE

fault.sh内容如下
[root@centos01 keepalived]# cat fault.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-http-state.log
echo "[Fault]">>$LOGFILE
date >> $LOGFILE

这三个脚本是监控keepalived角色的切换过程

4.2keepalived的启动过程分析
将配置好的master.sh backup.sh fault.sh三个文件一起复制到keepalived 备用节点的对应路径下面
先启动httd服务 最后启动keepalived服务

[root@centos01 keepalived]# /etc/init.d/httpd start
[root@centos01 keepalived]# /etc/init.d/keepalived start
观察主节点日志




Jul 28 11:40:41 centos01 Keepalived[65330]: Starting Keepalived v1.2.12 (07/28,2016)
Jul 28 11:40:41 centos01 Keepalived[65331]: Starting Healthcheck child process, pid=65333
Jul 28 11:40:41 centos01 Keepalived[65331]: Starting VRRP child process, pid=65334
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Netlink reflector reports IP 172.16.80.116 added
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Netlink reflector reports IP 172.16.10.100 added
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Netlink reflector reports IP 172.16.80.116 added
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Netlink reflector reports IP 172.16.10.100 added
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Netlink reflector reports IP fe80::20c:29ff:fe4c:62c9 added
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Netlink reflector reports IP fe80::20c:29ff:fe4c:62d3 added
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Registering Kernel netlink reflector
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Registering Kernel netlink command channel
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Registering gratuitous ARP shared channel
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Netlink reflector reports IP fe80::20c:29ff:fe4c:62c9 added
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Netlink reflector reports IP fe80::20c:29ff:fe4c:62d3 added
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Registering Kernel netlink reflector
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Registering Kernel netlink command channel
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Configuration is using : 7653 Bytes
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Configuration is using : 65695 Bytes
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: Using LinkWatch kernel netlink reflector...
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jul 28 11:40:41 centos01 Keepalived_healthcheckers[65333]: Using LinkWatch kernel netlink reflector...
Jul 28 11:40:41 centos01 Keepalived_vrrp[65334]: VRRP_Script(check_httpd) succeeded
Jul 28 11:40:43 centos01 Keepalived_vrrp[65334]: VRRP_Instance(HA_1) Transition to MASTER STATE
Jul 28 11:40:45 centos01 Keepalived_vrrp[65334]: VRRP_Instance(HA_1) Entering MASTER STATE
Jul 28 11:40:45 centos01 Keepalived_vrrp[65334]: VRRP_Instance(HA_1) setting protocol VIPs.
Jul 28 11:40:45 centos01 Keepalived_vrrp[65334]: VRRP_Instance(HA_1) Sending gratuitous ARPs on eth0 for 172.16.80.100
Jul 28 11:40:45 centos01 Keepalived_healthcheckers[65333]: Netlink reflector reports IP 172.16.80.100 added
Jul 28 11:40:50 centos01 Keepalived_vrrp[65334]: VRRP_Instance(HA_1) Sending gratuitous ARPs on eth0 for 172.16.80.100




[root@centos01 keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:4c:62:c9 brd ff:ff:ff:ff:ff:ff
inet 172.16.80.116/24 brd 172.16.80.255 scope global eth0
inet 172.16.80.100/32 scope global eth0
inet6 fe80::20c:29ff:fe4c:62c9/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:4c:62:d3 brd ff:ff:ff:ff:ff:ff
inet 172.16.10.100/24 brd 172.16.10.255 scope global eth1
inet6 fe80::20c:29ff:fe4c:62d3/64 scope link
valid_lft forever preferred_lft forever

观察备用节点日志



[root@centos02 keepalived]# tail -f /var/log/messages
Jul 28 11:40:52 centos02 Keepalived[4360]: Starting Keepalived v1.2.12 (07/22,2016)
Jul 28 11:40:52 centos02 Keepalived[4361]: Starting Healthcheck child process, pid=4363
Jul 28 11:40:52 centos02 Keepalived[4361]: Starting VRRP child process, pid=4364
Jul 28 11:40:52 centos02 Keepalived_healthcheckers[4363]: Netlink reflector reports IP 172.16.80.117 added
Jul 28 11:40:52 centos02 Keepalived_vrrp[4364]: Netlink reflector reports IP 172.16.80.117 added
Jul 28 11:40:52 centos02 Keepalived_healthcheckers[4363]: Netlink reflector reports IP fe80::20c:29ff:fe45:fe30 added
Jul 28 11:40:52 centos02 Keepalived_healthcheckers[4363]: Registering Kernel netlink reflector
Jul 28 11:40:52 centos02 Keepalived_healthcheckers[4363]: Registering Kernel netlink command channel
Jul 28 11:40:52 centos02 Keepalived_vrrp[4364]: Netlink reflector reports IP fe80::20c:29ff:fe45:fe30 added
Jul 28 11:40:52 centos02 Keepalived_vrrp[4364]: Registering Kernel netlink reflector
Jul 28 11:40:52 centos02 Keepalived_vrrp[4364]: Registering Kernel netlink command channel
Jul 28 11:40:52 centos02 Keepalived_vrrp[4364]: Registering gratuitous ARP shared channel
Jul 28 11:40:53 centos02 Keepalived_healthcheckers[4363]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 28 11:40:53 centos02 Keepalived_healthcheckers[4363]: Configuration is using : 7508 Bytes
Jul 28 11:40:53 centos02 Keepalived_healthcheckers[4363]: Using LinkWatch kernel netlink reflector...
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: Configuration is using : 65550 Bytes
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: Using LinkWatch kernel netlink reflector...
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: VRRP_Instance(HA_1) Entering BACKUP STATE
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jul 28 11:40:53 centos02 Keepalived_vrrp[4364]: VRRP_Script(check_httpd) succeeded
[root@centos02 keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:45:fe:30 brd ff:ff:ff:ff:ff:ff
inet 172.16.80.117/24 brd 172.16.80.255 scope global eth0
inet6 fe80::20c:29ff:fe45:fe30/64 scope link
valid_lft forever preferred_lft forever
可以看到在备用节点上面是不会VIP地址的，如果别用节点也出现了VIP地址，说明集群出现了脑裂，这是一种不正常的情况
这次我们先加快的的介绍到这里，keepalived的缺点就是配置文件没有语法检查功能，配置错了 依然能启动，这点需要特别注意，下次我们再来介绍故障切换及恢复 及通过vrrp_script实现对集群资源的监控