Heartbeat

Heartbeat

作者：Danbo 时间：2015-7-23

通过heartbeat可以将资源（VIP及程序服务等资源）从一台故障计算机快速转移到另一台运转正常的机器继续提供服务。

Heartbeat工作原理：
通过修改heartbeat的配置文件，指定哪一台heartbeat服务器作为主服务器，则另一台将自动成为备份服务器。然后在备份服务器上配置heartbeat守护程序来监听来自主服务器的心跳。如若备份服务器在指定时间内未监听来自主服务器的心跳，就会启动故障转移程序，并取得服务器上的相关资源服务的所有权，接替主服务器继续不间断的提供服务，以达到资源服务高可用的目的。以上是主备模式，还有一种是主主模式（互为主备），双方都发送报文，当指定时间内未收到对方发送的心跳报文，那么就认为对方失效，此时就会接管运行在对方主机上的资源或服务。

Heartbeat的心跳连接
两台heartbeat主机之间的通信一般常用的方法：
1.串行电缆（缺点：距离不能太远）
2.一根以太网电缆（网线）两网卡直连。（最常用）
3.以太网电缆，通过交换机等网络设备连接。（这样增加了交换机故障点，同时线路不是专用心跳，容易受到其他数据传输的影响）

Heartbeat裂脑
什么是裂脑？由于两台高可用服务器在规定的时间内，无法互相检测到对方能心跳而各自启动故障转移功能，取得了资源及服务的所有权，而此时的两台高可用服务器对都还活着并且正在正常运行，这样就会导致同一个IP或服务在两端同时启动而发送冲突的严重问题，最严重的是两天主机占用同一个IP地址，这种情况就被称为裂脑，英文：split brain。

导致裂脑发生的多种原因：
-高可用服务器对之间心跳链路故障，导致无法正常通信。
-高可用服务器对上开启了防火墙阻挡了心跳消息的传输。
-高可用服务器对上心跳网卡地址等信息配置不正确，导致传输失败。
-两端心跳方式不同，心跳广播冲突。

防止举措：
-加冗余线路，两台心跳线路
-检测到裂脑，强行关闭一个心跳节点。需要特殊设备支持比如：Stonith、Fence（可编程电源控制器）等。
-做好对裂脑的监控报警（邮件手机短信等）
-启动磁盘锁。正在服务的一方锁定共享磁盘。正在服务的一方只有发现心跳全部断开后才启动磁盘锁。
-增加仲裁机制：当心跳线完全断开时，两个节点各自ping一下参考网关，那一端不通则表明断点就在本端，能够ping通的一端接管服务，ping不通的一端自动重启。以此释放共享资源。

Heartbeat消息类型
-心跳消息
-集群转换消息
-重传请求

心跳消息：大约为150字节，可能为单播、广播、组播的方式
集群转换消息：ip-request和ip-request-resp
当主服务器恢复在线状态时，通过ip-request消息要求备机释放主服务器失败时备服务器取得的资源，然后备份服务器关闭资源及服务。备机释放资源及服务后，主服务器收到ip-request-resp消息通知主服务器接管资源及服务。
重传请求：rexmit-request控制重传心跳请求。

以上心跳控制消息使用的是UDP协议发送到/etc/ha.d/ha.cf文件指定的任意端口。

Heartbeat IP地址接管和故障转移
Heartbeat IP是通过IP地址接管和ARP广播进行地址转移的。
ARP广播（免费arp）：在主服务器故障时，备用节点接管资源后，会立即更新所有客户端本地的ARP表。
注意这里所谓的客户端是指与heartbeat高可用服务器在同一局域网中的客户端，并不是最终的联网用户。客户机是相对于heartbeat高可用服务器来说的。

VIP是虚拟IP，其实就是绑定在物理网卡上的别名IP，如eth0:x，你可以在一块网刊上绑定多个别名，在实际的生产环境中，需要在DNS配置中把网站域名地址解析到这个VIP地址，有这个VIP对用户提供服务。

配置VIP的方法：
heartbeat：ifconfig eth0:1 192.168.1.2 netmask 255.255.255.0 up
keepalived：ip addr add 192.168.1.2/24 broadcast 192.168.1.255 dev eth1（辅助IP）
ifconfig 能看到heartbeat配置的VIP，ip add方式能看到以辅助IP方式增加的VIP

删除VIP的方法：
heartbeat：ifconfig eth0:1 192.168.1.2 netmask 255.255.255.0 down 或 ifconfig eth0:1 down
keepalived：ip addr del 192.168.1.2/24 broadcast 192.168.1.255 dev eth1

Heartbeat配置文件
ha.cf heartbeat参数的配置文件
authke beartbeat认证文件
haresourcce heartbeat资源配置文件如IP资源及脚本程序。

部署Heartbeat需求
拓扑图如下：



Heartbeat服务器主机资源规划：

给虚拟机配置IP和主机名
IP配置:

DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.23.255
HWADDR=00:0C:29:1B:C7:C9
IPADDR=192.168.23.128
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.23.0
ONBOOT=yes

　　

我们配置主机名：/etc/sysconfig/network设置HOSTNAME=data1和data2
然后执行hostname data1和data2生效

配置hosts文件：用于解析内网，因为在heartbeat的配置文件中使用了机器名。
echo "172.16.1.129 data1" >>/etc/hosts
echo "172.16.1.132 data1" >>/etc/hosts

此时我们发现启动后系统卡在Starting sendmail处，此时我们需要关闭sendmail服务的自动启动:
killall sendmail
chkconfig --list | grep sendmail
chkconfig --level 2345 sendmail off

添加主机路由，用于心跳通信，两台机器都要添加。
[root@data1 /]# route add -host 10.0.1.132 dev eth2
[root@data1 /]# echo 'route add -host 10.0.1.132 dev eth2' >>/etc/rc.d/rc.local
上一条命令是让路由保存在开机文件中，下次启动自动加入。

安装heartbeat软件

yum安装rpm包后本地不清除安装后的rpm包：
sed -i 's/keepcache=0/keepcache=1/g' /etc/yum.conf
这里我们采用yum方式安装：yum -y install heartbeat
#注意必须要执行两边，第一遍会提醒：
Failed:
heartbeat.x86_64 0:2.1.3-3.el5.centos
-可以自动解决以来关系，就是rpm安装的方式，方面简单，当不灵活，编译方式可以定制安装，灵活，但复杂。不过大公司会自己定制适合自己使用的rpm包，放在yum仓库里。
安装完毕后进入heartbeat的doc的默认目录：cd /usr/share/doc/heartbeat-2.1.3/
我们主要看：ll ha.cf authkeys haresources

cat /ha.cf
首先与日志相关的配置如下所示：

# File to write debug messages to

debugfile /var/log/ha-debug
#
#  File to write other messages to
logfile /var/log/ha-log
#
# Facility to use for syslog()/logger
logfacility local0

以下是一些计时器的基础参数一般不改动：
keepalive 2
deadtime 30
warntime 10      #发出警告时间
initdead 120     #Heartbeat守护进程首次启动后应该等待120s后再启动主服务器上的资源。

以下行表示使用组播方式，需要改动的有eth2，即改为心跳先的那个网卡：
#serial /dev/ttyS0 #采用串口就在这配置。
mcast [dev] [mcast group] [port] [ttl] [loop]   mcast group这个每台主机都不能一样。
改为mcast eth2 255.0.0.129 694 1 0

以下行表示当Master宕机后重新恢复回来，即是否抢占模式。
auto_failback on   #开启抢占模式
node    data1   #即两台存储Server的主机名。这就是我们为什么要修改hosts绑定IP与主机名，必须与route -n显示的一致
node    data2

crm     no  #表示clusters resource management。这个no。
ping    10.10.10.254    #通过ping参考网关的方式来检测HA服务武器是否正常工作，这个可以用于检测裂脑行为。

　　

配置authkey文件

[root@data1 heartbeat-2.1.3]# cat authkeys
#
#       Authentication file.  Must be mode 600   #注意这个文件的权限不需为600
#
#
#       Must have exactly one auth directive at the front.
#       auth    send authentication using this method-id
#
#       Then, list the method and key that go with that method-id
#
#       Available methods: crc sha1, md5.  Crc doesn't need/want a key.(三种认证方式，CRC的不需要key)
#
#       You normally only have one authentication method-id listed in this file
#
#       Put more than one to make a smooth transition when changing auth
#       methods and/or keys.
#
#
#       sha1 is believed to be the "best", md5 next best.
#
#       crc adds no security, except from packet corruption.
#               Use only on physically secure networks.
#
auth 1  #配置哪个有效。
1 crc  (默认)
#2 sha1 HI!
#3 md5 Hello!

配置haresources

配置haresource文件
编辑配置heartbeat资源文件：
[root@data1 heartbeat-2.1.3]# cat haresources
#
#       This is a list of resources that move from machine to machine as
#       nodes go down and come up in the cluster.  Do not include
#       "administrative" or fixed IP addresses in this file.
#
# <VERY IMPORTANT NOTE>
#       The haresources files MUST BE IDENTICAL on all nodes of the cluster.
#
#       The node names listed in front of the resource group information
#       is the name of the preferred node to run the service.  It is
#       not necessarily the name of the current machine.  If you are running
#       auto_failback ON (or legacy), then these services will be started
#       up on the preferred nodes - any time they're up.
#
#       If you are running with auto_failback OFF, then the node information
#       will be used in the case of a simultaneous start-up, or when using
#       the hb_standby {foreign,local} command.
#
#       BUT FOR ALL OF THESE CASES, the haresources files MUST BE IDENTICAL.
#       If your files are different then almost certainly something
#       won't work right.
# </VERY IMPORTANT NOTE>
#
#
#       We refer to this file when we're coming up, and when a machine is being
#       taken over after going down.
#
#       You need to make this right for your installation, then install it in
#       /etc/ha.d
#
#       Each logical line in the file constitutes a "resource group".
#       A resource group is a list of resources which move together from
#       one node to another - in the order listed.  It is assumed that there
#       is no relationship between different resource groups.  These
#       resource in a resource group are started left-to-right, and stopped
#       right-to-left.  Long lists of resources can be continued from line
#       to line by ending the lines with backslashes ("\").
#
#       These resources in this file are either IP addresses, or the name
#       of scripts to run to "start" or "stop" the given resource.
#
#       The format is like this:
#
#node-name resource1 resource2 ... resourceN   #绑定服务器。
#IPaddr为heartbeat配置IP的默认脚本，其后的IP等都是脚本的参数。
data1 IPaddr::10.1.1.129/24/eth0     #这个为集群对外的VIP，初始启动在data1上，将VIP绑定在eth0上，为heartbeat提供对外服务的通信接口。相当于执行了：/etc/ha.d/resource.d/IPaddr 10.1.1.129/24/eth0 start
data2 IPaddr::10.1.1.129/24/eth0
#
#
#       If the resource name contains an :: in the middle of it, the
#       part after the :: is passed to the resource script as an argument.
#       Multiple arguments are separated by the :: delimeter
#
#       In the case of IP addresses, the resource script name IPaddr is
#       implied.
#
#       For example, the IP address 135.9.8.7 could also be represented
#       as IPaddr::135.9.8.7
#
#       THIS IS IMPORTANT!!     vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#
#       The given IP address is directed to an interface which has a route
#       to the given address.  This means you have to have a net route
#       set up outside of the High-Availability structure.  We don't set it
#       up here -- we key off of it.
#
#       The broadcast address for the IP alias that is created to support
#       an IP address defaults to the highest address on the subnet.
#
#       The netmask for the IP alias that is created defaults to the same
#       netmask as the route that it selected in in the step above.
#
#       The base interface for the IPalias that is created defaults to the
#       same netmask as the route that it selected in in the step above.
#
#       If you want to specify that this IP address is to be brought up
#       on a subnet with a netmask of 255.255.255.0, you would specify
#       this as IPaddr::135.9.8.7/24 .
#
#       If you wished to tell it that the broadcast address for this subnet
#       was 135.9.8.210, then you would specify that this way:
#               IPaddr::135.9.8.7/24/135.9.8.210
#
#       If you wished to tell it that the interface to add the address to
#       is eth0, then you would need to specify it this way:
#               IPaddr::135.9.8.7/24/eth0
#
#       And this way to specify both the broadcast address and the
#       interface:
#               IPaddr::135.9.8.7/24/eth0/135.9.8.210
#
#       The IP addresses you list in this file are called "service" addresses,
#       since they're they're the publicly advertised addresses that clients
#       use to get at highly available services.
#
#       For a hot/standby (non load-sharing) 2-node system with only
#       a single service address,
#       you will probably only put one system name and one IP address in here.
#       The name you give the address to is the name of the default "hot"
#       system.
#
#       Where the nodename is the name of the node which "normally" owns the
#       resource.  If this machine is up, it will always have the resource
#       it is shown as owning.
#
#       The string you put in for nodename must match the uname -n name
#       of your machine.  Depending on how you have it administered, it could
#       be a short name or a FQDN.
#
#-------------------------------------------------------------------
#
#       Simple case: One service address, default subnet and netmask
#               No servers that go up and down with the IP address
#
#just.linux-ha.org      135.9.216.110
#
#-------------------------------------------------------------------
#
#       Assuming the adminstrative addresses are on the same subnet...
#       A little more complex case: One service address, default subnet
#       and netmask, and you want to start and stop http when you get
#       the IP address...
#
#just.linux-ha.org      135.9.216.110 http
#-------------------------------------------------------------------
#
#       A little more complex case: Three service addresses, default subnet
#       and netmask, and you want to start and stop http when you get
#       the IP address...
#
#just.linux-ha.org      135.9.216.110 135.9.215.111 135.9.216.112 httpd
#-------------------------------------------------------------------
#
#       One service address, with the subnet, interface and bcast addr
#       explicitly defined.
#
#just.linux-ha.org      135.9.216.3/28/eth0/135.9.216.12 httpd
#
#-------------------------------------------------------------------
#
#       An example where a shared filesystem is to be used.
#       Note that multiple aguments are passed to this script using
#       the delimiter '::' to separate each argument.
#
#node1  10.0.0.170 Filesystem::/dev/sda1::/data1::ext2
#
#       Regarding the node-names in this file:
#
#       They must match the names of the nodes listed in ha.cf, which in turn
#       must match the `uname -n` of some node in the cluster.  So they aren't
#       virtual in any sense of the word.

此时在配置VIP的时候前面的主机名是Master的主机名。备份机上配置的也是Master的主机名。

当我们对以上三个文件配置完毕后，我们就需要将其配置好的文件上传到/etc/ha.d/目录内。
然后我们在两台虚拟机上启动heartbeat:/etc/init.d/heartbeat start
在此一定要关闭防火墙，否则两台主机都会启动，发生裂脑：/etc/init.d/iptables stop
此时我们发现主机为data1，此时我们发现VIP在data1的主机上：



而data2的主机上并没有启动eth0:0。我们关闭data1主机的heartbeat服务时，我们发现eth0:0网卡绑定在data2主机上了：
此时就完成了主备机的切换。这里我们强调一下我们在DNS里面绑定的主机名必须为uname -n的显示的结果。并且在ha.cf文件中绑定主机也要该。

以上我们配置的方式为两边其相同的IP，另一种方式是我们可以为一台机器上配置两个VIP，这样可以
当我们在主heartbeat服务器上执行hb_standby脚本把本地设置为standby，即模拟heartbeat服务器宕机（和停止heartbeat的效果一样），然后看备机接管的情况。这里使用的ha_standby这个脚本，其位于：/usr/lib64/heartbeat/hb_standby
使用命令：/usr/lib64/heartbeat/hb_standby
我们还可以使用另一个脚本：hb_takeover。这个脚本让主机变成，主分发机。