利用LVS+Keepalived DR 实现高性能高可用负载均衡

最近公司用做负责均衡，研究了一下lvs，做为我的第一篇博客吧。

一．            LVS+Keepalived 介绍

1.      LVS

LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；

八种调度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。

 

2.      Keepalvied

       Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现

 

二. 网站负载均衡拓朴图



三. 安装LVS和Keepalvied软件包

 

     #lsmod | rep ip_vs
#uname -r
显示2.6.18-53.el5PAE
#ln -s /usr/src/kernels/2.6.18-53.el5PAE-i686/ /usr/src/linux

#tar zxvf ipvsadm-1.24.tar.gz
#cd ipvsadm-1.24
#make all && make install(安装相关包：gcc*)
#find / -name ipvsadm # 查看ipvsadm的位置

#tar zxvf keepalived-1.1.15.tar.gz
#cd keepalived-1.1.15
#./configure (提示安装openssl*)
# make && make install
#find / -name keepalived # 查看keepalived位置

#cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
#cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
#mkdir /etc/keepalived
#cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
#cp /usr/local/sbin/keepalived /usr/sbin/
#chkconfig —add keepalived
#chkconfig keepalived on


 

四. 配置LVS

在LVS机上配置（两台都要配置）

   ＃vim /usr/local/sbin/lvsdr.sh
#!/bin/bash
# description: start LVS of DirectorServer
#Written by :NetSeek
GW=192.168.5.2

# website director vip.
WEB_VIP=192.168.6.200
WEB_RIP1=192.168.6.152
WEB_RIP2=192.168.6.159
. /etc/rc.d/init.d/functions

logger $0 called with $1

case "$1" in
start)
# Clear all iptables rules.
#/sbin/iptables -F
# Reset iptables counters.
#/sbin/iptables -Z
# Clear all ipvsadm rules/services.
/sbin/ipvsadm -C
#set lvs vip for dr
/sbin/ipvsadm --set 30 5 60
#/sbin/ifconfig eth0:0 $WEB_VIP broadcast $WEB_VIP netmask 255.255.255.255 up
/sbin/route add -host $WEB_VIP dev eth0
/sbin/ipvsadm -A -t $WEB_VIP:1758 -s wrr -p 3
/sbin/ipvsadm -a -t $WEB_VIP:1758 -r $WEB_RIP1:1758 -g -w 1
/sbin/ipvsadm -a -t $WEB_VIP:1758 -r $WEB_RIP2:1758 -g -w 1
touch /var/lock/subsys/ipvsadm >/dev/null 2>&1
# set Arp
/sbin/arping -I eth0 -c 5 -s $WEB_VIP $GW >/dev/null 2>&1 #备份节点上把这行注释掉
;;
stop)
/sbin/ipvsadm -C
/sbin/ipvsadm -Z
#ifconfig eth0:0 down
route del $WEB_VIP >/dev/null 2>&1
rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1
/sbin/arping -I eth0 -c 5 -s $WEB_VIP $GW #备份节点上把这行注释掉
echo "ipvsadm stoped"
;;
status)
if [ ! -e /var/lock/subsys/ipvsadm ];then
echo "ipvsadm is stoped"
exit 1
else
ipvsadm -ln
echo "..........ipvsadm is OK."
fi
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0


五. 利用Keepalvied实现负载均衡和和高可用性

1．主LVS上：修改/etc/keepalived/keepalived.conf

global_defs {
router_id LVS_DEVEL
}
vrrp_sync_group lvs_1 {
group {
VI_1
}
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 50
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.6.200
}
}

virtual_server 192.168.6.200 1758 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.6.152 1758 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 1758
}
}
real_server 192.168.6.159 1758 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 1758
}
}
}

2．从LVS：

global_defs {
router_id LVS_DEVEL
}
vrrp_sync_group lvs_1 {
group {
VI_1
}
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 50
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.6.200
}

virtual_server 192.168.6.200 1758 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.6.152 1758 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 1758
}
}
real_server 192.168.6.159 1758 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 1758
}
}
}

 

3.开启IP转发，可以不用,如果是nat方式就要配置了，用sysctl命令配置

net.ipv4.ip_forward=1

六、配置xcloud（我们项目的名字）端（192.168.6.152和192.168.6.159）

1.      配置IP转发功能

＃vim /etc/rc.d/init.d/realserver.sh

#!/bin/bash
# description: Config realserver lo and apply noarp

SNS_VIP=192.168.6.200
/etc/rc.d/init.d/functions

case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0

 

七、启动服务

＃/usr/local/sbin/lvsdr.sh start （把这条语句写到/etc/rc.local中，开机启动）

＃/etc/init.d/keepalivedstart  启动keepalived 服务，keepalived就能利用keepalived.conf 配置文件，实现负载均衡和高可用.

真是服务器上：运行 /etc/rc.d/init.d/realserver.shstart 也可以加入/etc/rc.local里面

八、测试

<
9fc0
p>在主lvs上：注意红色的ip
1.      ＃ ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu16436 qdisc noqueue state UNKNOWN

   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

   inet 127.0.0.1/8 scope host lo

   inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

   link/ether e4:1f:13:ec:d2:aa brd ff:ff:ff:ff:ff:ff

   inet 192.168.6.175/16 brd 192.168.255.255 scope global eth0

   inet 192.168.6.200/32 scope global eth0

   inet6 fe80::e61f:13ff:feec:d2aa/64 scope link

      valid_lft forever preferred_lft forever

3: eth1:<NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen1000

link/ethere4:1f:13:ec:d2:ab brd ff:ff:ff:ff:ff:ff

    ………

最后把主机宕机(通过在主机上用/etc/init.d/keepalived stop也可以)，到从机上使用ip addr show 查看，同上则OK。

2.用客户端连接一下xcloudd（192.168.6.200），如果连接成功，通过下面命令可以看到，真实连接到了哪台服务器。

  #watch ipvsadm –ln

IP Virtual Server version 1.2.1 (size=4096)

ProtLocalAddress:Port Scheduler Flags

 -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

TCP  192.168.6.200:1758 wrr persistent 3

 -> 192.168.6.152:1758          Route   1      0          0

 -> 192.168.6.159:1758           Route  1      1          0

 

 #tail –f /var/log/message  监听日志，查看状态。

 

3．将一台xcloudd服务器关闭，然后在LVS 上用ipvsadm 命令查看，关闭的服务器应该从

lvs集群中剔除了，再将关闭的服务器启动起来，用ipvsadm查看，又回来了。