RHEL6平台keepalived+lvs+iscsi+gfs搭建高可用负载均衡web集群

没啥新技术，基本上是把以前发过的帖子串了一下，练练手。

主要实现的是一个高可用负载均衡web服务器集群，适合lamp架构。

前端使用两台服务器做lvs+keepalived负载调度器，中间可以用N台做apache+php应用服务器，后面使用两台做mysql高可用双机，最后面用了一台虚拟机做文件服务器。

一共开了7个虚拟机。

系统环境：RHEL6.0 kvm虚拟机

lvs调度服务器：192.168.0.1 192.168.0.2

apache服务器:192.168.0.3 192.168.0.4

mysql服务器:192.168.0.7 192.168.0.8

文件服务器:192.168.0.10

-----------------------------------------------------------------------------------------------------------

一、lvs调度服务器

-----------------------------------------------------------------------------------------------------------

Keepalived+lvs

服务器环境:

系统均是:RHEL6.0 内核:2.6.32-71.el6.i686)

Virtual IP:192.168.0.50

Load Balancer:192.168.0.1

Backup:192.168.0.2

Real Server 1:192.168.0.3

Real Server 2:192.168.0.4

主备机上的软件包安装与配置

yum install ipvsadm kernel­-devel -y

wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz
tar zxf keepalived-1.1.20.tar.gz

cd keepalived­1.1.20

./configure ­­prefix=/usr/local/keepalived ­­with­kernel­dir=/usr/src/kernels/2.6.32-71.el6.i686/

86/

Keepalived configuration

­­­­­­­­­­­­­­­­­­­­­­Keepalived version : 1.1.17

Compiler: gcc

Compiler flags: ­g ­O2

Extra Lib: ­lpopt ­lssl ­lcrypto

Use IPVS Framework: Yes;注意编译时一定要支持 lvs

IPVS sync daemon support: Yes

Use VRRP Framework: Yes

Use LinkWatch: No

Use Debug flags: No

make

make install

mkdir /etc/keepalived

ln ­-s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/

ln ­-s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

ln ­-s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

ln -s /usr/local/keepalived/bin/* /bin/

ln -s /usr/local/keepalived/sbin/* /sbin/

vi /etc/rc.local

modprobe ip_vs

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

root@example.com#接收警报的 email 地址,可以添加多个

}

notification_email_from root@localhost

smtp_server 127.0.0.1#使用本机转发 email

smtp_connect_timeout 30

router_id LVS_DEVEL#load balancer 的标识 ID,用于 email 警报

}

vrrp_instance VI_1 {

state MASTER#备机改为 BACKUP,此状态是由 priority 的值来决定的,当前

priority 的值小于备机的值,那么将会失去 MASTER 状态

interface eth0#HA 监测网络接口

virtual_router_id 50#主、备机的 virtual_router_id 必须相同

priority 150#主机的优先级,备份机改为 50

advert_int 1#主备之间的通告间隔秒数

authentication {

auth_type PASS#主备切换时的验证

auth_pass 1111

}

virtual_ipaddress {

192.168.0.111#HA 虚拟 ip,可加多个

}

}

virtual_server 192.168.0.111 80 {

delay_loop 6#每隔 6 秒查询 realserver状态

lb_algo rr#lvs 调度算法,这里使用轮叫

lb_kind DR#lvs 负载均衡机制,这里使用直连路由

# persistence_timeout 50 #同一 IP 的连接 60 秒内被分配到同一台 realserver

protocol TCP #用 TCP 协议检查 realserver 状态

real_server 192.168.0.1 80 {

weight 1

TCP_CHECK {

connect_timeout 3 #故障重试秒数

nb_get_retry 3 #重试延迟

delay_before_retry 3

}

}

real_server 192.168.0.2 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}

}

分别在 realserver 上执行以下命令

vi /etc/sysctl.conf

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.lo.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.lo.arp_announce = 2

sysctl -p

ifconfig eth0:0 192.168.0.200 netmask 255.255.255.255 up

route add -host 192.168.0.200 dev eth0:0

vi /etc/rc.local

ifconfig eth0:0 192.168.0.200 netmask 255.255.255.255 up

route add -host 192.168.0.200 dev eth0:0

echo `hostname` > /var/www/html/index.html

service httpd start

测试:

访问 http://192.168.0.50,看到页面在两个 realserver 上切换表示成功!

你也可以通过 ipvsadm -Lnc 查看详细连接情况!

-------------------------------------------------------------------------------------------------------------------

二、mysql服务器

-------------------------------------------------------------------------------------------------------------------

mysql双主高可用

-------------------------------------

系统环境：

RHEL6.0_I386

VIP 192.168.0.51

real server1 192.167.0.7

real server2 192.168.0.8

-------------------------------------

1.server1和server2安装好mysql并修改配置文件：

yum install mysql-server

vi /etc/my.cnf:

[mysqld]

log-bin=MySQL-bin

server-id=1 ##在server2上配置为server-id=2

-------------------------------------

2.server1和server2相互设置为主从同步，（双主）.

server1:

mysql> grant replication slave on *.* to 'cluster'@'%' identified by 'cluster';

mysql> show master status;

-------------------------

MySQL-bin.000001 236

-------------------------

------------------------------------

server2:

mysql> change master to

-> master_host='192.168.0.7',

-> master_user='cluster',

-> master_password='cluster',

-> master_log_file='MySQL-bin.000001',

-> master_log_pos=236;

mysql> start slave;

mysql> show slave status;

在server2和server1上执行相反的操作，使其互为主从.

--------------------------------------

3.安装软件包

yum install gcc popt-devel kernel-devel openssl-devel ipvsadm make

tar xf keepalived-***.tar.gz

cd keepalived-*

./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-71.el6.i686

make && make install

modprobe ip_vs #此处如果系统没有自动加载此模块将导致keepalived启动后无法找到lvs负载均衡协议

mkdir /etc/keepalived/

ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/

ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

ln -s /usr/local/keepalived/bin/* /bin/

ln -s /usr/local/keepalived/sbin/* /sbin/

---------------------------------------------------

4.修改server1/server2配置文件

server1:

vi /etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {

notification_email {

root@example.com

}

notification_email_from root@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id MYSQL-HA ##确保和server2相同

}

vrrp_instance VI_1 {

state BACKUP

interface eth0

virtual_router_id 51 ##确保和server2相同，同网内不同集群此项必须不同,否则发生冲突

priority 100 ##此处server2上设置为50

advert_int 1

nopreempt ##不抢占，只在priority高的server1上设置，server2上此项注释掉

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.0.51

}

}

virtual_server 192.168.0.51 3306 {

delay_loop 2

lb_algo wrr

lb_kind DR

persistence_timeout 60

protocol TCP

real_server 192.168.0.7 3306 {

##server2上此处改为192.168.0.8(即server2本机ip)

weight 3

notify_down /usr/local/keepalived/bin/mysql.sh

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 3306

}

}

}

--------------------------------

server1和server2上都添加此检测脚本，作用是当mysql停止工作时自动关闭本机的keepalived

从而实现将故障机器踢出（因每台机器上keepalived只添加了本机为realserver）.

vi /usr/local/keepalived/bin/mysql.sh:

#!/bin/sh

pkill keepalived

--------------------------------

vi /etc/rc.local:

modprobe ip_vs ##此模块如果无法自动加载则需手动加载

--------------------------------

server1和server2启动keepalived守护进程.

/etc/init.d/keepalived start

------------------------------------------------------------------------------------------------------------

三、文件服务器

------------------------------------------------------------------------------------------------------------

data server:192.168.0.10

data client1:192.168.0.3

data client2:192.168.0.4

-----------------------------

data server:

yum install luci -y

/etc/init.d/luci start

访问https://192.168.0.10/:8084/ 使用系统帐号密码登录进入创建一个集群并将client1/2加为节点

然后添加一个虚拟fence设备并加入client1 client2

yum install scsi-target-utils -y

chkconfig tgtd on

/etc/init.d/tgtd start

tgtadm --lld iscsi --op new --mode target --tid 1 -T webdata

tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda

tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

tgtadm --lld iscsi --op show --mode target #验证是否ok

vi /etc/rc.local 将上面的配置命令写入rc.local

tgtadm --lld iscsi --op new --mode target --tid 1 -T webdata

tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda

tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

data client:

yum install ricci -y

/etc/init.d/ricci start

lvmconf --enable-cluster

yum install iscsi-initiator-utils -y

iscsiadm -m discovery -t sendtargets -p 192.168.0.10

iscsiadm -m node -T webdata -p 192.168.0.10 -l

信息自动保存至配置文件/var/lib/iscsi/nodes/webdata/192.168.0.10,3260,1/default

fdisk -l 查看可看到多了一块/dev/sda硬盘

pvcreate /dev/sda

vgcreate datavg /dev/sda

lvcreate -L 1020M -n lv1 datavg

cman_tool status|grep Name #查看cluster name

Cluster Name: web_cluster

创建gfs文件系统

mkfs.gfs2 -p lock_dlm -t web_cluster:gfs -j 2 /dev/datavg/lv1

#此处-j 2 提供两台client主机连接

mount -t gfs2 /dev/datavg/lv1 /mnt

vi /etc/fstab

/dev/datavg/lv1 /mnt gfs2 defaults 0 0

/etc/init.d/gfs2 start #文件系统自动挂在到/mnt

chkconfig cman on

chkconfig rgmanager on

chkconfig ricci on

chkconfig modclusterd on

chkconfig clvmd on

chkconfig gfs2 on

如果gfs开机时无法识别到而lvscan命令又遇到lv状态为inactive：

lvchange -ay /dev/gfsvg/gfs

-------------------------------------------------------------------------------------

就这样就搞定了。。。。