您的位置:首页 > 运维架构 > 网站架构

[ Openstack ] Openstack-Mitaka 高可用之 Pacemaker+corosync+pcs 高可用集群

2017-12-16 17:46 423 查看

目录

Openstack-Mitaka 高可用之 概述
Openstack-Mitaka 高可用之 环境初始化
Openstack-Mitaka 高可用之 Mariadb-Galera集群部署
Openstack-Mitaka 高可用之 Rabbitmq-server 集群部署
Openstack-Mitaka 高可用之 memcache
Openstack-Mitaka 高可用之 Pacemaker+corosync+pcs高可用集群
Openstack-Mitaka 高可用之 认证服务(keystone)
OpenStack-Mitaka 高可用之 镜像服务(glance)
Openstack-Mitaka 高可用之 计算服务(Nova)
Openstack-Mitaka 高可用之 网络服务(Neutron)
Openstack-Mitaka 高可用之 Dashboard
Openstack-Mitaka 高可用之 启动一个实例
Openstack-Mitaka 高可用之 测试

介绍及特点

Pacemaker:工作在资源分配层,提供资源管理器的功能
Corosync:提供集群的信息层功能,传递心跳信息和集群事务信息
Pacemaker + Corosync 就可以实现高可用集群架构

集群搭建

以下三个节点都需要执行:

# yum install pcs -y
# systemctl start  pcsd ; systemctl enable pcsd
# echo 'hacluster' | passwd --stdin hacluster
# yum install haproxy  rsyslog -y
# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf        # 启动服务的时候,允许忽视VIP的存在
# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf        # 开启内核转发功能
# sysctl -p


在任意节点创建用于haproxy监控Mariadb的用户

MariaDB [(none)]> CREATE USER 'haproxy'@'%' ;


配置haproxy用于负载均衡器

[root@controller1 ~]# egrep -v "^#|^$" /etc/haproxy/haproxy.cfg
log         127.0.0.1 local2
chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
defaults
mode                    http
log                     global
option                  httplog
option                  dontlognull
option http-server-close
option forwardfor       except 127.0.0.0/8
option                  redispatch
retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          1m
timeout server          1m
timeout http-keep-alive 10s
timeout check           10s
maxconn                 4000
listen galera_cluster
mode tcp
bind 192.168.0.10:3306
balance source
option mysql-check user haproxy
server controller1 192.168.0.11:3306 check inter 2000 rise 3 fall 3 backup
server controller2 192.168.0.12:3306 check inter 2000 rise 3 fall 3
server controller3 192.168.0.13:3306 check inter 2000 rise 3 fall 3 backup

listen memcache_cluster
mode tcp
bind 192.168.0.10:11211
balance source
option tcplog
server controller1 192.168.0.11:11211 check inter 2000 rise 3 fall 3
server controller2 192.168.0.12:11211 check inter 2000 rise 3 fall 3
server controller3 192.168.0.13:11211 check inter 2000 rise 3 fall 3


注意:

(1)确保haproxy配置无误,建议首先修改ip和端口启动测试是否成功。
(2)Mariadb-Galera和rabbitmq默认监听到 0.0.0.0 修改调整监听到本地 192.168.0.x
(3)将haproxy正确的配置拷贝到其他节点,无需手动启动haproxy服务

为haproxy配置日志(所有controller节点执行):

# vim /etc/rsyslog.conf
…
$ModLoad imudp
$UDPServerRun 514
…
local2.*                                                /var/log/haproxy/haproxy.log
…

# mkdir -pv /var/log/haproxy/
mkdir: created directory ‘/var/log/haproxy/’

# systemctl restart rsyslog


启动haproxy进行验证操作:

# systemctl start haproxy
[root@controller1 ~]# netstat -ntplu | grep ha
tcp        0      0 192.168.0.10:3306       0.0.0.0:*               LISTEN      15467/haproxy
tcp        0      0 192.168.0.10:11211      0.0.0.0:*               LISTEN      15467/haproxy
udp        0      0 0.0.0.0:43268           0.0.0.0:*                           15466/haproxy

验证成功,关闭haproxy
# systemctl stop haproxy


在controller1节点上执行:

[root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p hacluster --force
controller3: Authorized
controller2: Authorized
controller1: Authorized


创建集群:

[root@controller1 ~]# pcs cluster setup --name openstack-cluster controller1 controller2 controller3  --force
Destroying cluster on nodes: controller1, controller2, controller3...
controller3: Stopping Cluster (pacemaker)...
controller2: Stopping Cluster (pacemaker)...
controller1: Stopping Cluster (pacemaker)...
controller3: Successfully destroyed cluster
controller1: Successfully destroyed cluster
controller2: Successfully destroyed cluster

Sending 'pacemaker_remote authkey' to 'controller1', 'controller2', 'controller3'
controller3: successful distribution of the file 'pacemaker_remote authkey'
controller1: successful distribution of the file 'pacemaker_remote authkey'
controller2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
controller1: Succeeded
controller2: Succeeded
controller3: Succeeded

Synchronizing pcsd certificates on nodes controller1, controller2, controller3...
controller3: Success
controller2: Success
controller1: Success
Restarting pcsd on the nodes in order to reload the certificates...
controller3: Success
controller2: Success
controller1: Success


启动集群的所有节点:

[root@controller1 ~]# pcs cluster start --all
controller2: Starting Cluster...
controller1: Starting Cluster...
controller3: Starting Cluster...
[root@controller1 ~]# pcs cluster enable --all
controller1: Cluster Enabled
controller2: Cluster Enabled
controller3: Cluster Enabled


查看集群信息:

[root@controller1 ~]# pcs status
Cluster name: openstack-cluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:43 2017
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3

3 nodes configured
0 resources configured

Online: [ controller1 controller2 controller3 ]

No resources

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@controller1 ~]# pcs cluster status
Cluster Status:
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 19:30:52 2017
Last change: Thu Nov 30 19:30:17 2017 by hacluster via crmd on controller3
3 nodes configured
0 resources configured

PCSD Status:
controller2: Online
controller3: Online
controller1: Online


三个节点都在线
默认的表决规则建议集群中的节点个数为奇数且不低于3。当集群只有2个节点,其中1个节点崩坏,由于不符合默认的表决规则,集群资源不发生转移,集群整体仍不可用。no-quorum-policy="ignore"可以解决此双节点的问题,但不要用于生产环境。换句话说,生产环境还是至少要3节点。
pe-warn-series-max、pe-input-series-max、pe-error-series-max代表日志深度。
cluster-recheck-interval是节点重新检查的频率。

[root@controller1 ~]#  pcs property set pe-warn-series-max=1000 pe-input-series-max=1000 pe-error-series-max=1000 cluster-recheck-interval=5min


禁用stonith:
stonith是一种能够接受指令断电的物理设备,环境无此设备,如果不关闭该选项,执行pcs命令总是含其报错信息。

[root@controller1 ~]# pcs property set stonith-enabled=false


二个节点时,忽略节点quorum功能:

[root@controller1 ~]# pcs property set no-quorum-policy=ignore


验证集群配置信息

[root@controller1 ~]# crm_verify -L -V


为集群配置虚拟 ip

[root@controller1 ~]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 \
ip="192.168.0.10" cidr_netmask=32 nic=eno16777736 op monitor interval=30s


到此,Pacemaker+corosync 是为 haproxy服务的,添加haproxy资源到pacemaker集群

[root@controller1 ~]# pcs resource create lb-haproxy systemd:haproxy --clone


说明:创建克隆资源,克隆的资源会在全部节点启动。这里haproxy会在三个节点自动启动。

查看Pacemaker资源情况

[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1        # 心跳的资源绑定在第三个节点的
Clone Set: lb-haproxy-clone [lb-haproxy]        # haproxy克隆资源
Started: [ controller1 controller2 controller3 ]


注意:这里一定要进行资源绑定,否则每个节点都会启动haproxy,造成访问混乱
将这两个资源绑定到同一个节点上

[root@controller1 ~]# pcs constraint colocation add lb-haproxy-clone ClusterIP INFINITY


绑定成功

[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller3
Clone Set: lb-haproxy-clone [lb-haproxy]
Started: [ controller1]
Stopped: [ controller2 controller3 ]


配置资源的启动顺序,先启动vip,然后haproxy再启动,因为haproxy是监听到vip

[root@controller1 ~]# pcs constraint order ClusterIP then lb-haproxy-clone


手动指定资源到某个默认节点,因为两个资源绑定关系,移动一个资源,另一个资源自动转移。

[root@controller1 ~]# pcs constraint location ClusterIP prefers controller1
[root@controller1 ~]# pcs resource
ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller1
Clone Set: lb-haproxy-clone [lb-haproxy]
Started: [ controller1 ]
Stopped: [ controller2 controller3 ]
[root@controller1 ~]# pcs resource defaults resource-stickiness=100        # 设置资源粘性,防止自动切回造成集群不稳定
现在vip已经绑定到controller1节点
[root@controller1 ~]# ip a | grep global
inet 192.168.0.11/24 brd 192.168.0.255 scope global eno16777736
inet 192.168.0.10/32 brd 192.168.0.255 scope global eno16777736
inet 192.168.118.11/24 brd 192.168.118.255 scope global eno33554992


尝试通过vip连接数据库

Controller1:

[root@controller1 haproxy]# mysql -ugalera -pgalera -h 192.168.0.10



Controller2:



 

高可用配置成功。

测试高可用是否正常

在controller1节点上直接执行 poweroff -f

[root@controller1 ~]# poweroff -f


vip很快就转移到controller2节点上



再次尝试访问数据库



无任何问题,测试成功。

查看集群信息:

[root@controller2 ~]# pcs status
Cluster name: openstack-cluster
Stack: corosync
Current DC: controller3 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Thu Nov 30 23:57:28 2017
Last change: Thu Nov 30 23:54:11 2017 by root via crm_attribute on controller1

3 nodes configured
4 resources configured

Online: [ controller2 controller3 ]
OFFLINE: [ controller1 ]            # controller1 已经下线

Full list of resources:

ClusterIP    (ocf::heartbeat:IPaddr2):    Started controller2
Clone Set: lb-haproxy-clone [lb-haproxy]
Started: [ controller2 ]
Stopped: [ controller1 controller3 ]

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: