SaltStack一键自动化部署高可用负载均衡集群

本节内容涉及的saltstack配置以及各服务的安装包和配置文件均打包上传到了百度云，可自由下载使用

实验环境（rhel6.5 x86_64bit virtual machine）

172.25.5.91 salt-master rhel65-lockey1
172.25.5.92 salt-minion rhel65-lockey2 （将自动部署keepalived，haproxy）
172.25.5.93 salt-minion rhel65-lockey3 （将自动部署keepalived，haproxy）
172.25.5.94 salt-minion rhel65-lockey4 （将自动部署nginx）
172.25.5.95 salt-minion rhel65-lockey5 （将自动部署nginx）

172.25.5.100 physical machine(for testing)

一、准备工作

1. 做好各主机的解析

vim /etc/hosts

172.25.5.91  rhel65-lockey1
172.25.5.92  rhel65-lockey2
172.25.5.93  rhel65-lockey3
172.25.5.94  rhel65-lockey4
172.25.5.95  rhel65-lockey5

2. 各主机需要关闭selinux以及防火墙设置

3. 在master端执行salt-key -L发现所有minion，并且通过salt-key -A命令执行认证，得到下面结果：

[root@rhel65-lockey1 ~]# salt-key -L



4. master端取消以下注释并重启服务生效：

file_roots:
base:
- /srv/salt

pillar_roots:
base:
- /srv/pillar

二、此高可用负载均衡集群组成

高可用：keepalived

负载均衡：haproxy

后端web server：NGINX

三、Get start

salt自动化部署集群目录结构

3.1 pillar目录

为了结合jinja模板实现一些配置文件中参数的动态调整（如keepalived中MASTER与BACKUP以及优先级的值）

[root@rhel65-lockey1 pillar]# tree

.
├── top.sls
└── webservice
└── web.sls

1 directory, 2 files

[root@rhel65-lockey1 pillar]# cat webservice/web.sls

{% if grains['host'] == 'rhel65-lockey2' %}
state: MASTER
priority: 100
{% elif grains['host'] == 'rhel65-lockey3' %}
state: BACKUP
priority: 50
{% endif %}

[root@rhel65-lockey1 pillar]# cat top.sls

base:
'rhel65-lockey2':
- webservice.web
'rhel65-lockey3':
- webservice.web

运行命令 salt ‘*’ saltutil.refresh_pillar使得选定主机动态参数设置生效

[root@rhel65-lockey1 pillar]# salt ‘*’ pillar.items#查看结果

3.2 _grains目录

为服务主机添加role设置，用以根据role标签决定主机需要安装的软件

[root@lrhel65-lockey1 _grains]# pwd

/srv/salt/_grains

[root@rhel65-lockey1 _grains]# cat my_grains.py

#!/usr/bin/env python

def my_grains():
grains =  {}
grains['roles'] = 'nginx'
grains['name'] = 'lockey'
return grains

[root@rhel65-lockey1 salt]# salt rhel65-lockey[5,4] saltutil.sync_grains

#为2和4号主机添加nginx的role，后面部署的时候将通过这个标签匹配进行nginx的安装

rhel65-lockey4:
- grains.my_grains
rhel65-lockey5:
- grains.my_grains

[root@rhel65-lockey1 salt]# salt rhel65-lockey[5,4] grains.item roles#查看结果

rhel65-lockey4:
----------
roles:
nginx
rhel65-lockey5:
----------
roles:
nginx

3.3 haproxy目录:

haproxy安装、服务配置文件、启动脚本和安装包

[root@lockey6 haproxy]# cat install.sls

include:
- pkg.nginx-pre
- user.haproxy

haproxy-install:
file.managed:
- name: /mnt/haproxy-1.6.11.tar.gz
- source: salt://haproxy/files/haproxy-1.6.11.tar.gz

cmd.run:
- name: cd /mnt && tar zxf haproxy-1.6.11.tar.gz && cd haproxy-1.6.11 && make TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 PREFIX=/usr/local/haproxy && make TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 PREFIX=/usr/local/haproxy install
- creates: /usr/local/haproxy

/etc/haproxy:
file.directory:
- mode: 755

/etc/haproxy/haproxy.cfg:
file.managed:
- source: salt://haproxy/files/haproxy.cfg

/etc/init.d/haproxy:
file.managed:
- source: salt://haproxy/files/haproxy
- mode: 755

[root@lockey6 haproxy]# cat service.sls

include:
- haproxy.install

haproxy-service:
service.running:
- name: haproxy
- enable: true
- reload: true
- watch:
- file: /etc/haproxy/haproxy.cfg

3.4 keepalived目录

keepalived安装、服务配置文件、启动脚本和安装包

[root@lockey6 keepalived]# cat install.sls

include:
- pkg.nginx-pre

keepalived-install:
file.managed:
- name: /mnt/keepalived-1.3.5.tar.gz
- source: salt://keepalived/files/keepalived-1.3.5.tar.gz

cmd.run:
- name: cd /mnt && tar zxf keepalived-1.3.5.tar.gz && cd keepalived-1.3.5 && ./configure --prefix=/usr/local/keepalived --with-init=SYSV && make && make install
- creates: /usr/local/keepalived

/etc/sysconfig/keepalived:
file.managed:
- source: salt://keepalived/files/keepalived-sys
- mode: 644
- user: root

/etc/keepalived:
file.directory:
- mode: 755

/sbin/keepalived:
file.symlink:
- target: /usr/local/keepalived/sbin/keepalived

/etc/keepalived/keepalived.conf:
file.managed:
- source: salt://keepalived/files/keepalived.conf
- mode: 644
- user: root
- group: root
- template: jinja###注意jinja模板的用法
- context:
state: {{ pillar['state'] }}
priority: {{ pillar['priority'] }}

/etc/init.d/keepalived:
file.managed:
- source: salt://keepalived/files/keepalived
- mode: 755
- require:
- file: /etc/sysconfig/keepalived
- file: /etc/keepalived/keepalived.conf

[root@lockey6 keepalived]# cat service.sls

include:
- keepalived.install

keepalived-service:
service.running:
- name: keepalived
- enable: true
- reload: true
- watch:
- file: /etc/keepalived/keepalived.conf

3.5 nginx目录

nginx安装以及服务配置文件和启动脚本、安装包

[root@lockey6 nginx]# cat install.sls

include:
- pkg.nginx-pre

nginx-source-install:
file.managed:
- name: /mnt/nginx-1.12.1.tar.gz
- source: salt://nginx/files/nginx-1.12.1.tar.gz

cmd.run:
- name: cd /mnt && tar zxf nginx-1.12.1.tar.gz && cd nginx-1.12.1 && sed -i.bak 's/#define NGINX_VER          "nginx\/" NGINX_VERSION/#define NGINX_VER          "nginx"/g' src/core/nginx.h && sed -i.bak 's/CFLAGS="$CFLAGS -g"/#CFLAGS="$CFLAGS -g"/g' auto/cc/gcc && ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module && make && make install
- creates: /usr/local/nginx

[root@lockey6 nginx]# cat service.sls

include:
- nginx.install
- user.nginx

/usr/local/nginx/conf/nginx.conf:
file.managed:
- source: salt://nginx/files/nginx.conf
- mode: 644

/etc/init.d/nginx:
file.managed:
- source: salt://nginx/files/nginx
- mode: 755

nginx-service:
service.running:
- name: nginx
- enable: true
- reload: true
- require:
- file: /etc/init.d/nginx
- watch:
- file: /usr/local/nginx/conf/nginx.conf

3.6 pkg目录

为各软件安装解决依赖包问题，需要包含在最先执行的安装文件中

[root@lockey6 pkg]# cat nginx-pre.sls

pkg-init:
pkg.installed:
- pkgs:
- gcc
- gcc-c++
- zlib-devel
- openssl-devel
- pcre-devel

3.7 top.sls

顶级文件，用来进行集群的分节点软件安装

[root@lockey6 salt]# cat top.sls

base:
'rhel65-lockey2':
- keepalived.service
- haproxy.service
'rhel65-lockey3':
- keepalived.service
- haproxy.service
'roles:nginx':
- match: grain
- nginx.service

3.8 user目录

创建部分服务运行的用户和组（如haproxy，nginx）

[root@lockey6 salt]# cat user/nginx.sls

nginx:
user.present:
- uid: 800
- shell: /sbin/nologin
- home: /usr/local/nginx
- createhome: false

[root@lockey6 salt]# cat user/haproxy.sls

haproxy:
group.present:
- gid: 200
user.present:
- uid: 200
- gid: 200
- shell: /sbin/nologin
- home: /usr/local/haproxy
- createhome: false

各服务的配置文件以及启动脚本就不贴出了，下面提出几点需要注意的地方：

1.jinja模板调用的格式以及对应的文件中的写法

- template: jinja#此部分对应keepalived.conf
- context:
state: {{ pillar['state'] }}
priority: {{ pillar['priority'] }}

keepalived.conf 中调用时写法如下

state {{ state }}
priority {{ priority }}

2.对于pillar和grains文件写成后，必须要对特定的主机进行标签的刷新，保证最后运行时能够对应进行软件安装：

salt rhel65-lockey[4,5] saltutil.sync_grains
salt ‘*’ saltutil.refresh_pillar

3. 对于haproxy的配置文件，务必指定后端主机以及监听：

frontend public
bind          www.lockey.com *:80
default_backend dynamic

backend dynamic
balance         roundrobin
server          web1 172.25.5.94:80 cookie s1 check inter 1000
server          web2 172.25.5.95:80 cookie s2 check inter 1000

4.top文件的书写

[root@lockey6 salt]# cat top.sls

base:
'rhel65-lockey2':#2和3主机安装keepalived和haproxy
- keepalived.service
- haproxy.service
'rhel65-lockey3':
- keepalived.service
- haproxy.service
'roles:nginx':#具有nginx标签的主机（4和5）安装nginx服务
- match: grain
- nginx.service

5. 确保后端主机的web服务端口80未被占用，以及selinux和防火墙

四、进行整体自动安装测试然后执行

4.1 首先测试一下，如果有问题解决后再执行安装

[root@rhel65-lockey1 salt]# salt ‘*’ state.highstate test=true
注意：如果提示keepalived/haproxy/nginx服务无法启动的错误是正常的，因为启动脚本还未推送过去

4.2 测试顺利则执行推送开始自动安装：

[root@rhel65-lockey1 salt]# salt ‘*’ state.highstate

五、集群可用性测试

1. 查看vip是否在keepalived定义的MASTER上



停掉master的keepalived服务之后发生ip漂移



3.访问http://vip/admin/stats查看各主机的状态

因为集群部署成功之后是通过VIP(172.25.5.99)提供服务的，我在物理机中加了一条解析记录然后通过curl命令进行负载均衡测试：

vim /etc/hosts

172.25.5.99 www.lockey.com

成功结果如下：