haproxy+keepalived+nginx+php-fpm

手头有三台机器，领导要求做成高可用的网站，一台做数据库肯定是没错的，剩下两台机器只能做web了。

原本打算用nginx+keepalived，keepalived做高可用没问题，一主一从，但是从机完全standby，资源有些浪费。于是想到了负载均衡集群。考虑到nginx又做负载均衡，又做web，怕管理上麻烦，负载均衡就用了haproxy做，并且haproxy的健康检查非常到位。而nginx的健康检查实际上是假的，应该较故障转移，因为一旦timeout过期了，流量仍然会分过去，发现不通后再进行故障转移，这会导致网页间歇性加载缓慢。

本文做法的好处是，从服务器平时也能分担主服务器的流量，任何一个机器宕机，web服务都可以正常访问。

环境：CentOS6.4 x86_64

VIP 192.168.122.10

web01 192.168.122.11

web02 192.168.122.12

1、安装软件

yum install haproxy keepalived nginx php-fpm php-gd php-mysql php-xml php-cli -y
chkconfig keepalived on
chkconfig haproxy on
chkconfig nginx on
chkconfig php-fpm on
chkconfig iptables off #防火墙暂时不配置

2、系统和内核设置

sed -i ‘s/enforcing/disabled/g' /etc/sysconfig/selinux

sysctl -w "net.ipv4.ip_nonlocal_bind = 1"
echo "net.ipv4.ip_nonlocal_bind = 1" >>/etc/sysctl.conf

上面这个内核参数，可以允许服务器监听在一个不存在的地址上。

3、配置keepalived（主从配置略有不同）

配置主服务器web01

global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id web01
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
preempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.122.10 label eth0:1
}
}

配置从服务器

global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived02@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id web02
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
#preempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.122.10 label eth0:1
}
}

4、配置haproxy（主从服务器配置完全相同，监听地址192.168.122.10：80）

global
log         127.0.0.1 local2
chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

defaults
mode                    http
log                     global
option                  httplog
option                  dontlognull
option http-server-close
option forwardfor       except 127.0.0.0/8
option                  redispatch
retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          1m
timeout server          1m
timeout http-keep-alive 10s
timeout check           10s
maxconn                 3000

listen status
bind *:10086
stats uri /haproxy-status
stats auth admin：123456
stats hide-version

frontend  haproxy-nlb
bind    192.168.122.10:80
default_backend             nginx-web

backend nginx-web
option  httpchk HEAD /check.txt HTTP/1.0
balance roundrobin
server  web01 192.168.122.11:80 weight 3 check inter 5s rise 2 fall 3
server  web02 192.168.122.12:80 weight 3 check inter 5s rise 2 fall 3

5、配置nginx（除了监听地址不同，其余全一样）

主服务器监听192.168.122.11：80

从服务器监听192.168.122.12：80

user  nginx;
worker_processes  2;
worker_rlimit_nofile 65535;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

google_perftools_profiles /tmp/tcmalloc;

events {
use epoll;
worker_connections  2048;
}

http {
include       /etc/nginx/mime.types;
include       /etc/nginx/naxsi_core.rules;
default_type  application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

#access_log  /var/log/nginx/access.log  main;

sendfile        on;
server_tokens   off;

#tcp_nopush     on;

keepalive_timeout  65;

gzip  on;
gzip_static on;
gzip_disable "msie6";
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x$
gzip_buffers 16 8k;

client_max_body_size 20m;
client_body_buffer_size 128k;
server {
listen       192.168.122.11:8080;
server_name  localhost;
root        /usr/share/nginx/html;
index       index.html index.htm index.php;

#charset koi8-r;
access_log  /var/log/nginx/host.access.log  main;

location / {
include /etc/nginx/naxsi_conf ;

if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;
}
}

#error_page  404              /404.html;

# redirect server error pages to the static page /50x.html
#
error_page   500 502 503 504  /50x.html;

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
#try_files $uri = 404;
fastcgi_pass   127.0.0.1:9000;
#fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index  index.php;
fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_read_timeout 240;
include        fastcgi_params;
}

if ($fastcgi_script_name ~ \..*\/.*php) {
return 403;
}

# deny access to hiden file . (filename begin with ".")
location ~ /\. {
access_log off;
log_not_found off;
deny all;
}

# deny access to bakup file .(any filename end with "~" )
location ~ ~$ {
access_log off;
log_not_found off;
deny all;
}

# cache image file
location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml|swf)$ {
expires           1d;
}

# don't log robots  and favion
location = /robots.txt { access_log off; log_not_found off; }
location = /favicon.ico { access_log off; log_not_found off; }

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
location ~ /\.ht {
deny  all;
}
}
}

6、配置php-fpm

sed -i ‘s/apache/nginx/g’ /etc/php-fpm.d/www.conf

7、其他

给web01加上检测页面

echo web01 >/usr/share/nginx/html/check.txt

给web02加上检测页面

echo web02 >/usr/share/nginx/html/check.txt

8、重启两台机器

9、验证

由于是轮询，我们很容易检测负载均衡（最好不要用浏览器，浏览器有缓存）

$ for n in {1..10};do curl http://192.168.122.10/check.txt;done web02
web01
web02
web01
web02
web01
web02
web01
web02
web01

OK，大功告成！