LAMP+haproxy+varnish实现网站访问的动静分离及静态资源缓存

实验目标：
1. LAMP节点提供用户动态请求访问，数据库单独有数据库节点提供；
2. LAMP动态网站有两台服务器，提供负载均衡；
3. 静态网站服务器节点提供用户的静态资源请求访问；存在两台静态web服务器，其网站静态资源在静态服务器上存放；
4. 用户的静态请求访问后缓存在varnish服务器上，实现访问加速
5. 前端的haproxy提供反向代理功能，将用户的动态资源请求发送给后端LAMP节点，静态资源请求发往后端静态web服务器；
6. 该架构考虑还不健全，如静态内容的一致性，数据库的单点故障，只是一个不成熟的架构实现简单的动静分离及缓存服务器实现；

实现过程：

一. LAMP构建
================ ha.stu31.com : 172.16.31.10 ================
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.31.10 ha.stu31.com ha
172.16.31.11 node1.stu31.com node1
172.16.31.12 node2.stu31.com node2
172.16.31.13 node3.stu31.com node3
172.16.31.14 node4.stu31.com node4
172.16.31.15 mysql.stu31.com mysql
172.16.31.16 node5.stu31.com node5
172.16.31.17 node6.stu31.com node6
=======
ssh-keygen -t rsa -P "" -f "/root/.ssh/id_rsa"
for i in {1..6}; do ssh-copy-id -i .ssh/id_rsa.pub node$i; done
for i in {1..6}; do scp /etc/hosts node$i:/etc/hosts; done
for i in {1..6}; do ssh node$i "ntpdate -u 202.112.10.60"; done
for i in {1..6}; do ssh node$i date; done
for i in {1..6}; do ssh node$i "mount /dev/cdrom /mnt/cdrom"; done
for i in {1..2}; do ssh node$i "yum -y install httpd php mysql php-mysql"; done
for i in {1..2}; do ssh node$i "service httpd start"; done
vim index.php
<?php
phpinfo();
?>
///////////
for i in {1..2}; do scp index.php root@node$i:/var/www/html; done

====================== windows ======================
浏览器访问 http://172.16.31.11/index.php 浏览器访问 http://172.16.31.12/index.php
============== mysql.stu31.com : 172.16.31.15 ================
mount /dev/cdrom /mnt/cdrom/
yum -y install mysql-server mysql
service mysqld start
mysql
mysql> create schema bbsdb;
mysql> grant all on bbsdb.* to 'bbsadmin'@'172.16.%.%'identified by 'oracle';
mysql> flush privileges;
mysql> quit

============== node1.stu31.com : 172.16.31.11 ================
wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
cd /var/www/html/upload
chmod -R go+w config/ data uc_*

====================== windows ======================
浏览器访问 http://172.16.31.11/upload/install 1. 检查安装环境：
2. 设置运行环境：点击选择全新安装
3. 安装数据库
填写数据库信息
数据库服务器: 172.16.31.15
数据库名: bbsdb
数据库用户名: bbsadmin
数据库密码: oracle
数据表前缀: pre_
系统邮箱: admin@stu31.com
填写管理员信息
管理员账号: admin
管理员密码: admin
重复密码: admin
管理员Email: admin@stu31.com
设置好数据库后点击安装即可

============== node2.stu31.com : 172.16.31.12 ===============

1. cd /var/www/html; scp -r upload/ readme/ utility/ node2:/var/www/html/
2. 重新解压软件包到node2，重新安装一遍，删除数据库重新安装(因为数据库中无重要内容)。

访问测试
====================== windows ====================== http://172.16.31.11/upload/forum.php http://172.16.31.12/upload/forum.php
能正常访问证明LAMP平台构建成功

二. 静态web服务器构建

============== node5.stu31.com : 172.16.31.16 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip) unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start

============== node6.stu31.com : 172.16.31.17 ===============
scp node1:/root/Discuz_X3.2_SC_UTF8.zip ./
(wget http://download.comsenz.com/DiscuzX/3.2/Discuz_X3.2_SC_UTF8.zip) unzip -d /var/www/html/ Discuz_X3.2_SC_UTF8.zip
service httpd start

访问测试
====================== windows ====================== http://172.16.31.16/upload/forum.php http://172.16.31.17/upload/forum.php
只能看到静态界面

三. Varnish缓存服务器构建

============== node3.stu31.com : 172.16.31.13 ===============

wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-3.0.6-1.el6.x86_64.rpm wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-docs-3.0.6-1.el6.x86_64.rpm wget http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/varnish/varnish-libs-3.0.6-1.el6.x86_64.rpm yum -y install *.rpm

vim /etc/sysconfig/varnish
NFILES=131072
MEMLOCK=82000
#NPROCS="unlimited" #加注释
DAEMON_COREFILE_LIMIT="unlimited" #去注释
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80 #修改
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=1 #修改
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
VARNISH_STORAGE_SIZE=1G
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-t ${VARNISH_TTL} \
-w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_T
-u varnish -g varnish \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
/////////////////////// ( grep -v ^# /etc/sysconfig/varnish |sed '/^$/d' )

cp /etc/varnish/default.vcl{,.bak}
vim /etc/varnish/default.vcl
:set nu
:15,$s@^#@@g

backend default {
.host = "127.0.0.1";
.port = "80";
}
#
# Below is a commented-out copy of the default VCL logic. If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
sub vcl_recv {
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (lookup);
}

sub vcl_pipe {
# Note that only the first request to the backend will have
# X-Forwarded-For set. If you use X-Forwarded-For and want to
# have it set for all requests, make sure to have:
# set bereq.http.connection = "close";
# here. It is not set by default as it might break some broken web
# applications, like IIS with NTLM authentication.
return (pipe);
}

sub vcl_pass {
return (pass);
}

sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (hash);
}

sub vcl_hit {
return (deliver);
}

sub vcl_miss {
return (fetch);
}

sub vcl_fetch {
if (beresp.ttl <= 0s ||
beresp.http.Set-Cookie ||
beresp.http.Vary == "*") {
/*
* Mark as "Hit-For-Pass" for the next 2 minutes
*/
set beresp.ttl = 120 s;
return (hit_for_pass);
}
return (deliver);
}

sub vcl_deliver {
return (deliver);
}

sub vcl_error {
set obj.http.Content-Type = "text/html; charset=utf-8";
set obj.http.Retry-After = "5";
synthetic {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>"} + obj.status + " " + obj.response + {"</title>
</head>
<body>
<h1>Error "} + obj.status + " " + obj.response + {"</h1>
<p>"} + obj.response + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + req.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
"};
return (deliver);
}

sub vcl_init {
return (ok);
}

sub vcl_fini {
return (ok);
}
///////////////////

vim /etc/varnish/web.vcl
:set paste
######定义ACL
acl purgers { #定义acl，实现IP地址过滤
"127.0.0.1";
"172.16.0.0"/16;
}
######定义健康状态检测
probe dynamic { #设置动态网站服务器健康状态检测
.url = "/index.html";
.interval = 5s;
.timeout = 1s;
.expected_response= 200;
} #这里设置了两个健康状态检测主要是为了区分动、静网站
probe static { #设置动态网站服务器健康状态检测
.url = "/index.html"; #定义检测的页面
.interval = 5s; #探测请求的发送周期，默认为5秒
.timeout = 1s; #每次探测请求的过期时间
.expected_response = 200;
}
######定义后端服务器
backend app1 { #定义一个后端服务器
.host = "172.16.31.11"; #服务器地址
.port = "80"; #服务器监听端口
.probe = dynamic; #健康状态检测
}
backend app2 {
.host = "172.16.31.12";
.port = "80";
.probe = dynamic;
}
backend web1 {
.host = "172.16.31.16";
.port = "80";
.probe = static;
}
backend web2 {
.host = "172.16.31.17";
.port = "80";
.probe = static;
}
######定义后端动态服务器组，实现负载均衡效果
director apps random { #定义一个后端服务器组，实现负载均衡效果
{
.backend = app1; #调用前面已定义过的后端主机
.weight = 2; #设置权重
}
{
.backend = app2;
.weight = 2;
}
}
######定义后端静态服务器组，实现负载均衡效果
director webs random {
{
.backend = web1;
.weight = 2 ;
}
{
.backend = web2;
.weight = 2 ;
}
}
######定义vcl_recv函数，实现请求到达并成功接收后调用此函数中定义的规则
sub vcl_recv {
######定义动、静分离,以".php"或".php?后面跟所有文件"结尾的请求都发送到动态服务器，其他请求都发送到静态服务器
if (req.url ~ "\.php(\?\.*|$)") {
set req.backend = apps;
}else {
set req.backend = webs;
}
return(lookup);
######定义允许清除缓存的IP地址,调用的是前面定义的ACL
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "Method not allowed";
}
return(lookup);
}
######重新定义http请求首部，让后端服务器可以记录请求客户端的真实IP地址
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For +", " + client.ip;
} else {
set req.http.X-Forwarded-For =client.ip;
}
}
######除了定义的请求方法外，其他请求都到后端服务器
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
return (pipe);
}
if (req.request != "GET" && req.request !="HEAD") {
return (pass);
}
######定义不缓存认证与Cookie信息
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
######定义压缩功能
if (req.http.Accept-Enconding) {
if (req.url ~ "\.(jpg|jpeg|gif|bmp|png|flv|gz|tgz|tbz|mp3)$"){
remove req.http.Accept-Encoding;
remove req.http.Cookie;
} else if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else if (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else { remove req.http.Accept-Encoding;
}
}
######定义指定格式结尾的文件去除Cookie信息
if (req.request == "GET" && req.url ~"\.(jpeg|jpg|gif|png|bmp|swf)$") {
unset req.http.cookie;
}
######定义防盗链设置
if (req.http.referer ~ "http://.*") {
if (!(req.http.referer ~ "http://.*\.baidu\.com" ||req.http.referer ~ "http://.*\.google\.com.*")) {
set req.http.host ="www.stu31.com";
set req.url = "http://172.16.31.10/error.html";
}
}
}
######定义vcl_hash函数
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
}else {
hash_data(server.ip);
}
return(hash);
}
######定义vcl_hit函数
sub vcl_hit {
if (req.request == "PURGE") { #语法方法为"PURGE"
purge; #清除缓存
error 200 "Purged."; #返回错误状态码为"200"
}
return(deliver);
}
######定义vcl_miss函数
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 404 "Not In Cache.";
}
return(fetch);
}
######定义vcl_psss函数
sub vcl_pass {
if (req.request == "PURGE") {
error 502 "Purged On A Passed Object.";
}
return(pass);
}
######定义vcl_fetch函数
sub vcl_fetch {
######定义缓存，如果匹配到已定义文件结尾的缓存1天，其他则缓存1小时
if (req.request == "GET" && req.url ~"\.(html|jpg|png|bmp|jpeg|gif|js|ico|swf|css)$") {
set beresp.ttl = 1d;
set beresp.http.expires = beresp.ttl;
}else {
set beresp.ttl = 1h;
}
return(deliver);
}
######定义在http首部中，如果请求命中显示"HIT",未命中则显示"MISS",通过F12可查看缓存命中状态及varnish服务器IP
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "Hit from " + server.ip;
}else {
set resp.http.X-Cache = "MISS";
}
}
//////////////////////////// end for /etc/varnish/web.vcl

/etc/rc.d/init.d/varnish start
netstat -tnlp | grep varnish

varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache
varnish> vcl.list
varnish> quit

============== node4.stu31.com : 172.16.31.14 ===============
scp node3:/root/*.rpm ./
yum -y install *.rpm
scp node3:/etc/sysconfig/varnish /etc/sysconfig/
scp node3:/etc/varnish/*.vcl /etc/varnish

/etc/rc.d/init.d/varnish start
netstat -tnlp | grep varnish

varnishadm -S/etc/varnish/secret -T 127.0.0.1:6082
varnish> vcl.load cache web.vcl
varnish> vcl.use cache #立刻生效, 重启varnish失效
varnish> vcl.list
varnish> quit

如果我们希望后端的web服务器记录客户端访问的真实IP地址，我们需要配置httpd的配置文件中的日志格式：
node1, node2, node5, node6四个节点都要
vim /etc/httpd/conf/httpd.conf
LogFormat "%{X-Forwarded-For}i %l %u%t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""combined
///////////
service httpd reload
echo "ok" > /var/www/html/index.html #此页面用于varnish做健康检测

四. HAProxy反向代理构建

=============== ha.stu31.com : 172.16.31.10 ================
mount /dev/cdrom /mnt/cdrom
yum -y install haproxy

vim /etc/rsyslog.conf
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local2.* /var/log/haproxy.log #添加此行
//////////////
service rsyslog restart

cd /etc/haproxy/
cp haproxy.cfg{,.bak}
vim haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

:set paste

listen stats #添加此块
mode http
bind 0.0.0.0:1080
stats enable
stats hide-version
stats uri /haproxyadmin?stats
stats realm Haproxy\ Statistics
stats auth admin:admin
stats admin if TRUE

frontend http-in #修改很多内容
bind *:80
mode http
log global
option httpclose
option logasap
option dontlognull
capture request header Host len 20
capture request header Referer len 60
acl url_static path_beg -i /upload/static/image/
acl url_static path_end -i .html .jpeg .gif .png .jpg
acl url_dynamic path_end -i .php .css .js .jsp
use_backend static_servers if url_static
use_backend dynamic_servers if url_dynamic
default_backend static_servers

backend static_servers
balance roundrobin
server staticsrv1 172.16.31.13:80 check maxconn 3000
server staticsrv2 172.16.31.14:80 check maxconn 3000

backend dynamic_servers
balance source
server dynamicsrv1 172.16.31.11:80 check maxconn 3000
server dynamicsrv2 172.16.31.12:80 check maxconn 3000
//////////////////

service haproxy start

五. 测试访问

====================== windows =====================
访问论坛 http://172.16.31.10/upload/forum.php
访问haproxy的状态页 http://172.16.31.10:1080/haproxyadmin?stats admin
admin

window浏览器F12检查缓存命中情况

varnish上的日志
[root@node3 ~]# varnishlog

原始服务器的日志
[root@node1 ~]# tail /var/log/httpd/access_log