LAMP搭建15：Apache禁止指定user_agent访问
2017-01-16 15:57
387 查看
查看Apache访问日志的user_agent：
[root@centos6 apache2]# cd logs/
[root@centos6 logs]# ls
access_log test.com-access_20170114_log test.com-error_log
error_log test.com-access_20170115_log
httpd.pid test.com-access_log
[root@centos6 logs]#
[root@centos6 logs]# ls
access_log test.com-access_20170114_log test.com-error_log
error_log test.com-access_20170115_log
httpd.pid test.com-access_log
[root@centos6 logs]# tail test.com-access_20170115_log
……
192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /data/info.php HTTP/1.1" 403 215 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /favicon.ico HTTP/1.1" 200 5558 "http://www.test.com/data/info.php" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
127.0.0.1 - - [15/Jan/2017:01:40:13 +0800] "GET http://www.test.com/data/info.php HTTP/1.1" 200 20 "-" "curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
可以禁止指定user_agent访问我们的网站，尤其是浏览器爬虫偷跑流量。编辑虚拟主机配置文件：
[root@centos6 logs]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf
加入如下内容：禁止curl和chrome
……
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.bbb.com$
RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]
RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]
RewriteRule .* - [F]
</IfModule>
<Directory /data/www/important/>
AllowOverride AuthConfig
AuthName "username"
AuthType Basic
……
[root@centos6 apache2]# apachectl -t
Syntax OK
[root@centos6 apache2]# apachectl graceful
禁用后chrome浏览器不再能访问我们的网站，
但是其他浏览器仍可以访问：IE浏览器可以访问
360浏览器也可以访问
curl测试返回值也都是403，
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
</body></html>
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/forum.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /forum.php
on this server.</p>
</body></html>
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/data/info.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /data/info.php
on this server.</p>
</body></html>
[root@centos6 apache2]#
正常情况下的curl能找到的页面返回200，找不到的页面返回404，禁止curl之后都返回403。这里我们还需要使用curl进行测试，先去掉curl的禁用：
[root@centos6 apache2]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf
……
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.bbb.com$
RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]
#RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]
RewriteRule .* - [F]
</IfModule>
……
[root@centos6 apache2]# apachectl -t
Syntax OK
[root@centos6 apache2]# apachectl graceful
[root@centos6 apache2]# curl -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 19:38:46 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
X-Powered-By: PHP/5.4.36
Set-Cookie: sTi8_2132_saltkey=QmAybXxa; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/; httponly
Set-Cookie: sTi8_2132_lastvisit=1484419126; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_checkpatch=1; expires=Sat, 14-Jan-2017 19:39:46 GMT; path=/
Set-Cookie: sTi8_2132_lastact=1484422726%09forum.php%09; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:43:46 GMT; path=/
Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Cache-Control: max-age=0
Expires: Sat, 14 Jan 2017 19:38:46 GMT
Content-Type: text/html; charset=gbk
使用-A选项模拟user_agent，不含"chrome"时可以访问，加入"chrome"后不能访问
[root@centos6 apache2]# curl -A "asdfghjkl" -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 19:40:17 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
X-Powered-By: PHP/5.4.36
Set-Cookie: sTi8_2132_saltkey=ZggZv4O6; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/; httponly
Set-Cookie: sTi8_2132_lastvisit=1484419217; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_lastact=1484422817%09forum.php%09; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:45:17 GMT; path=/
Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Cache-Control: max-age=0
Expires: Sat, 14 Jan 2017 19:40:17 GMT
Content-Type: text/html; charset=gbk
[root@centos6 apache2]# curl -A "asdchromefghjkl" -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 403 Forbidden
Date: Sat, 14 Jan 2017 19:41:33 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
Content-Type: text/html; charset=iso-8859-1
[root@centos6 apache2]# cd logs/
[root@centos6 logs]# ls
access_log test.com-access_20170114_log test.com-error_log
error_log test.com-access_20170115_log
httpd.pid test.com-access_log
[root@centos6 logs]#
[root@centos6 logs]# ls
access_log test.com-access_20170114_log test.com-error_log
error_log test.com-access_20170115_log
httpd.pid test.com-access_log
[root@centos6 logs]# tail test.com-access_20170115_log
……
192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /data/info.php HTTP/1.1" 403 215 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /favicon.ico HTTP/1.1" 200 5558 "http://www.test.com/data/info.php" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
127.0.0.1 - - [15/Jan/2017:01:40:13 +0800] "GET http://www.test.com/data/info.php HTTP/1.1" 200 20 "-" "curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
可以禁止指定user_agent访问我们的网站，尤其是浏览器爬虫偷跑流量。编辑虚拟主机配置文件：
[root@centos6 logs]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf
加入如下内容：禁止curl和chrome
……
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.bbb.com$
RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]
RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]
RewriteRule .* - [F]
</IfModule>
<Directory /data/www/important/>
AllowOverride AuthConfig
AuthName "username"
AuthType Basic
……
[root@centos6 apache2]# apachectl -t
Syntax OK
[root@centos6 apache2]# apachectl graceful
禁用后chrome浏览器不再能访问我们的网站，
但是其他浏览器仍可以访问：IE浏览器可以访问
360浏览器也可以访问
curl测试返回值也都是403，
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
</body></html>
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/forum.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /forum.php
on this server.</p>
</body></html>
[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/data/info.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /data/info.php
on this server.</p>
</body></html>
[root@centos6 apache2]#
正常情况下的curl能找到的页面返回200，找不到的页面返回404，禁止curl之后都返回403。这里我们还需要使用curl进行测试，先去掉curl的禁用：
[root@centos6 apache2]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf
……
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.bbb.com$
RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]
#RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]
RewriteRule .* - [F]
</IfModule>
……
[root@centos6 apache2]# apachectl -t
Syntax OK
[root@centos6 apache2]# apachectl graceful
[root@centos6 apache2]# curl -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 19:38:46 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
X-Powered-By: PHP/5.4.36
Set-Cookie: sTi8_2132_saltkey=QmAybXxa; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/; httponly
Set-Cookie: sTi8_2132_lastvisit=1484419126; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_checkpatch=1; expires=Sat, 14-Jan-2017 19:39:46 GMT; path=/
Set-Cookie: sTi8_2132_lastact=1484422726%09forum.php%09; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:43:46 GMT; path=/
Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/
Cache-Control: max-age=0
Expires: Sat, 14 Jan 2017 19:38:46 GMT
Content-Type: text/html; charset=gbk
使用-A选项模拟user_agent，不含"chrome"时可以访问，加入"chrome"后不能访问
[root@centos6 apache2]# curl -A "asdfghjkl" -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 19:40:17 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
X-Powered-By: PHP/5.4.36
Set-Cookie: sTi8_2132_saltkey=ZggZv4O6; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/; httponly
Set-Cookie: sTi8_2132_lastvisit=1484419217; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_lastact=1484422817%09forum.php%09; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:45:17 GMT; path=/
Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/
Cache-Control: max-age=0
Expires: Sat, 14 Jan 2017 19:40:17 GMT
Content-Type: text/html; charset=gbk
[root@centos6 apache2]# curl -A "asdchromefghjkl" -x192.168.147.132:80 www.test.com/forum.php -I
HTTP/1.1 403 Forbidden
Date: Sat, 14 Jan 2017 19:41:33 GMT
Server: Apache/2.2.9 (Unix) PHP/5.4.36
Content-Type: text/html; charset=iso-8859-1
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- LAMP 2.6 Apache 禁止指定user_agent
- LAMP - Apache禁止指定user_agent
- LAMP--Apache 禁止指定 user_agent
- Apache禁止指定的user_agent、指定目录被访问
- apache学习笔记（访问控制|禁止解析|禁止指定user_agent）
- apache‘禁止指定user_agent’ 和 ‘禁止用户通过浏览器访问某个目录’
- Apache禁止指定user_agent
- LAMP（apache禁止解析php，限制user_agent，php配置，open_bashdir
- Centos7.2 Apache根据User-Agent设置访问禁止
- apache禁止指定user_agent和禁止解析php
- Apache2.4使用require指令进行访问控制--允许或限制IP访问/通过User-Agent禁止不友好网络爬虫
- 2.6-Apache禁止指定user_agent
- Apache配置——禁止指定user_agent
- Apache2.4使用require指令进行访问控制–允许或限制IP访问/通过User-Agent禁止不友好网络爬虫
- Apache 禁止指定user_agent---模块
- 【LAMP环境搭建】限定某个目录禁止解析php、限制user_agent和php相关配置
- LNMP搭建13：Nginx禁止指定user_agent
- Apache2.4使用require指令进行访问控制--允许或限制IP访问/通过User-Agent禁止不友好网络爬虫 从Apache2.2升级到Apache2.4后，发现原来用来限制部分I
- Apache禁止指定user_agent
- Apache2.4使用require指令进行访问控制--允许或限制IP访问/通过User-Agent禁止不友好网络爬虫