在CentOS 6上使用awstats分析httpd和tomcat日志

原创作品，允许转载，转载时请务必以超链接形式标明文章 原始出处 、作者信息和本声明。否则将追究法律责任。/article/4270235.html
[align=center]用Awstats工具来分析httpd和tomcat的访问日志[/align][align=left]准备工作：[/align][align=left]Awstats 是由perl语言编写的，所以要首先准备好awstats的运行环境。[/align][align=left]# yum install –y perl*[/align]一、首先，要安装apache服务器，并且启动httpd服务。[align=left]我的apache是yum安装的，配置文件路径为：/etc/httpd/conf/httpd.conf[/align][align=left]下载awstats工具。[/align][align=left]下载地址：http://www.awstats.org/files/awstats-7.1.tar.gz[/align][align=left]# cd /opt[/align][align=left]# wget http://www.awstats.org/files/awstats-7.1.tar.gz[/align][align=left]# tar xvf awstats-7.1.tar.gz[/align][align=left]# mv awstats-7.1 /usr/local/awstats[/align]二、配置awstats分析httpd的访问日志# cd /usr/local/awstats# ls# cd tools# perl awstats_configure.pl[align=left]----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----[/align][align=left]This tool will help you to configure AWStats to analyze statistics for[/align][align=left]one web server. You can try to use it to let it do all that is possible[/align][align=left]in AWStats setup, however following the step by step manual setup[/align][align=left]documentation (docs/index.html) is often a better idea. Above all if:[/align][align=left]- You are not an administrator user,[/align][align=left]- You want to analyze downloaded log files without web server,[/align][align=left]- You want to analyze mail or ftp log files instead of web log files,[/align][align=left]- You need to analyze load balanced servers log files,[/align][align=left]- You want to 'understand' all possible ways to use AWStats...[/align][align=left]Read the AWStats documentation (docs/index.html).[/align][align=left][/align][align=left]-----> Running OS detected: Linux, BSD or Unix[/align][align=left][/align][align=left]-----> Check for web server install[/align][align=left][/align][align=left]Enter full config file path of your Web server.[/align][align=left]Example: /etc/httpd/httpd.conf[/align][align=left]Example: /usr/local/apache2/conf/httpd.conf[/align][align=left]Example:c:\Programfiles\apachegroup\apache\conf\httpd.conf[/align][align=left]Config file path ('none' to skip web server setup):[/align][align=left]>/etc/httpd/conf/httpd.conf 根据自己的httpd服务安装的具体路径填写[/align][align=left]-----> Check and complete web server config file[/align][align=left]'/etc/httpd/conf/httpd.conf'[/align][align=left] Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'[/align][align=left] Add 'Alias /awstatscss[/align][align=left]"/usr/local/awstats/wwwroot/css/"'[/align][align=left] Add 'Alias /awstatsicons[/align][align=left]"/usr/local/awstats/wwwroot/icon/"'[/align][align=left] Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'[/align][align=left] Add '<Directory>' directive[/align][align=left] AWStats directives added to Apache config file.[/align][align=left][/align][align=left]-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'[/align][align=left] File awstats.model.conf updated.[/align][align=left][/align][align=left]-----> Need to create a new config file ?[/align][align=left]Do you want me to build a new AWStats config/profile[/align][align=left]file (required if first install) [y/N] ?y[/align][align=left][/align][align=left]-----> Define config file name to create[/align][align=left]What is the name of your web site or profile analysis ?[/align][align=left]Example: www.mysite.com[/align][align=left]Example: demo[/align][align=left]Your web site, virtual server or profile name:[/align][align=left]>lingling 可以是任意的名字，也可以是完整的域名格式，只是为了区分你要分析的那份日志的来源的网站，自己注意不要混淆就好。[/align][align=left][/align][align=left]-----> Define config file path[/align][align=left]In which directory do you plan to store your config file(s) ?[/align][align=left]Default: /etc/awstats[/align][align=left]Directory path to store config file(s) (Enter for default):[/align][align=left]>[/align][align=left]默认的awstats生成的配置文件目录，根据喜好可以更改。[/align][align=left]-----> Create config file '/etc/awstats/awstats.lingling.conf'[/align][align=left] Config file /etc/awstats/awstats.lingling.conf created.[/align][align=left][/align][align=left]-----> Restart Web server with '/sbin/service httpd restart'[/align][align=left]Stopping httpd: [OK][/align][align=left]Starting httpd: [OK][/align][align=left][/align][align=left]-----> Add update process inside a scheduler[/align][align=left]Sorry, configure.pl does not support automatic add to cron yet.[/align][align=left]You can do it manually by adding the following command to your cron:[/align][align=left]/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling[/align][align=left]Or if you have several config files and prefer having only one command:[/align][align=left]/usr/local/awstats/tools/awstats_updateall.pl now[/align][align=left]Press ENTER to continue...[/align][align=left][/align][align=left]A SIMPLE config file has been created: /etc/awstats/awstats.lingling.conf[/align][align=left]You should have a look inside to check and change manually main parameters.[/align][align=left]You can then manually update your statistics for 'lingling' with command:[/align][align=left]> perl awstats.pl -update -config=lingling[/align][align=left]You can also read your statistics for 'lingling' with URL:[/align][align=left]> http://localhost/awstats/awstats.pl?config=lingling[/align][align=left][/align][align=left]Press ENTER to finish...[/align][align=left][/align]1、由于httpd的log文件默认是/var/log/httpd/access.log，[align=left]所以要修改/etc/awstats/awstats.lingling.conf文件里的LogFile：[/align][align=left]把LogFile="/var/log/httpd/mylog.log"改为LogFile="/var/log/httpd/access_log"[/align][align=left]或者LogFile="var/log/access_log.%YYYY-0%MM-0%DD-0.log"[/align]2、然后，手动更新一下：[align=left]# cd /usr/local/awstats/wwwroot/cgi-bin/[/align][align=left]# perl awstats.pl –update –config=lingling[/align][align=left]或者：# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling[/align][align=left]Error: AWStats database directory defined in config file by 'DirData' parameter (/var/lib/awstats) does not exist or is not writable.[/align][align=left]Setup ('/etc/awstats/awstats.lingling.conf' file, web server or permissions) may be wrong.[/align][align=left]Check config file, permissions and AWStats documentation (in 'docs' directory).[/align][align=left]出错：忘记了创建awstats的默认数据存放目录[/align][align=left]解决：# mkdir –m 755 /var/lib/awstats[/align][align=left]# perl/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling[/align][align=left]Create/Update database for config "/etc/awstats/awstats.lingling.conf" by AWStats version 7.0 (build 1.976)[/align][align=left]From data in log file "/var/log/httpd/access_log"...[/align][align=left]Phase 1 : First bypass old records, searching new record...[/align][align=left]Direct access after last parsed record (after line 33664)[/align][align=left]Jumped lines in file: 33664[/align][align=left] Found 33664 already parsed records.[/align][align=left]Parsed lines in file:7[/align][align=left] Found 0 dropped records,[/align][align=left] Found 0 comments,[/align][align=left] Found 0 blank records,[/align][align=left] Found 0 corrupted records,[/align][align=left] Found 7 old records,[/align][align=left] Found 0 new qualified records.[/align]3、打开浏览器，用awstats分析日志：[align=left]http://10.100.10.11/awstats/awstats.pl?config=lingling[/align][align=left]分析结果如下图:[/align]

4、可以将更新的命令作为执行计划，使其每天执行一次，方便分析前一天的日。[align=left]# crontab –e[/align][align=left]10 1 * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling > /dev/null 2&>1[/align]三、用awstats分析tomcat的访问日志1、要分析tomcat的日志，就要首先了解其日志格式。[align=left]并比较与httpd的访问日志格式有什么不同之处，然后就可以参照awstats分析httpd日志的格式来定义awstats分析tomcat的日志。[/align][align=left]我的tomcat服务器上定义的访问日志格式如下：[/align][align=left]<Valve className="org.apache.catalina.valves.[/align][align=left]AccessLogValve" directory="logs"[/align][align=left]prefix="localhost_access_log." suffix=".txt"[/align][align=left] pattern="%h %l %u %t "%r" %s %b" />[/align][align=left]%...a: 远程IP地址[/align][align=left]%...A: 本地IP地址[/align][align=left]%...B: 已发送的字节数，不包含HTTP头[/align][align=left]%...b: CLF格式的已发送字节数量，不包含HTTP头。[/align][align=left]例如当没有发送数据时，写入‘-’而不是0。[/align][align=left]%e: 环境变量FOOBAR的内容[/align][align=left]%...f: 文件名字[/align][align=left]%...h: 远程主机[/align][align=left]%...H 请求的协议[/align][align=left]%i: Foobar的内容，发送给服务器的请求的标头行。[/align][align=left]%...l: 远程登录名字（来自identd，如提供的话）[/align][align=left]%...m 请求的方法[/align][align=left]%n: 来自另外一个模块的注解“Foobar”的内容[/align][align=left]%o: Foobar的内容，应答的标头行[/align][align=left]%...p: 服务器响应请求时使用的端口[/align][align=left]%...P: 响应请求的子进程ID。[/align][align=left]%...q 查询字符串（如果存在查询字符串，则包含“?”后面的[/align][align=left]部分；否则，它是一个空字符串。）[/align][align=left]%...r: 请求的第一行[/align][align=left]%...s: 状态。对于进行内部重定向的请求，这是指*原来*请求[/align][align=left]的状态。如果用%...>s，则是指后来的请求。[/align][align=left]%...t: 以公共日志时间格式表示的时间（或称为标准英文格式）[/align][align=left]%t: 以指定格式format表示的时间[/align][align=left]%...T: 为响应请求而耗费的时间，以秒计[/align][align=left]%...u: 远程用户（来自auth；如果返回状态（%s）是401则可能是伪造的）[/align][align=left]%...U: 用户所请求的URL路径[/align][align=left]%...v: 响应请求的服务器的ServerName [/align][align=left]%...V: 依照UseCanonicalName设置得到的服务器名字[/align][align=left]最后的tomcat的访问日志内容如下：[/align][align=left]203.156.200.162 - - [29/Aug/2012:11:16:58 +0800] "GET /front/magazine/getContent.htm?contentId=124504 HTTP/1.1" 200 20001[/align]2、由于我的tomcat服务器是在其他机器上，所以我将tomcat的服务日志copy到本机的/var/log/httpd/下即可。[align=left]如copy的文件是：localhost_access_log.2012-08-29.txt[/align]3、配置awstats分析此日志(tomcat 的域名并不是httpd的虚拟主机，所以没有写进httpd.conf文件里面)[align=left]# cd /usr/local/awstats/tools[/align][align=left]# perl awstats_configure.pl[/align][align=left]----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----[/align][align=left]This tool will help you to configure AWStats to analyze statistics for[/align][align=left]one web server. You can try to use it to let it do all that is possible[/align][align=left]in AWStats setup, however following the step by step manual setup[/align][align=left]documentation (docs/index.html) is often a better idea. Above all if:[/align][align=left]- You are not an administrator user,[/align][align=left]- You want to analyze downloaded log files without web server,[/align][align=left]- You want to analyze mail or ftp log files instead of web log files,[/align][align=left]- You need to analyze load balanced servers log files,[/align][align=left]- You want to 'understand' all possible ways to use AWStats...[/align][align=left]Read the AWStats documentation (docs/index.html).[/align][align=left][/align][align=left]-----> Running OS detected: Linux, BSD or Unix[/align][align=left][/align][align=left]-----> Check for web server install[/align][align=left][/align][align=left]Enter full config file path of your Web server.[/align][align=left]Example: /etc/httpd/httpd.conf[/align][align=left]Example: /usr/local/apache2/conf/httpd.conf[/align][align=left]Example: c:\Program files\apache group\apache\conf\httpd.conf[/align][align=left]Config file path ('none' to skip web server setup):[/align][align=left]>none[/align][align=left]Your web server config file(s) could not be found.[/align][align=left]You will need to setup your web server manually to declare AWStats[/align][align=left]script as a CGI, if you want to build reports dynamically.[/align][align=left]See AWStats setup documentation (file docs/index.html)[/align][align=left][/align][align=left]-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'[/align][align=left] File awstats.model.conf updated.[/align][align=left][/align][align=left]-----> Need to create a new config file ?[/align][align=left]Do you want me to build a new AWStats config/profile[/align][align=left]file (required if first install) [y/N] ? y[/align][align=left][/align][align=left]-----> Define config file name to create[/align][align=left]What is the name of your web site or profile analysis ?[/align][align=left]Example: www.mysite.com[/align][align=left]Example: demo[/align][align=left]Your web site, virtual server or profile name:[/align][align=left]>buoqu.com[/align][align=left]-----> Define config file path[/align][align=left]In which directory do you plan to store your config file(s) ?[/align][align=left]Default: /etc/awstats[/align][align=left]Directory path to store config file(s) (Enter for default):[/align][align=left]>[/align][align=left][/align][align=left]-----> Create config file '/etc/awstats/awstats.buoqu.com.conf'[/align][align=left] Config file /etc/awstats/awstats.buoqu.com.conf created.[/align][align=left][/align][align=left]-----> Add update process inside a scheduler[/align][align=left]Sorry, configure.pl does not support automatic add to cron yet.[/align][align=left]You can do it manually by adding the following command to your cron:[/align][align=left]/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=buoqu.com[/align][align=left]Or if you have several config files and prefer having only one command:[/align][align=left]/usr/local/awstats/tools/awstats_updateall.pl now[/align][align=left]Press ENTER to continue...[/align][align=left][/align][align=left][/align][align=left]A SIMPLE config file has been created: /etc/awstats/awstats.buoqu.com.conf[/align][align=left]You should have a look inside to check and change manually main parameters.[/align][align=left]You can then manually update your statistics for 'buoqu.com' with command:[/align][align=left]> perl awstats.pl -update -config=buoqu.com[/align][align=left]You can also build static report pages for 'buoqu.com' with command:[/align][align=left]> perl awstats.pl -output=pagetype -config=buoqu.com[/align][align=left][/align][align=left]Press ENTER to finish...[/align]4、修改要分析日志文件[align=left]# vim /etc/awstats/awstats.buoqu.com.conf[/align][align=left]将LogFile="/var/log/httpd/mylog.log"[/align][align=left]改为LogFile="/var/log/httpd/localhost_access_log.2012-08-29.txt"[/align][align=left]LogFile="/var/log/httpd/localhost_access_log.YYY-0%MM-0%DD-0.txt"都可以。[/align]5、重启httpd服务，并分析日志[align=left]# service httpd restart[/align][align=left]# cd /usr/local/awstats/wwwroot/cgi-bin[/align][align=left]# perl awstats.pl -update -config=buoqu.com[/align][align=left]Create/Update database for config "/etc/awstats/awstats.buoqu.com.conf" by AWStats version 7.0 (build 1.976)[/align][align=left]From data in log file "/var/log/httpd/localhost_access_log.2012-08-29.txt"...[/align][align=left]Phase 1 : First bypass old records, searching new record...[/align][align=left]Searching new records from beginning of log file...[/align][align=left]AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.[/align][align=left]Your log file /var/log/httpd/localhost_access_log.2012-08-29.txt must have a bad format or LogFormat parameter setup does not match this format.[/align][align=left]Your AWStats LogFormat parameter is:[/align][align=left]1[/align][align=left]This means each line in your web server log file need to have "combined log format" like this:[/align][align=left]111.22.33.44 - - [10/Jan/2001:02:14:14 +0200] "GET / HTTP/1.1" 200 1234 "http://www.fromserver.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"[/align][align=left]And this is an example of records AWStats found in your log file (the record number 50 in your log):[/align][align=left]203.208.60.231 - - [29/Aug/2012:00:02:47 +0800] "GET /front/magazine/getContent.htm?contentId=52253 HTTP/1.1" 200 18419[/align][align=left]Setup ('/etc/awstats/awstats.buoqu.com.conf' file, web server or permissions) may be wrong.[/align][align=left]Check config file, permissions and AWStats documentation (in 'docs' directory).[/align]出错：日志格式不匹配。[align=left]解决：这个时候，就知道我为什么要先了解怎么定义tomcat的日志格式了。[/align][align=left]修改文件/etc/awstats/awstats.buoqu.com.conf[/align][align=left]# vim /etc/awstats/awstats.buoqu.com.conf[/align][align=left]LogFormat = 1[/align][align=left]LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot" 这两行是注释的，1表示是web日志，下面是httpd的日志格式[/align][align=left]LogFormat ="%host %other %logname %time1 %methodurl %code"[/align][align=left]这是我修改后的匹配tomcat的日志格式[/align][align=left]# perl awstats.pl -update -config=buoqu.com[/align][align=left]Create/Update database for config "/etc/awstats/awstats.buoqu.com.conf" by AWStats version 7.0 (build 1.976)[/align][align=left]From data in log file "/var/log/httpd/localhost_access_log.2012-08-29.txt"...[/align][align=left]Phase 1 : First bypass old records, searching new record...[/align][align=left]Searching new records from beginning of log file...[/align][align=left]Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...[/align][align=left]Jumped lines in file: 0[/align][align=left]Parsed lines in file: 22442[/align][align=left] Found 0 dropped records,[/align][align=left] Found 0 comments,[/align][align=left] Found 0 blank records,[/align][align=left] Found 0 corrupted records,[/align][align=left] Found 0 old records,[/align][align=left] Found 22442 new qualified records.[/align]6、打开网址查看分析结果：[align=left]http://10.100.10.11/awstats/awstats.pl?config=buoqu.com[/align]

7、手动执行命令可写入crontab。[align=left]①、如果，想在分析页面上直接刷新，可以开启AllowToUpdateStatsFromBrowser=1，默认情况下是关闭的。[/align][align=left]②、若是想每个页面上都直接有“立即更新”的按钮，而不想每次都手动的修改配置文件的话，可以再awstats的基本配置文件里修改。[/align][align=left]# cd /usr/local/awstats/wwwroot/cgi-bin[/align][align=left]# vim awstats.model.conf[/align][align=left]将AllowToUpdateStatsFromBrowser=0改为AllowToUpdateStatsFromBrowser=1即可。[/align][align=left]这样，以后的网页都可以直接点击刷新的。[/align][align=left]注意：每次修改配置文件后要重启httpd服务[/align][align=left]③、若是要在浏览器上直接刷新，那么apache用户就要有对数据文件操作的权限[/align][align=left]# chown apache.apache –R /var/lib/awstats[/align][align=left]# chmod 755 /var/log/httpd[/align][align=left]效果如图：[/align]

四、添加一些插件，使awstats看起来更人性化和直观化。1、首先下载所需要的插件：[align=left]# cd /opt[/align][align=left]#wget http://www.maxmind.com/download/geoip/api/c/GeoIP-1.4.8.tar.gz[/align][align=left]# tar xvf GeoIP-1.4.8.tar.gz[/align][align=left]# cd GeoIP-1.4.8[/align][align=left]# ./configure[/align][align=left]# make && make install[/align][align=left]# cd ..[/align][align=left]#wget http://www.maxmind.com/download/geoip/api/perl/Geo-IP-1.40.tar.gz[/align][align=left]# tar xvf Geo-IP-1.40.tar.gz[/align][align=left]# cd Geo-IP-1.40[/align][align=left]# perl Makeinstall.pl[/align][align=left]# make && make install[/align][align=left]# cd ..[/align][align=left]# wget http://www.maxmind.com/download/geoip/api/pureperl/Geo-IP-PurePerl-1.25.tar.gz[/align][align=left]#tar xvf Geo-IP-PurePerl-1.25.tar.gz[/align][align=left]# perl –MCPAN –e shell[/align][align=left]cpan[1]>install Geo::IP[/align][align=left]cpan[2]>install Geo::IP::PurePerl[/align][align=left]# cd /opt[/align][align=left]# wgethttp://mirrors.download3k.com/token/4b6cdc111cdc95edb29534189f0ef839/GeoIP.dat.gz[/align][align=left]# tar xvf GeoIP.dat.gz[/align]2、定义插件的使用方法：[align=left]可在各自的/etc/awstats/awstats.domain.conf的配置文件中定义，也可在awstats的主配置文件定义，然后全局生效。[/align][align=left]如：[/align][align=left]# vim /usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf[/align][align=left]将以下语句的#注释去掉：[/align][align=left]#LoadPlugin="tooltips" 在html报告中增加一些提示信息[/align][align=left]#LoadPlugin="decodeutfkeys" 处理搜索引擎UTF8编码的关键字[/align][align=left]#LoadPlugin="geoip GEOIP_STANDARD /opt/GeoIP.dat"[/align][align=left]从Internet IP-Country数据库生成访问者来自国家的统计图表[/align][align=left]并增加一条LoadPlugin="qqhostinfo"[/align][align=left]在访问主机表格中增加一列，显示ip对应的位置[/align][align=left]qqhostinfo.pm及需要的qqwry.pl、QQWry.dat在http://down.51cto.com/data/496275里，有使用需求的可下载。[/align][align=left]下载之后，要将这三个文件，放至/usr/local/awstats/wwwroot/cgi-bin/plugins/下面：[/align][align=left]# cd /usr/local/awstats/wwwroot/cgi-bin/plugins[/align][align=left]# chmod 777 . –R[/align]