CentOS 6.4 利用 Awstats 7.2 分析 Nginx 日志

一、配置Nginx 日志分割 (略)

二、配置FCGI

1、安装CPAN

wget http://search.cpan.org/CPAN/authors/id/A/AN/ANDK/CPAN-2.00.tar.gz
tar zxf CPAN-2.00.tar.gz

cd CPAN-2.00

perl Makefile.PL

make && make install

2、安装FCGI和FCGI::ProcManager

wget http://search.cpan.org/CPAN/authors/id/F/FL/FLORA/FCGI-0.74.tar.gz
tar zxf FCGI-0.74.tar.gz

cd FCGI-0.74

第一种安装方法：perl -MCPAN -e 'install FCGI'

第二种安装方法：perl Makefile.PL

make&&make install

wget http://search.cpan.org/CPAN/authors/id/B/BO/BOBTFISH/FCGI-ProcManager-0.24.tar.gz
tar zxf FCGI-ProcManager-0.24.tar.gz

cd FCGI-ProcManager-0.24

第一种安装方法：perl -MCPAN -e 'install FCGI::ProcManager'

第二种安装方法：perl Makefile.PL

make&&make install

3、创建FCGI启动文件

vi /opt/local/nginx/sbin/fcgi

#!/usr/bin/perl

use FCGI;

#perl -MCPAN -e 'install FCGI'

use Socket;

use POSIX qw(setsid);

#use Fcntl;

require 'syscall.ph';

&daemonize;

#this keeps the program alive or something after exec'ing perl scripts

END() { } BEGIN() { }

*CORE::GLOBAL::exit = sub { die "fakeexit\nrc=".shift()."\n"; };

eval q{exit};

if ($@) {

exit unless $@ =~ /^fakeexit/;

};

&main;

sub daemonize() {

chdir '/' or die "Can't chdir to /: $!";

defined(my $pid = fork) or die "Can't fork: $!";

exit if $pid;

setsid or die "Can't start a new session: $!";

umask 0;

}

sub main {

#$socket = FCGI::OpenSocket( "127.0.0.1:8999", 10 );

$socket = FCGI::OpenSocket( "/opt/local/nginx/fastcgi_temp/perl_cgi-dispatch.sock", 10 );

#use UNIX sockets - user running this script must have w access to the 'nginx' folder!!

$request = FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%req_params, $socket );

if ($request) { request_loop()};

FCGI::CloseSocket( $socket );

}

sub request_loop {

while( $request->Accept() >= 0 ) {

#processing any STDIN input from WebServer (for CGI-POST actions)

$stdin_passthrough ='';

$req_len = 0 + $req_params{'CONTENT_LENGTH'};

if (($req_params{'REQUEST_METHOD'} eq 'POST') && ($req_len != 0) ){

my $bytes_read = 0;

while ($bytes_read < $req_len) {

my $data = '';

my $bytes = read(STDIN, $data, ($req_len - $bytes_read));

last if ($bytes == 0 || !defined($bytes));

$stdin_passthrough .= $data;

$bytes_read += $bytes;

}

}

#running the cgi app

if ( (-x $req_params{SCRIPT_FILENAME}) && #can I execute this?

(-s $req_params{SCRIPT_FILENAME}) && #Is this file empty?

(-r $req_params{SCRIPT_FILENAME}) #can I read this file?

){

pipe(CHILD_RD, PARENT_WR);

my $pid = open(KID_TO_READ, "-|");

unless(defined($pid)) {

print("Content-type: text/plain\r\n\r\n");

print "Error: CGI app returned no output - Executing $req_params

{SCRIPT_FILENAME} failed !\n";

next;

}

if ($pid > 0) {

close(CHILD_RD);

print PARENT_WR $stdin_passthrough;

close(PARENT_WR);

while(my $s = <KID_TO_READ>) { print $s; }

close KID_TO_READ;

waitpid($pid, 0);

} else {

foreach $key ( keys %req_params){

$ENV{$key} = $req_params{$key};

}

# cd to the script's local directory

if ($req_params{SCRIPT_FILENAME} =~ /^(.*)\/[^\/]+$/) {

chdir $1;

}

close(PARENT_WR);

close(STDIN);

#fcntl(CHILD_RD, F_DUPFD, 0);

syscall(&SYS_dup2, fileno(CHILD_RD), 0);

#open(STDIN, "<&CHILD_RD");

exec($req_params{SCRIPT_FILENAME});

die("exec failed");

}

}

else {

print("Content-type: text/plain\r\n\r\n");

print "Error: No such CGI app - $req_params{SCRIPT_FILENAME} may not exist or is

not executable by this process.\n";

}

}

}

创建完成后，需要赋予fcgi执行权限：

chmod 755 /opt/local/nginx/sbin/fcgi

启动FPM（FastCGI 进程管理器）

perl /opt/local/nginx/sbin/fcgi

(执行出错,Can't locate sys/syscall.ph in @INC (did you run h2ph?) (@INC contains: /usr/lib/perl5/i686-linux…… )

运行 cd /usr/include

h2ph *.h */*.h

[root@localhost tools]# ./awstats_configure.pl

----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----

This tool will help you to configure AWStats to analyze statistics for

one web server. You can try to use it to let it do all that is possible

in AWStats setup, however following the step by step manual setup

documentation (docs/index.html) is often a better idea. Above all if:

- You are not an administrator user,

- You want to analyze downloaded log files without web server,

- You want to analyze mail or ftp log files instead of web log files,

- You need to analyze load balanced servers log files,

- You want to 'understand' all possible ways to use AWStats...

Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix

Warning: AWStats standard directory on Linux OS is '/usr/local/awstats'.

If you want to use standard directory, you should first move all content

of AWStats distribution from current directory:

/opt/local/awstats

to standard directory:

/usr/local/awstats

And then, run configure.pl from this location.

Do you want to continue setup from this NON standard directory [yN] ? y 非默认配置路径 是否 继续

-----> Check for web server install

Enter full config file path of your Web server.

Example: /etc/httpd/httpd.conf

Example: /usr/local/apache2/conf/httpd.conf

Example: c:\Program files\apache group\apache\conf\httpd.conf 是否选择这几个配置文件 Nginx 选 none 为否

Config file path ('none' to skip web server setup):

> none

Your web server config file(s) could not be found.

You will need to setup your web server manually to declare AWStats

script as a CGI, if you want to build reports dynamically.

See AWStats setup documentation (file docs/index.html)

-----> Update model config file '/opt/local/awstats/wwwroot/cgi-bin/awstats.model.conf'

File awstats.model.conf updated.

-----> Need to create a new config file ?

Do you want me to build a new AWStats config/profile

file (required if first install) [y/N] ? y 是否创建配置文件 选择 y

-----> Define config file name to create

What is the name of your web site or profile analysis ?

Example: www.mysite.com

Example: demo

Your web site, virtual server or profile name: 网站域名

> aws.qq.com

-----> Define config file path

In which directory do you plan to store your config file(s) ?

Default: /etc/awstats

Directory path to store config file(s) (Enter for default): 配置文件保存路径

> /opt/local/awstats/

-----> Create config file '/opt/local/awstats//awstats.aws.qq.com.conf'

Config file /opt/local/awstats//awstats.aws.qq.com.conf created.

-----> Add update process inside a scheduler

Sorry, configure.pl does not support automatic add to cron yet.

You can do it manually by adding the following command to your cron:

/opt/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=aws.qq.com

Or if you have several config files and prefer having only one command:

/opt/local/awstats/tools/awstats_updateall.pl now

Press ENTER to continue...

A SIMPLE config file has been created: /opt/local/awstats//awstats.aws.qq.com.conf

You should have a look inside to check and change manually main parameters.

You can then manually update your statistics for 'aws.qq.com' with command:

> perl awstats.pl -update -config=aws.qq.com

You can also build static report pages for 'aws.qq.com' with command:

> perl awstats.pl -output=pagetype -config=aws.qq.com

Press ENTER to finish...

修改配置文件 vi /opt/local/awstats/awstats.aws.qq.com.conf

LogFile 多台可用 logresolvemerge.pl 合并多个日志文件 配合通配符

LogFile="/opt/htdocs/upload/mxaws/logresolvemerge.pl /data/access_*.log |"

DirData 配置日志分析记录

DirData"/opt/local/awstats/data"

配置 更新按钮

AllowToUpdateStatsFromBrowser=0 修改为 AllowToUpdateStatsFromBrowser=1

配置脚本,然后生成awstats统计信息数据库

vi awstats_up.sh

#!/bin/sh

/opt/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=/opt/local/awstats/awstats.aws.qq.com.conf >/dev/null 2>&1

四、配置日志分析页面的来访ip的地址位置显示 (略)

这里我们用国内最准确的ip数据库——QQ纯真库我在博客附件里面放了这个工具，大家下载之后通过CRT用lrzsz工具传上去，具体步骤这里不写了。

附件里面有三个文件qqhostinfo.pm，qqwry.pl和QQWry.Dat，我们将这三个文件统统都放到/opt/local/awstats/wwwroot/cgi-bin/plugins中。

接下来，我们修改qqwry.pl文件，将./QQWry.Dat修改为${DIR}/plugins/QQWry.Dat

修改配置文件

将LoadPlugin="hostinfo"替换为LoadPlugin="qqhostinfo"即可。

五、配置nginx

server {

listen 80;

server_name aws.qq.com;

location / {

root /www/sunsky;

index index.html index.htm;

access_log /app/logs/www_access.log main;

}

location ~* ^/cgi-bin/.*\.pl$ {

root /usr/local/awstats/wwwroot;

fastcgi_pass unix:/usr/local/nginx/fastcgi_temp/perl_cgi-dispatch.sock;

fastcgi_index index.pl;

include fastcgi_params;

charset gb2312;

auth_basic "Restricted"; #有些网站不愿意公开网站流量信息，所以加个认证

auth_basic_user_file /usr/local/nginx/htpasswd.pass; #该文件由apache的加密认证工具htpasswd创建

}

location ~ ^/icon/ {

root /usr/local/awstats/wwwroot/;

index index.html;

access_log off;

error_log off;

charset gb2312;

}

}

为了保持nginx.conf主配置文件更加整洁干净，所以我们将fastcgi_param的一系列参数添加

vi /opt/local/nginx/conf/fastcgi_params

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

fastcgi_param QUERY_STRING $query_string;

fastcgi_param REQUEST_METHOD $request_method;

fastcgi_param CONTENT_TYPE $content_type;

fastcgi_param CONTENT_LENGTH $content_length;

fastcgi_param GATEWAY_INTERFACE CGI/1.1;

fastcgi_param SERVER_SOFTWARE nginx;

fastcgi_param SCRIPT_NAME $fastcgi_script_name;

fastcgi_param REQUEST_URI $request_uri;

fastcgi_param DOCUMENT_URI $document_uri;

fastcgi_param DOCUMENT_ROOT $document_root;

fastcgi_param SERVER_PROTOCOL $server_protocol;

fastcgi_param REMOTE_ADDR $remote_addr;

fastcgi_param REMOTE_PORT $remote_port;

fastcgi_param SERVER_ADDR $server_addr;

fastcgi_param SERVER_PORT $server_port;

fastcgi_param SERVER_NAME $server_name;

fastcgi_read_timeout 60;

针对上面的加密，由于nginx没有好的加密认证工具，需要借助apache的htpasswd来实现加密认证功能：

htpasswd -c -m /opt/local/nginx/htpasswd.pass jicki

无法查看icon 是~权限问题! chmod 授权访问目录

浏览器输入 http://myurl/cgi-bin/awstats.pl?config=myurl