nginx+awstats 统计web服务器log 推荐
2008-09-07 14:11
483 查看
nginx+perl+fastcgi+awstats
因为不再用apache，迁移到了nginx
默认编译安装的nginx对cgi的支持并不好(所以在编译的时候一般都没打开这个功能)
google了一把。大家都用fastcgi来支持。就照搬吧
http://wiki.codemongers.com/NginxSimpleCGI
wiki中有一篇，貌似这个是原创，哈哈
很多人懒得看英文，我就翻译一下吧。
如果你没有cgi的文件，可以自己建立一个简单的。
只需要如下三行代码就可以了。
#vi index.cgi
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "<html><body>Hello, world.</body></html>";
放到web根目录下的cgi-bin目录里面，记得要给它执行权限
接下来是用perl写一个daemon程序来处理cgi文件，幸运的是源代码已经有了。嘿嘿
#vi cgiwrap-fcgi.pl
#!/usr/bin/perl -w
use FCGI;
use Socket;
use FCGI::ProcManager;
sub shutdown { FCGI::CloseSocket($socket); exit; }
sub restart { FCGI::CloseSocket($socket); &main; }
use sigtrap 'handler', \&shutdown, 'normal-signals';
use sigtrap 'handler', \&restart, 'HUP';
require 'syscall.ph';
use POSIX qw(setsid);
#&daemonize; we don't daemonize when running under runsv
#this keeps the program alive or something after exec'ing perl scripts
END() { }
BEGIN() { }
{
no warnings;
*CORE::GLOBAL::exit = sub { die "fakeexit\nrc=" . shift() . "\n"; };
};
eval q{exit};
if ($@) {
exit unless $@ =~ /^fakeexit/;
}
&main;
sub daemonize() {
chdir '/' or die "Can't chdir to /: $!";
defined( my $pid = fork ) or die "Can't fork: $!";
exit if $pid;
setsid() or die "Can't start a new session: $!";
umask 0;
}
sub main {
#$socket = FCGI::OpenSocket( "127.0.0.1:8999", 10 ); #use IP sockets
#$socket = FCGI::OpenSocket( "/var/run/nginx/perl_cgi-dispatch.sock", 10 ); #use UNIX sockets - user running this script must have w access to the 'nginx' folder!!
#foreach $item (keys %ENV) { delete $ENV{$item}; }
$proc_manager = FCGI::ProcManager->new( {n_processes => 5} );
$socket = FCGI::OpenSocket( "/opt/nginx/fcgi/cgi.sock", 10 )
; #use UNIX sockets - user running this script must have w access to the 'nginx' folder!!
$request =
FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%req_params, $socket,
&FCGI::FAIL_ACCEPT_ON_INTR );
$proc_manager->pm_manage();
if ($request) { request_loop() }
FCGI::CloseSocket($socket);
}
sub request_loop {
while ( $request->Accept() >= 0 ) {
$proc_manager->pm_pre_dispatch();
#processing any STDIN input from WebServer (for CGI-POST actions)
$stdin_passthrough = '';
{ no warnings; $req_len = 0 + $req_params{'CONTENT_LENGTH'}; };
if ( ( $req_params{'REQUEST_METHOD'} eq 'POST' ) && ( $req_len != 0 ) )
{
my $bytes_read = 0;
while ( $bytes_read < $req_len ) {
my $data = '';
my $bytes = read( STDIN, $data, ( $req_len - $bytes_read ) );
last if ( $bytes == 0 || !defined($bytes) );
$stdin_passthrough .= $data;
$bytes_read += $bytes;
}
}
#running the cgi app
if (
( -x $req_params{SCRIPT_FILENAME} ) && #can I execute this?
( -s $req_params{SCRIPT_FILENAME} ) && #Is this file empty?
( -r $req_params{SCRIPT_FILENAME} ) #can I read this file?
)
{
pipe( CHILD_RD, PARENT_WR );
pipe( PARENT_ERR, CHILD_ERR );
my $pid = open( CHILD_O, "-|" );
unless ( defined($pid) ) {
print("Content-type: text/plain\r\n\r\n");
print
"Error: CGI app returned no output - Executing $req_params{SCRIPT_FILENAME} failed !\n";
next;
}
$oldfh = select(PARENT_ERR);
$| = 1;
select(CHILD_O);
$| = 1;
select($oldfh);
if ( $pid > 0 ) {
close(CHILD_RD);
close(CHILD_ERR);
print PARENT_WR $stdin_passthrough;
close(PARENT_WR);
$rin = $rout = $ein = $eout = '';
vec( $rin, fileno(CHILD_O), 1 ) = 1;
vec( $rin, fileno(PARENT_ERR), 1 ) = 1;
$ein = $rin;
$nfound = 0;
while ( $nfound =
select( $rout = $rin, undef, $ein = $eout, 10 ) )
{
die "$!" unless $nfound != -1;
$r1 = vec( $rout, fileno(PARENT_ERR), 1 ) == 1;
$r2 = vec( $rout, fileno(CHILD_O), 1 ) == 1;
$e1 = vec( $eout, fileno(PARENT_ERR), 1 ) == 1;
$e2 = vec( $eout, fileno(CHILD_O), 1 ) == 1;
if ($r1) {
while ( $bytes = read( PARENT_ERR, $errbytes, 4096 ) ) {
print STDERR $errbytes;
}
if ($!) {
$err = $!;
die $!;
vec( $rin, fileno(PARENT_ERR), 1 ) = 0
unless ( $err == EINTR or $err == EAGAIN );
}
}
if ($r2) {
while ( $bytes = read( CHILD_O, $s, 4096 ) ) {
print $s;
}
if ( !defined($bytes) ) {
$err = $!;
die $!;
vec( $rin, fileno(CHILD_O), 1 ) = 0
unless ( $err == EINTR or $err == EAGAIN );
}
}
last if ( $e1 || $e2 );
}
close CHILD_RD;
close PARENT_ERR;
waitpid( $pid, 0 );
} else {
foreach $key ( keys %req_params ) {
$ENV{$key} = $req_params{$key};
}
# cd to the script's local directory
if ( $req_params{SCRIPT_FILENAME} =~ /^(.*)\/[^\/]+$/ ) {
chdir $1;
}
close(PARENT_WR);
#close(PARENT_ERR);
close(STDIN);
close(STDERR);
#fcntl(CHILD_RD, F_DUPFD, 0);
syscall( &SYS_dup2, fileno(CHILD_RD), 0 );
syscall( &SYS_dup2, fileno(CHILD_ERR), 2 );
#open(STDIN, "<&CHILD_RD");
exec( $req_params{SCRIPT_FILENAME} );
die("exec failed");
}
} else {
print("Content-type: text/plain\r\n\r\n");
print
"Error: No such CGI app - $req_params{SCRIPT_FILENAME} may not exist or is not executable by this process.\n";
}
}
}
这个文件只需要修改一个地方/opt/nginx/fcgi/cgi.sock，改到你想放的地方，比如/var/run/fcgi.sock 注意要有权限，以便nginix可以正常访问它，如果不知道怎么配置权限
就chmod 777 好了（汗，一个很烂的习惯），正确的做法是将该文件的属主和数组设为nginx的运行用户，
比如nginx用nobody用户运行，则 chown nobody:nobody /var/run/cgi.sock 注意，上层目录也要有执行权限，这样才能访问这个unix socket（访问不到目录，自然不能访问文件了）
这些都做好以后可以启动这个文件了，
#chmod 755 cgiwrap-fcgi.pl
#./cgiwrap-fcgi.pl &
再后台运行。不过这该死的程序居然在终端打印日志，一会调试成功了，可以用
./cgiwrap-fcgi.pl >/dev/null 2>&1 &
这样世界就清净了
如果想保留日志，自己改perl代码吧，不是很难，加上输出文件句柄就可以了。保存到文件中。嘿嘿。
接下来就修改nginx.conf
location ~ ^/cgi-bin/.*\.cgi$ {
gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
fastcgi_pass unix:/var/run/nginx/cgi.sock;
fastcgi_index index.cgi;
fastcgi_param SCRIPT_FILENAME /var/www/cgi-bin$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
下面这么多其实可以用一个文件来包含起来，不用这么多，看的心乱
location ~ ^/cgi-bin/.*\.cgi$ {
gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
fastcgi_pass unix:/var/run/nginx/cgi.sock;
fastcgi_index index.cgi;
include fastcgi_params;
}
ok,接下来重启nginx
kill -HUP `cat /usr/local/nginx/nginx.pid`
然后我们访问刚才的那个cgi文件，http://XXXXXXXXXX/cgi-bin/index.cgi 如果看到hello world ，就说明成功了。嘿嘿，就这么简单。
awstats默认的配置中只有IIS和apache的LOG格式，需要我们自定义一下
在/etc/awstats/awstats.mysite.conf里设置log格式如下(默认值是1):
LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %otherquot"
在nginx中也配置好log格式
log_format wwwlogs '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for';
access_log /var/log/nginx/blog_log.log wwwlogs;
再设置log的轮转
3)新建一个日志回滚,vi /etc/logrotate.d/nginx (路径根据操作系统不同而变化)
内容如下:
/var/log/nginx {
daily
missingok
rotate 7
compress
delaycompress notifempty
create 644 nginx root sharedscripts
prerotate
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl --config=www.mysite.com -update
endscript
postrotate
if [ -f /usr/local/nginx/nginx.pid ]; then
kill -USR1 `cat /usr/local/nginx/nginx.pid`
fi
endscript
}
因为不再用apache，迁移到了nginx
默认编译安装的nginx对cgi的支持并不好(所以在编译的时候一般都没打开这个功能)
google了一把。大家都用fastcgi来支持。就照搬吧
http://wiki.codemongers.com/NginxSimpleCGI
wiki中有一篇，貌似这个是原创，哈哈
很多人懒得看英文，我就翻译一下吧。
如果你没有cgi的文件，可以自己建立一个简单的。
只需要如下三行代码就可以了。
#vi index.cgi
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "<html><body>Hello, world.</body></html>";
放到web根目录下的cgi-bin目录里面，记得要给它执行权限
接下来是用perl写一个daemon程序来处理cgi文件，幸运的是源代码已经有了。嘿嘿
#vi cgiwrap-fcgi.pl
#!/usr/bin/perl -w
use FCGI;
use Socket;
use FCGI::ProcManager;
sub shutdown { FCGI::CloseSocket($socket); exit; }
sub restart { FCGI::CloseSocket($socket); &main; }
use sigtrap 'handler', \&shutdown, 'normal-signals';
use sigtrap 'handler', \&restart, 'HUP';
require 'syscall.ph';
use POSIX qw(setsid);
#&daemonize; we don't daemonize when running under runsv
#this keeps the program alive or something after exec'ing perl scripts
END() { }
BEGIN() { }
{
no warnings;
*CORE::GLOBAL::exit = sub { die "fakeexit\nrc=" . shift() . "\n"; };
};
eval q{exit};
if ($@) {
exit unless $@ =~ /^fakeexit/;
}
&main;
sub daemonize() {
chdir '/' or die "Can't chdir to /: $!";
defined( my $pid = fork ) or die "Can't fork: $!";
exit if $pid;
setsid() or die "Can't start a new session: $!";
umask 0;
}
sub main {
#$socket = FCGI::OpenSocket( "127.0.0.1:8999", 10 ); #use IP sockets
#$socket = FCGI::OpenSocket( "/var/run/nginx/perl_cgi-dispatch.sock", 10 ); #use UNIX sockets - user running this script must have w access to the 'nginx' folder!!
#foreach $item (keys %ENV) { delete $ENV{$item}; }
$proc_manager = FCGI::ProcManager->new( {n_processes => 5} );
$socket = FCGI::OpenSocket( "/opt/nginx/fcgi/cgi.sock", 10 )
; #use UNIX sockets - user running this script must have w access to the 'nginx' folder!!
$request =
FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%req_params, $socket,
&FCGI::FAIL_ACCEPT_ON_INTR );
$proc_manager->pm_manage();
if ($request) { request_loop() }
FCGI::CloseSocket($socket);
}
sub request_loop {
while ( $request->Accept() >= 0 ) {
$proc_manager->pm_pre_dispatch();
#processing any STDIN input from WebServer (for CGI-POST actions)
$stdin_passthrough = '';
{ no warnings; $req_len = 0 + $req_params{'CONTENT_LENGTH'}; };
if ( ( $req_params{'REQUEST_METHOD'} eq 'POST' ) && ( $req_len != 0 ) )
{
my $bytes_read = 0;
while ( $bytes_read < $req_len ) {
my $data = '';
my $bytes = read( STDIN, $data, ( $req_len - $bytes_read ) );
last if ( $bytes == 0 || !defined($bytes) );
$stdin_passthrough .= $data;
$bytes_read += $bytes;
}
}
#running the cgi app
if (
( -x $req_params{SCRIPT_FILENAME} ) && #can I execute this?
( -s $req_params{SCRIPT_FILENAME} ) && #Is this file empty?
( -r $req_params{SCRIPT_FILENAME} ) #can I read this file?
)
{
pipe( CHILD_RD, PARENT_WR );
pipe( PARENT_ERR, CHILD_ERR );
my $pid = open( CHILD_O, "-|" );
unless ( defined($pid) ) {
print("Content-type: text/plain\r\n\r\n");
"Error: CGI app returned no output - Executing $req_params{SCRIPT_FILENAME} failed !\n";
next;
}
$oldfh = select(PARENT_ERR);
$| = 1;
select(CHILD_O);
$| = 1;
select($oldfh);
if ( $pid > 0 ) {
close(CHILD_RD);
close(CHILD_ERR);
print PARENT_WR $stdin_passthrough;
close(PARENT_WR);
$rin = $rout = $ein = $eout = '';
vec( $rin, fileno(CHILD_O), 1 ) = 1;
vec( $rin, fileno(PARENT_ERR), 1 ) = 1;
$ein = $rin;
$nfound = 0;
while ( $nfound =
select( $rout = $rin, undef, $ein = $eout, 10 ) )
{
die "$!" unless $nfound != -1;
$r1 = vec( $rout, fileno(PARENT_ERR), 1 ) == 1;
$r2 = vec( $rout, fileno(CHILD_O), 1 ) == 1;
$e1 = vec( $eout, fileno(PARENT_ERR), 1 ) == 1;
$e2 = vec( $eout, fileno(CHILD_O), 1 ) == 1;
if ($r1) {
while ( $bytes = read( PARENT_ERR, $errbytes, 4096 ) ) {
print STDERR $errbytes;
}
if ($!) {
$err = $!;
die $!;
vec( $rin, fileno(PARENT_ERR), 1 ) = 0
unless ( $err == EINTR or $err == EAGAIN );
}
}
if ($r2) {
while ( $bytes = read( CHILD_O, $s, 4096 ) ) {
print $s;
}
if ( !defined($bytes) ) {
$err = $!;
die $!;
vec( $rin, fileno(CHILD_O), 1 ) = 0
unless ( $err == EINTR or $err == EAGAIN );
}
}
last if ( $e1 || $e2 );
}
close CHILD_RD;
close PARENT_ERR;
waitpid( $pid, 0 );
} else {
foreach $key ( keys %req_params ) {
$ENV{$key} = $req_params{$key};
}
# cd to the script's local directory
if ( $req_params{SCRIPT_FILENAME} =~ /^(.*)\/[^\/]+$/ ) {
chdir $1;
}
close(PARENT_WR);
#close(PARENT_ERR);
close(STDIN);
close(STDERR);
#fcntl(CHILD_RD, F_DUPFD, 0);
syscall( &SYS_dup2, fileno(CHILD_RD), 0 );
syscall( &SYS_dup2, fileno(CHILD_ERR), 2 );
#open(STDIN, "<&CHILD_RD");
exec( $req_params{SCRIPT_FILENAME} );
die("exec failed");
}
} else {
print("Content-type: text/plain\r\n\r\n");
"Error: No such CGI app - $req_params{SCRIPT_FILENAME} may not exist or is not executable by this process.\n";
}
}
}
这个文件只需要修改一个地方/opt/nginx/fcgi/cgi.sock，改到你想放的地方，比如/var/run/fcgi.sock 注意要有权限，以便nginix可以正常访问它，如果不知道怎么配置权限
就chmod 777 好了（汗，一个很烂的习惯），正确的做法是将该文件的属主和数组设为nginx的运行用户，
比如nginx用nobody用户运行，则 chown nobody:nobody /var/run/cgi.sock 注意，上层目录也要有执行权限，这样才能访问这个unix socket（访问不到目录，自然不能访问文件了）
这些都做好以后可以启动这个文件了，
#chmod 755 cgiwrap-fcgi.pl
#./cgiwrap-fcgi.pl &
再后台运行。不过这该死的程序居然在终端打印日志，一会调试成功了，可以用
./cgiwrap-fcgi.pl >/dev/null 2>&1 &
这样世界就清净了
如果想保留日志，自己改perl代码吧，不是很难，加上输出文件句柄就可以了。保存到文件中。嘿嘿。
接下来就修改nginx.conf
location ~ ^/cgi-bin/.*\.cgi$ {
gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
fastcgi_pass unix:/var/run/nginx/cgi.sock;
fastcgi_index index.cgi;
fastcgi_param SCRIPT_FILENAME /var/www/cgi-bin$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
下面这么多其实可以用一个文件来包含起来，不用这么多，看的心乱
location ~ ^/cgi-bin/.*\.cgi$ {
gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
fastcgi_pass unix:/var/run/nginx/cgi.sock;
fastcgi_index index.cgi;
include fastcgi_params;
}
ok,接下来重启nginx
kill -HUP `cat /usr/local/nginx/nginx.pid`
然后我们访问刚才的那个cgi文件，http://XXXXXXXXXX/cgi-bin/index.cgi 如果看到hello world ，就说明成功了。嘿嘿，就这么简单。
awstats默认的配置中只有IIS和apache的LOG格式，需要我们自定义一下
在/etc/awstats/awstats.mysite.conf里设置log格式如下(默认值是1):
LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %otherquot"
在nginx中也配置好log格式
log_format wwwlogs '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for';
access_log /var/log/nginx/blog_log.log wwwlogs;
再设置log的轮转
3)新建一个日志回滚,vi /etc/logrotate.d/nginx (路径根据操作系统不同而变化)
内容如下:
/var/log/nginx {
daily
missingok
rotate 7
compress
delaycompress notifempty
create 644 nginx root sharedscripts
prerotate
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl --config=www.mysite.com -update
endscript
postrotate
if [ -f /usr/local/nginx/nginx.pid ]; then
kill -USR1 `cat /usr/local/nginx/nginx.pid`
fi
endscript
}
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Nginx+awstats统计web服务器log(延续版)
- Nginx+awstats统计web服务器log
- awstats 线上web服务器访问统计
- WEB服务器-Nginx之虚拟主机、日志、认证及优化 推荐
- centos6.4下DNS+squid+nginx+mysql搭建高可用web服务器 推荐
- Shell+AWStats+ELK分析Nginx的Log 推荐
- keepalived架设简单高可用的nginx的web服务器 ----那些你不知道的秘密 推荐
- 轻松使用Nginx搭建web服务器 推荐
- 构建基于Nginx的web服务器 推荐
- 基于红帽6.4（64位系统）web服务器的又一利器Nginx 推荐
- Nginx搭建部署Web服务器并与NFS结合搭建负载均衡服务器 推荐
- Nginx实战基础篇一 源码包编译安装部署web服务器 推荐
- 三大WEB服务器对比分析（apache ,lighttpd,nginx）
- 高性能Web服务器Nginx的配置与部署研究（14）平滑升级你的Nginx
- ELK日志服务器的快速搭建并收集nginx日志 推荐
- 20个Nginx Web服务器最佳安全实践(1)
- Web服务器Nginx多方位优化策略
- Linux+Nginx+Php架设高性能WEB服务器
- Nginx是什么，有什么优点？为什么选择Nginx做web服务器软件?（经典经典）
- 三大WEB服务器对比分析（apache ,lighttpd,nginx）