nginx运行中出现大量的Too many open files异常

现象描述:

通过nginx代理的请求返回/50X.html页面,通过查看nginx日志(/usr/local/nginx/log/error.log),发现大量Too many open files异常,

2020/04/30 10:15:42 [alert] 2794#0: *188092 socket() failed (24: Too many open files) while connecting to upstream, client: 118.114.245.36, server: intest.pzhsteelmobile.cn, request: "GET / HTTP/1.1", ups
tream: "http://10.*.*.*:16024/", host: "intest.....cn"
2020/04/30 10:15:42 [crit] 2794#0: *188092 open() "/usr/local/nginx/html/50x.html" failed (24: Too many open files), client: 118.114.245.36, server: intest.....cn, request: "GET / HTTP/1.1", up
stream: "http://10.*.*.*:16024/", host: "intest.....cn"
2020/04/30 10:15:42 [alert] 2794#0: *188229 socket() failed (24: Too many open files) while connecting to upstream, client: 118.114.245.36, server: intest.....cn, request: "GET / HTTP/1.1", ups
tream: "http://10.*.*.*:16024/", host: "intest.....cn"
2020/04/30 10:15:42 [crit] 2794#0: *188229 open() "/usr/local/nginx/html/50x.html" failed (24: Too many open files), client: 118.114.245.36, server: intest.....cn, request: "GET / HTTP/1.1", up
stream: "http://10.*.*.*:16024/", host: "intest.pzhsteelmobile.cn"
2020/04/30 10:15:43 [alert] 2794#0: *188635 socket() failed (24: Too many open files) while connecting to upstream, client: 118.114.245.36, server: intest.....cn, request: "GET / HTTP/1.1", ups
tream: "http://10.*.*.*:16024/", host: "intest.....cn"

查看当前的端口占用情况:

[root@fxdl1 logs]#  netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a,S[a]}'
ESTABLISHED 1104
FIN_WAIT1 3
FIN_WAIT2 6
TIME_WAIT 42

发现端口占用数始终在一千左右徘徊,因此判断该异常是nginx并发数设置过小,当前的访问量耗尽允许使用的并发连接数.

处理方法

增加系统最大连接数,同时允许nginx使用更多的连接
此次修改将系统最大连接数设置为65535, nginx最大连接数设置为30000

处理步骤

1. 增加系统文件打开数限制
   # 所有用户软件最大文件数
   * soft nofile 65535
   # 所有用户硬件最大文件数
   * hard nofile 65535
2. 重启系统生效
3. 增加nginx最大文件数限制
   修改/usr/local/nginx/conf/nginx.conf文件,添加
   worker_rlimit_nofile 30000;
4. /usr/local/nginx/sbin/nginx -s reload加载nginx配置

处理后压力测试

同时并发4000个http请求,查询后台建立连接数为3242,同时查看/usr/local/nginx/log/error.log,无新增异常

[root@fxdl1 logs]#  netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a,S[a]}'
LAST_ACK 161
SYN_RECV 255
ESTABLISHED 3242
FIN_WAIT2 43
CLOSING 15
TIME_WAIT 9584