Nginx: 99: Cannot assign requested address for upstream
2016-11-28 14:52
525 查看
A few days ago, I ran into the following interesting Nginx error message.
zhuanzi: https://ma.ttias.be/nginx-cannot-assign-requested-address-for-upstream/
[crit] 12889#0: *32401195 connect() to 127.0.0.1:80 failed (99: Cannot assign requested address) while connecting to upstreamMy configuration was very simple. This was an Nginx proxy that did all the SSL encryption and sent traffic to a Varnish instance, running on port :80 on localhost. The big takeway here is that it was a pretty high traffic Nginx proxy.Even with keepalive enabled in the nginx upstream, the error popped up. But what did it mean?
TCP ports and limits
It's good to know a thing or two about networking besides just servers once in a while. The problem occurred because the server couldn't get a free TCP port quickly enough to make the connection to127.0.0.1.
$ ss -s Total: 3130 (kernel 3431) TCP: 51582 (estab 2866, closed 48611, orphaned 92, synrecv 0, timewait 48611/0), ports 35279The ss tool gives you stats on the sockets/ports on the server. In this case, I had 51.582 TCP sessions in use (either active, closed, awaiting to be closed, ...).A normal server has around 28.000 possible TCP ports it can use to make a TCP connection to a remote (or local) system. Everything that talks via an IP address will pick a free port from this range to serve as source port for the outgoing connection. This port range is defined by the
ip_local_port_rangesysctl parameter.
$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 32768 61000The format is "minimum maximum" port. So 61000 – 32768 = 28 232 available source ports.An nginx SSL proxy that connects to a Varnish instance running on localhost will look like this in your
netstat.
$ netstat -anp | grep 127.0.0.1 ... tcp 0 0 127.0.0.1:37713 127.0.0.1:80 TIME_WAIT -The key takeaways here the source connection
127.0.0.1:37713that connects to its endpoint
127.0.0.1:80. For every source connection a new TCP source port is selected from the range in the
ip_local_port_rangeparameter.The combination of a source IP, source port, destination IP and destination IP needs to be unique. This is what's called a quadruplet in networking terms. You likely can't (easily) change the source IP. The source port is dynamically picked. That only leaves the destination IP and the destination port that are free to play with.
Solving the source port limitation
There are a couple of easy fixes. First, theip_local_port_rangecan be increased on a Linux machine (for more reading material, see increase ip_local_port_range TCP port range in Linux).
$ echo 15000 64000 > /proc/sys/net/ipv4/ip_local_port_rangeThis effectively increases the total port range from its default 28 232 ports to 49 000 ports.If that's not enough, you can add more destination IPs to connect to. Remember that each connection consists of the 4 parts (called quadruplets) with source IP and source port, destination IP and destination port. If you can't change the source port or IP, just change the destination IPs.Consider this kind of upstream definition in Nginx;
upstream varnish { server 127.0.0.1:80; }Such a definition can be used in your nginx configurations with the
proxy_passdirective.
server { listen 443; ... location / { proxy_pass http://varnish; ... } }Now if you know that each server usually has 2 IPs or more, it's very easy to add more quadruplets to your networking stack by adding an addition IP to your Nginx upstream. You've already added
127.0.0.1, but your server will have another IP (its public or DHCP IP) that you can safely add too, if your webserver binds to all ports.
upstream varnish { server 127.0.0.1:80; server 31.193.180.217:80; server 10.50.5.1:80; ... }Every IP you add in your upstream is effectively adding 49.000 local ports to your networking stack. You can even add non-routable local IPs to your server, as interface aliases, just to use as new destination IPs for your proxy configurations.Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. If you're interested in keeping up with me, have a look at my podcast and weekly newsletter below. For more updates, follow me on Twitter as @mattiasgeniar.
zhuanzi: https://ma.ttias.be/nginx-cannot-assign-requested-address-for-upstream/
相关文章推荐
- ipython notebook error: [Errno 99] Cannot assign requested address
- php连接memcahed出现Cannot assign requested address (99)的解决方法
- php连接memcahed出现Cannot assign requested address (99)的解决方法
- php连接memcahed出现Cannot assign requested address (99)的解决方法
- fastdfs errno: 99, error info: Cannot assign requested address
- time_wait太多导致[error] (99)Cannot assign requested address: proxy
- 服务器启动socket服务报错 java.net.BindException Cannot assign requested address
- tomcat启动报java.net.BindException: Cannot assign requested address
- Tomcat启动异常 java.net.BindException: Cannot assign requested address: JVM_Bind
- [tomcat] Cannot assign requested address: JVM_Bind 非端口占用冲突
- 关于错误“Cannot connect to the Citrix MetaFrame server.Can't assign requested address”的解决方法
- Cannot assign requested address 问题
- SIOCSIFFLAGS: Cannot assign requested address
- Cannot assign requested address: JVM_Bind
- Caused by: java.net.BindException: Cannot assign requested address: bind
- Cannot assign requested address: JVM_Bind
- Cannot assign requested address的解决办法
- Cannot assign requested address问题
- Tomcat Cannot assign requested address: JVM_Bind 非端口占用冲突
- Tomcat Cannot assign requested address: JVM_Bind 非端口占用冲突