HttpClient3.1 未releaseConnection导致应用服务器宕机

报警短信突然一条接一条的出现了，某台服务器宕机了。赶紧ssh到问题服务器，jstack把栈信息导出分析。此时服务器cpu正常，但是resin无响应。初步判断是线程死锁或者线程堵死的问题。
首先我们来分析jstack栈信息。首先应该查找Waiting状态的线程,发现有1000多个线程，我晕。。
分析其中的一个，部分栈信息如下，不过足以说明问题了：

"http--8005-1040$786011311" daemon prio=10 tid=0x00002aab44a25800 nid=0x7d71 in Object.wait() [0x000000004b097000..0x000000004b09ac90]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <0x00002aaac21d8bc0> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)

at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)

- locked <0x00002aaac21d8bc0> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)

at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)

at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)

at com.sohu.twap.service.util.HttpUtil.get(HttpUtil.java:442)

at com.sohu.twap.service.util.HttpUtil.getOriginalContent(HttpUtil.java:462)

at com.sohu.twap.action.AbstractServiceAction.getJsSourceCode(AbstractServiceAction.java:879)

程序在MultiThreadedHttpConnectionManager.doGetConnection方法中的object.wait() 方法中等待了。。。查看代码和google，发现应该是connection连接未释放的问题的。程序的其他地方已经升级到4.0。所以建议大家还是使用最新的4.1版本。参考其官方文档：

Release the Connection

This is a crucial step to keep things flowing. We must tell HttpClient that we are done with the connection and that it can now be reused. Without doing this HttpClient will
wait indefinitely for a connection to free up so that it can be reused.

method.releaseConnection();

如果和我们一样使用了MultiThreadedHttpConnectionManager的话，

One main side effect of connection management is that connections must be manually released when no longer used. This is due to the fact that HttpClient cannot
determine when a method is no longer using its connection. This occurs because a method's response body is not read directly by HttpClient, but by the application using HttpClient. When the response is read it must obviously make use of the method's connection.
Thus, a connection cannot be released from a method until the method's response body is read which is after HttpClient finishes executing the method. The application therefore must manually release the connection by calling releaseConnection() on the method
after the response body has been read. To safely ensure connection release HttpClient should be used in the following manner:

MultiThreadedHttpConnectionManager connectionManager = 
                new MultiThreadedHttpConnectionManager();
        HttpClient client = new HttpClient(connectionManager);
                        ...
        // and then from inside some thread executing a method
        GetMethod get = new GetMethod("http://httpcomponents.apache.org/");
        try {
            client.executeMethod(get);
            // print response to stdout
            System.out.println(get.getResponseBodyAsStream());
        } finally {
            // be sure the connection is released back to the connection 
            // manager
            get.releaseConnection();
        }

Particularly, notice that the connection is released regardless of what the result of executing the method was or whether or not an exception was thrown. For every call to HttpClient.executeMethod there must be a matching
call to method.releaseConnection().

另外一个关于这个问题的email讨论
http://markmail.org/thread/ygejgcnp5kz6f7ox#query:

关于其参数说明：

maxConnectionsPerHost	The maximum number of connections that will be created for any particular HostConfiguration. Defaults to 2.
maxTotalConnections	The maximum number of active connections. Defaults to 20.

下面是我们程序的参数：

httpClientManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = httpClientManager .getParams();
params.setStaleCheckingEnabled( true );
params.setMaxTotalConnections(PropConfig.getInstance().getInt( "http.max.total.connections" ,
320));
params.setDefaultMaxConnectionsPerHost(PropConfig.getInstance().getInt( "http.max.connection.perhost" ,
160));
params.setConnectionTimeout(PropConfig.getInstance().getInt( "http.connection.timeout" ,
1000));
params.setSoTimeout(PropConfig.getInstance().getInt( "http.socket.timeout" ,
1000));

httpClient = new HttpClient(httpClientManager );
默认MaxConnectionsPerHost是2，对于SOA应用的话，这样就限制的对一台服务器调用只用两个并发，注意一定要更改，其实httpclient发表于2003年，作者没有意识到现在SOA架构的广泛使用，况且httpclient原来的目的是实现浏览器的连接部分，设置默认值为2也是理所当然的。

关于超时和maxconenectionperhost的讨论：
http://www.scriptkiddie.org/blog/2010/09/06/why-i-hate-java-httpclient-maxconnectionsperhost/