网站无端卡死

开发了一个小型cms系统，其中有一个审核模块

每次审核到100条左右的时候，系统就会卡死，由于用的negix+tomcat

经常暴504错误，修改了negix配置，问题依然存在，困惑了一个月，今晚加班

想看看是不是c3p0的问题，于是开始逐条查看c3p0配置，果不其然

这条文章给了我很大帮助
http://blog.csdn.net/wozeze1/article/details/6037338 http://blog.csdn.net/kaishuaige/article/details/14451423

DatabaseConnectionLeak

使用C3P0连接池的连接泄露诊断方法

启用C3P0的连接归还超时设置

在C3P0的属性配置文件（0.datasource.config）中，设置以下属性：

debugUnreturnedConnectionStackTraces=true
unreturnedConnectionTimeout=1200

其中： debugUnreturnedConnectionStackTraces是一个开关，启用之后，对于每个从连接池拿出去的数据库连接，如果一段时间内没有归还，C3P0就会强制关闭这个连接，并将获取连接时的stack trace，以抛出异常的方式显示出来。 unreturnedConnectionTimeout设置多长时间会超时，以秒为单位。

假设系统中怀疑有连接泄露，那么，就可以启用这个功能，评估所有页面执行完成需要的最长时间，例如20分钟，然后将unreturnedConnectionTimeout设置为略大于这个时间。

设置C3P0的log输出

为了能看到C3P0抛出的异常信息，还需要设置C3P0的log属性。最简单的方式就是让C3P0使用JDK标准的log API，这样不需要加载额外的jar。在启动JVM时，设置启动参数： Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog

在Tomcat中，可修改catalina.sh，设置JAVA_OPTS变量包含上述内容。

可以做一个测试：设置C3P0属性debugUnreturnedConnectionStackTraces=true，unreturnedConnectionTimeout=1，然后访问一个运行时间超过1秒的页面。这样，就会在tomcat的输出窗口看到这样一段信息：

信息: Logging the stack trace by which the overdue resource was checked-out.
java.lang.Exception: DEBUG ONLY: Overdue resource check-out stack trace.
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:<span class="hl-number">506</span>)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection
(C3P0PooledConnectionPool.java:<span class="hl-number">525</span>)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection
(AbstractPoolBackedDataSource.java:<span class="hl-number">128</span>)
at aurora.database.service.SqlServiceContext.initConnection(SqlServiceContext.java:<span class="hl-number">168</span>)
at aurora.database.service.BusinessModelService.prepareForRun(BusinessModelService.java:<span class="hl-number">112</span>)
at aurora.database.service.BusinessModelService.query(BusinessModelService.java:<span class="hl-number">188</span>)
at aurora.database.actions.ModelQuery.doQuery(ModelQuery.java:<span class="hl-number">53</span>)
at aurora.database.actions.AbstractQueryAction.query(AbstractQueryAction.java:<span class="hl-number">104</span>)
at aurora.database.actions.AbstractQueryAction.run(AbstractQueryAction.java:<span class="hl-number">115</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">253</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">290</span>)
at uncertain.proc.Procedure.run(Procedure.java:<span class="hl-number">94</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">247</span>)
at aurora.application.features.AbstractProcedureInvoker.runProcedure(AbstractProcedureInvoker.java:<span class="hl-number">40</span>)
at aurora.application.features.InitProcedureInvoker.doInvoke(InitProcedureInvoker.java:<span class="hl-number">33</span>)
at aurora.application.features.InitProcedureInvoker.onCreateModel(InitProcedureInvoker.java:<span class="hl-number">37</span>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:<span class="hl-number">39</span>)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:<span class="hl-number">25</span>)
at java.lang.reflect.Method.invoke(Method.java:<span class="hl-number">597</span>)
at uncertain.proc.ReflectionMethodHandle.handleEvent(ReflectionMethodHandle.java:<span class="hl-number">93</span>)
at uncertain.proc.ReflectionMethodHandle.handleEvent(ReflectionMethodHandle.java:<span class="hl-number">50</span>)
at uncertain.event.Configuration.fireEventInternal(Configuration.java:<span class="hl-number">403</span>)
at uncertain.event.Configuration.fireEvent(Configuration.java:<span class="hl-number">341</span>)
at uncertain.proc.ProcedureRunner.fireEvent(ProcedureRunner.java:<span class="hl-number">312</span>)
at uncertain.proc.Action.run(Action.java:<span class="hl-number">171</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">253</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">290</span>)
at uncertain.proc.Procedure.run(Procedure.java:<span class="hl-number">94</span>)
at uncertain.proc.Switch.run(Switch.java:<span class="hl-number">87</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">253</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">290</span>)
at uncertain.proc.Procedure.run(Procedure.java:<span class="hl-number">94</span>)
at uncertain.proc.ProcedureRunner.run(ProcedureRunner.java:<span class="hl-number">247</span>)
at aurora.service.ServiceInstance.invoke(ServiceInstance.java:<span class="hl-number">124</span>)
at aurora.service.http.AbstractFacadeServlet.service(AbstractFacadeServlet.java:<span class="hl-number">116</span>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:<span class="hl-number">722</span>)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:<span class="hl-number">306</span>)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:<span class="hl-number">210</span>)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:<span class="hl-number">240</span>)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:<span class="hl-number">161</span>)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:<span class="hl-number">164</span>)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:<span class="hl-number">100</span>)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:<span class="hl-number">541</span>)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:<span class="hl-number">118</span>)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:<span class="hl-number">383</span>)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:<span class="hl-number">243</span>)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:<span class="hl-number">188</span>)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:<span class="hl-number">166</span>)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:<span class="hl-number">288</span>)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:<span class="hl-number">886</span>)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:<span class="hl-number">908</span>)
at java.lang.Thread.run(Thread.java:<span class="hl-number">680</span>)

这就是C3P0发现连接1秒钟后没有归还，强制关闭连接后抛出的信息，从stack trace中可以看出，获取连接是发生在aurora.database.service.BusinessModelService.query，也就是调用BM的query。

由于页面确实是需要使用连接超过1秒，正在使用的连接被强制关闭以后，还会看到这样的信息：

<span class="hl-number">011</span>-<span class="hl-number">6</span>-<span class="hl-number">22</span> <span class="hl-number">13</span>:<span class="hl-number">24</span>:<span class="hl-number">15</span> com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
警告: [c3p0] A PooledConnection that has already signalled a Connection error is still in use!

后续SQL操作再使用这个连接时，会发现连接已经关闭，于是就会抛出类似这样的异常：

<span class="hl-number">2011</span>-<span class="hl-number">6</span>-<span class="hl-number">22</span> <span class="hl-number">13</span>:<span class="hl-number">24</span>:<span class="hl-number">15</span> com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
警告: [c3p0] Another error has occurred [ java.sql.SQLException: 关闭的连接 ]
which will not be reported to listeners!
java.sql.SQLException: 关闭的连接
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:<span class="hl-number">134</span>)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:<span class="hl-number">179</span>)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:<span class="hl-number">269</span>)
at oracle.jdbc.driver.OracleConnection.rollback(OracleConnection.java:<span class="hl-number">1439</span>)
at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:<span class="hl-number">855</span>)
at aurora.transaction.UserTransactionImpl.rollback(UserTransactionImpl.java:<span class="hl-number">58</span>)
at aurora.service.http.AbstractFacadeServlet.service(AbstractFacadeServlet.java:<span class="hl-number">141</span>)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:<span class="hl-number">722</span>)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:<span class="hl-number">306</span>)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:<span class="hl-number">210</span>)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:<span class="hl-number">240</span>)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:<span class="hl-number">161</span>)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:<span class="hl-number">164</span>)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:<span class="hl-number">100</span>)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:<span class="hl-number">541</span>)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:<span class="hl-number">118</span>)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:<span class="hl-number">383</span>)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:<span class="hl-number">243</span>)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:<span class="hl-number">188</span>)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:<span class="hl-number">166</span>)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:<span class="hl-number">288</span>)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:<span class="hl-number">886</span>)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:<span class="hl-number">908</span>)
at java.lang.Thread.run(Thread.java:<span class="hl-number">680</span>)