MySQL5.7 Attempted to use a closed or broken resource pool 处理过程

1，tomcat启动报错不能获取JDBC连接

[b]mysql版本：5.6.10
tomcat版本：7.0
jdk版本：1.7.0_55

开发人员强调说是以前经历过重启下mysql数据库就好了，不过我不太相信，因为我通过sqlyog客户端连接数据库完全正常的，而且我使用以下生成的kill命令清除了所有数据库连接，然后再重启7100的tomcat，也没有发现任何tomcat发起的connection连接过来，所以我怀疑不是mysql数据库的问题：

SELECT CONCAT('kill ',t.`ID`,';') FROM information_schema.`PROCESSLIST` t WHERE t.`HOST` NOT IN('localhost') AND t.`COMMAND` IN ('Sleep');

tomcat启动详细日志如下：

2016-07-11 19:57:23,378 [main:INFO] com.google.code.yanf4j.core.impl.AbstractController - Controller has been stopped. 2016-07-11 19:57:23,381 [main:ERROR] org.springframework.web.context.ContextLoader - Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityFilter': Injection of autowired dependencies failed; nested exception is org.springframew ork.beans.factory.BeanCreationException: Could not autowire field: private com.plocc.auth.common.CommonSecurityMetadataSource com.plocc.auth.common.CommonSecurityFilter.securityMetadataSource ; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init method failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:287) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:913) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:384) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:283) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.plocc.auth.common.CommonSecurityMetadataSource com.plocc.auth.common.CommonSecurityFi lter.securityMetadataSource; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init meth od failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoke d by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException wa s provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:506) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284) ... 37 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init method failed; nested exception is org.spr ingframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: com.mchang e.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: c om.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:394) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1448) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:848) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:790) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:707) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:478) ... 39 more Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following f ailure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the fo llowing failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:195) at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:293) at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:287) at com.plocc.auth.dao.impl.AuthResourceDAOImpl.selectResourceAll(AuthResourceDAOImpl.java:147) at com.plocc.auth.dao.impl.AuthResourceDAOImpl$$FastClassByCGLIB$$14a1e188.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) at com.plocc.auth.dao.impl.AuthResourceDAOImpl$$EnhancerByCGLIB$$dad3756c.selectResourceAll(<generated>) at com.plocc.auth.service.impl.AuthenticationServiceImpl.getAuthResourceAll(AuthenticationServiceImpl.java:456) at sun.reflect.GeneratedMethodAccessor2222.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy79.getAuthResourceAll(Unknown Source) at sun.reflect.GeneratedMethodAccessor2222.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:306) at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:221) at org.springframework.remoting.caucho.HessianExporter.doInvoke(HessianExporter.java:198) at org.springframework.remoting.caucho.HessianExporter.invoke(HessianExporter.java:118) at org.springframework.remoting.caucho.HessianServiceExporter.handleRequest(HessianServiceExporter.java:66) at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) at javax.servlet.http.HttpServlet.service(HttpServlet.java:643) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65) at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:62) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:531) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:182) ... 49 more Caused by: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool at com.mchange.v2.resourcepool.BasicResourcePool.ensureNotBroken(BasicResourcePool.java:1632) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:524) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525) ... 52 more 七月 11, 2016 7:57:23 下午 org.apache.catalina.core.StandardContext start 严重: Error listenerStart 七月 11, 2016 7:57:23 下午 org.apache.catalina.core.StandardContext start 严重: Context [/PLATFORM_SSO_Web] startup failed due to previous errors 七月 11, 2016 7:57:23 下午 org.apache.catalina.startup.HostConfig deployDirectory 信息: Deploying web application directory test 七月 11, 2016 7:57:23 下午 org.apache.coyote.http11.Http11Protocol start 信息: Starting Coyote HTTP/1.1 on http-7100 七月 11, 2016 7:57:23 下午 org.apache.jk.common.ChannelSocket init 信息: JK: ajp13 listening on /0.0.0.0:7109 七月 11, 2016 7:57:23 下午 org.apache.jk.server.JkMain start 信息: Jk running ID=0 time=0/29 config=null 七月 11, 2016 7:57:23 下午 org.apache.catalina.startup.Catalina start 信息: Server startup in 3616 ms

2，检查工程spring的数据库配置文件

Jdbc配置文件check后，都正常，有相应的maxIdleTime等设置，c3p0正常，如下：

vim ./webapps/PLATFORM_SSO_Web/WEB-INF/classes/META-INF/spring/db/jdbc/applicationContext_jdbc.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd"> <beans> <!-- C3P0连接池配置--> <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close"> <property name="driverClass"> <value>${sso.datasource.driverClassName}</value> </property> <property name="jdbcUrl"> <value>${sso.datasource.url}</value> </property> <property name="user"> <value>${sso.datasource.username}</value> </property> <property name="password"> <value>${sso.datasource.password}</value> </property> <!--连接池中保留的最小连接数。--> <property name="minPoolSize"> <value>${sso.datasource.minPoolSize}</value> </property> <!--连接池中保留的最大连接数。Default: 15 --> <property name="maxPoolSize"> <value>${sso.datasource.maxPoolSize}</value> </property> <!--初始化时获取的连接数，取值应在minPoolSize与maxPoolSize之间。Default: 3 --> <property name="initialPoolSize"> <value>${sso.datasource.initialPoolSize}</value> </property> <!--最大空闲时间,60秒内未使用则连接被丢弃。若为0则永不丢弃。Default: 0 --> <property name="maxIdleTime"> <value>${sso.datasource.maxIdleTime}</value> </property> <!--当连接池中的连接耗尽的时候c3p0一次同时获取的连接数。Default: 3 --> <property name="acquireIncrement"> <value>${sso.datasource.acquireIncrement}</value> </property> <!--JDBC的标准参数，用以控制数据源内加载的PreparedStatements数量。但由于预缓存的statements 属于单个connection而不是整个连接池。所以设置这个参数需要考虑到多方面的因素。 如果maxStatements与maxStatementsPerConnection均为0，则缓存被关闭。Default: 0--> <property name="maxStatements"> <value>${sso.datasource.maxStatements}</value> </property> <!--每60秒检查所有连接池中的空闲连接。Default: 0 --> <property name="idleConnectionTestPeriod"> <value>${sso.datasource.idleConnectionTestPeriod}</value> </property> <!--定义在从数据库获取新连接失败后重复尝试的次数。Default: 30 --> <property name="acquireRetryAttempts"> <value>${sso.datasource.acquireRetryAttempts}</value> </property> <!--获取连接失败将会引起所有等待连接池来获取连接的线程抛出异常。但是数据源仍有效 保留，并在下次调用getConnection()的时候继续尝试获取连接。如果设为true，那么在尝试 获取连接失败后该数据源将申明已断开并永久关闭。Default: false--> <property name="breakAfterAcquireFailure"> <value>${sso.datasource.breakAfterAcquireFailure}</value> </property> <!--因性能消耗大请只在需要的时候使用它。如果设为true那么在每个connection提交的 时候都将校验其有效性。建议使用idleConnectionTestPeriod或automaticTestTable 等方法来提升连接测试的性能。Default: false --> <property name="testConnectionOnCheckout"> <value>${sso.datasource.testConnectionOnCheckout}</value> </property> </bean> <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager"> <property name="dataSource" ref="dataSource"/> </bean> <!-- C3P0连接池配置--> </beans>

3，查看工程参数配置文件

查看工程启动的参数配置文件，check里面的数据库配置等等，一切正常使用，如下所示：

[tomcat@dev_121_21 apache-tomcat-6.0.37_7100]$ more ./webapps/PLATFORM_SSO_Web/WEB-INF/classes/system.properties |grep -v "#" system.id=1000 system.env=default properties.hessian.serviceUrl=http://bs.powerlong.com/PLATFORM_PROPERTIES_Service/remoting/PropertiesService sso.server.login.url=http://bs.powerlong.com/PLATFORM_SSO_Web/login.htm sso.domain= sso.restriction.policy.error.number=5 sso.exception.error.page= mobile.verify.interface.url= auth.server.failed.url=http://bs.powerlong.com/PLATFORM_AUTH_Web/auth/failed.htm sso.datasource.url=jdbc:mysql://zitandbtest.powerlong.com:3306/platform_db?useUnicode=true&characterEncoding=utf-8&autoReconnect=true&preferredTestQuery=select now() sso.datasource.driverClassName=com.mysql.jdbc.Driver sso.datasource.username=platform_user sso.datasource.password=ysh1124 sso.datasource.maxPoolSize=120 sso.datasource.minPoolSize=5 sso.datasource.initialPoolSize=10 sso.datasource.maxIdleTime=10 sso.datasource.acquireIncrement=5 sso.datasource.maxStatements=0 sso.datasource.idleConnectionTestPeriod=10 sso.datasource.acquireRetryAttempts=30 sso.datasource.breakAfterAcquireFailure=true sso.datasource.testConnectionOnCheckout=true url.static=http://bsstatic.powerlong.com url.static.https=http://bsstatic.powerlong.com url.www=http://bs.powerlong.com/PLATFORM_MANAGER_Web/manager/index.htm url.address=http://bsstatic.powerlong.com base_path = http://bs.powerlong.com/PLATFORM_MANAGER_Web auth_web_path = http://bs.powerlong.com/PLATFORM_AUTH_Web/ [tomcat@dev_121_21 apache-tomcat-6.0.37_7100]$

4，最终的解决办法

（1） 重启7100的tomcat后，仍然一模一样的错误。
（2） Kill掉mysql5.7上的所有session线程后重启tomcat，仍然报一模一样的错误。
（3） 重启mysql数据库后再重启tomcat，仍然报一模一样的错误。

上午是好的，下午出问题了，重启tomcat和mysql都不能解决，那么问题就在别的地方了？那么问题在哪里？再次检查参数文件里面，有
properties.hessian.serviceUrl=http://bs.powerlong.com/PLATFORM_PROPERTIES_Service/remoting/PropertiesService

这个工程是外部工程，在7200的tomcat上面，难道是外部的工程引起的？

然后ps –eaf|grep 7200; 发现有3个启动的7200进程，那么就是7200启动失败而且卡住了，强行使用kill -9 pid清除7200进程，然后重启7200tomcat成功后，再次启动7100的tomcat进程，这次成功了，没有报错了。

此次问题是由于服务器上的7200端口的tomcat异常，启动了3个，而7100的tomcat需要先去7200上面获取service接口，去通过调用7200的service接口来初始化数据库连接，所以7200异常了，那么7100启动获取资源建立数据库连接池就会报错了。

[/b]