linux的clientmqueue问题和No space left on device

今日打开程序发现无法访问，一看后台日志提示

2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3f0b70 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1080f38 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f5438 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@d26103
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1df3255
Pool thread stack traces:
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
		java.net.SocketInputStream.socketRead0(Native Method)
		java.net.SocketInputStream.read(Unknown Source)
		oracle.net.ns.Packet.receive(Unknown Source)
		oracle.net.ns.NSProtocol.connect(Unknown Source)
		oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)
		oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)
		oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
		oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
		oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
		oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
		java.net.SocketInputStream.socketRead0(Native Method)
		java.net.SocketInputStream.read(Unknown Source)
		oracle.net.ns.Packet.receive(Unknown Source)
		oracle.net.ns.NSProtocol.connect(Unknown Source)
		oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)
		oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)
		oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
		oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
		oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
		oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
		java.net.SocketInputStream.socketRead0(Native Method)
		java.net.SocketInputStream.read(Unknown Source)
		oracle.net.ns.Packet.receive(Unknown Source)
		oracle.net.ns.NSProtocol.connect(Unknown Source)
		oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)
		oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)
		oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
		oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
		oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
		oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

然后连接oracle的工具也打不开，感觉需要重启oracle服务，但是去服务器采用Linu命令重启后就出现【No space left on device 】

[oracle@SERVER233 bin]$ lsnrctl start

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 29-NOV-2012 10:30:23

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

Starting /oracle10g/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.1.0 - Production
System parameter file is /oracle10g/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Log messages written to /oracle10g/app/oracle/product/10.2.0/db_1/network/log/listener.log
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
TNS-12549: TNS:operating system resource quota exceeded
 TNS-12560: TNS:protocol adapter error
  TNS-00519: Operating system resource quota exceeded
   Linux Error: 28: No space left on device

咋一看貌似，磁盘空间不够，然后采用命令 df -i
出现

[oracle@SERVER233 bin]$ df -i
Filesystem             Inode (I)已用 (I)可用 (I)已用% 挂载点
/dev/sda11           25608192  191904 25416288    1% /
/dev/sda1            6406144      44 6406100    1% /boot
none                  214052       1  214051    1% /dev/shm
/dev/sda10           6406144     212 6405932    1% /home
/dev/sda5            25608192      19 25608173    1% /opt
/dev/sda2            29949952      12 29949940    1% /sowft
/dev/sda9            2562240      60 2562180    1% /tmp
/dev/sda8            12812288  305651 12506637    3% /usr
/dev/sda6            2562240     991 2561249    1% /usr/local
/dev/sda7            6406144 6406144       0  100% /var
[oracle@SERVER233 bin]$ df -h
Filesystem            容量  已用 可用 已用% 挂载点
/dev/sda11            193G   17G  167G   9% /
/dev/sda1              49G   91M   46G   1% /boot
none                  3.0G     0  3.0G   0% /dev/shm
/dev/sda10             49G  135M   46G   1% /home
/dev/sda5             193G   93M  183G   1% /opt
/dev/sda2             225G  731M  213G   1% /sowft
/dev/sda9              20G   77M   19G   1% /tmp
/dev/sda8              97G  6.4G   85G   7% /usr
/dev/sda6              20G  177M   19G   1% /usr/local
/dev/sda7              49G   26G   21G  55% /var

觉得是var目录占用空间比较大 然后进入clientmqueue 发现占用比较大
日志是定期分析处理的，怎么会占这么大的空间？

查看日志文件大小
# du -sh /var/log
31M /var/log

好小耶，那到底是什么文件占用了大量空间了？

继续找。

du -sh /var/spool/clientmqueue/
889M /var/spool/clientmqueue

找到clientmqueue 目录占了大量空间。

这个目录底下的文件到底是干嘛的？？

分析：
系统中有用户开启了cron，而cron中执行的程序有输出内容，输出内容会以邮件形式发给cron的用户，而sendmail没有启动所以就产生了这些文件；

解决办法:
将crontab里面的命令后面加上> /dev/null 2>&1
注：2>&1：把错误重定向到输出要送到的地方。即把上述命令的执行结果重定向到/dev/null，即抛弃，同时，把产生的错误也抛弃。

rm -rf /var/spool/clientmqueue 删除这个目录。

重新修改所有的cron

这样的话就解决了clinetmqueue占用空间的问题。

参考网址：http://baozoutianshi.blog.sohu.com/203000776.html