MongoDB 副本集集群从节点控制台报错10061：由于目标计算机积极拒绝，无法连接

    2018-01-05，mongo副本集集群报错10061：由于目标计算机积极拒绝，无法连接，问题处理过程汇总如下：

环境：

    操作系统：windows server 2008R2

    数据库版本：mongodb 3.2.10

--------------------------------------------------------------------------------------------------------------------------------------------

    首先查看集群3个节点的控制台日志

1、集群三台服务器控制台日志

192.168.72.33

2018-01-05T09:46:24.281+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05 05:16:28:3e9 

2018-01-05T09:46:24.432+0800 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 

2018-01-05T09:46:24.432+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory 'd:/mongodata/rs0-2/diagnostic.data' 

2018-01-05T09:46:24.443+0800 I NETWORK [initandlisten] waiting for connections on port 27013 

2018-01-05T09:46:25.485+0800 W NETWORK [ReplicationExecutor] Failed to connect 
to 192.168.72.31:27011, reason: errno:10061 由于目标计算机积极拒绝，无法连接。 

2018-01-05T09:46:25.533+0800 I REPL [ReplicationExecutor] New replica set co 

nfig in use: { _id: "rs0", version: 8, protocolVersion: 1, members: [ { _id: 0, 

host: "mongodb-rs0-0:27011", arbiterOnly: false, buildIndexes: true, hidden: fal 

se, priority: 100.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "mongo 

db-rs0-1:27012", arbiterOnly: false, buildIndexes: true, hidden: false, priority 

: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "mongodb-rs0-2:27013 

", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { 

}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatInte 

rvalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLas 

tErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: Obje 

ctId('59365592734d0747ee26e2a6') } } 

2018-01-05T09:46:25.534+0800 I REPL [ReplicationExecutor] This node is mongo db-rs0-2:27013 in
the config

192.168.72.32

2018-01-05T09:46:17.064+0800 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 

2018-01-05T09:46:17.064+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory 'd:/mongodata/rs0-1/diagnostic.data' 

2018-01-05T09:46:17.076+0800 I NETWORK [initandlisten] waiting for connections on port 27012 

2018-01-05T09:46:18.102+0800 W NETWORK [ReplicationExecutor] Failed to connect 
to 192.168.72.31:27011, reason: errno:10061 由于目标计算机积极拒绝，无法连接。 
2018-01-05T09:46:19.149+0800 W NETWORK [ReplicationExecutor] Failed to connect 
to 192.168.72.33:27013, reason: errno:10061 由于目标计算机积极拒绝，无法连接。 

2018-01-05T09:46:19.150+0800 I REPL [ReplicationExecutor] New replica set co 

nfig in use: { _id: "rs0", version: 8, protocolVersion: 1, members: [ { _id: 0, 

host: "mongodb-rs0-0:27011", arbiterOnly: false, buildIndexes: true, hidden: fal 

se, priority: 100.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "mongo 

db-rs0-1:27012", arbiterOnly: false, buildIndexes: true, hidden: false, priority 

: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "mongodb-rs0-2:27013 

", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { 

}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatInte 

rvalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLas 

tErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: Obje 

ctId('59365592734d0747ee26e2a6') } } 

2018-01-05T09:46:19.150+0800 I REPL [ReplicationExecutor] This node is mongo db-rs0-1:27012 in the config

192.168.72.31

2018-01-05T15:56:42.999+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05
05:12:59:b4a 

2018-01-05T15:56:43.000+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05
05:13:08:8df 

2018-01-05T15:56:43.000+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05 05:14:05:329 

2018-01-05T15:56:43.001+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05 05:15:30:25f 

2018-01-05T15:56:43.002+0800 I STORAGE [initandlisten] Placing a marker at optime Jan 05 05:15:39:4b1

    根据以上日志信息推测：由于集群主节点192.168.72.31发生存储类型的等待事件，导致主节点192.168.72.31拒绝2个从节点192.168.72.32/33的TCP连接

 

2、根据步骤1中的提示，查看mongo服务在操作系统层次的日志，操作系统日志从2018-1-5 4:59:25秒就已经告警提示D盘已经满载



3、查看192.168.72.31存储情况，果然如操作系统日志提示，D盘只剩余58MB的可用空间



4、由以上信息可以断定：由于Mongo集群主节点192.168.72.31存储空间满，导致主节点192.168.72.31的Mongo进程无法完成写操作从而拒绝2个从节点的连接导致整个mongo集群服务中断。经沟通得知，地市技术对当前Mongo主节点192.168.72.31数据做了备份，没有注意到D盘存储情况。


事后，地市技术立即删除节点192.168.72.31的冗余数据备份释放D盘空间，由于调度程序处于僵死状态，地市技术决定重启整个mongo集群服务器192.168.72.31/32/33。

 

5、重启完成后，mongo集群恢复正常，主节点192.168.72.31的mongo控制台提示调度程序bmi被接受连接到mongo集群的admin库



 

到此，本次湖北卫计委项目mongo集群修复支持工作全部成功完成！

建议：

1、大数据操作前，地市技术一定要做数据量、服务器磁盘预留空间的评估

2、数据变更操作过程中，密切监视数据库、数据库服务器相关的表空间、cpu、内存、IO等资源的使用情况

3、数据变更操作后，统计数据库、数据库服务器相关资源如表空间、磁盘空间的使用情况

4、地市技术最好每天上午上班、中午、下午下班定期查看数据库必须的资源如：磁盘空间、CPU、内存、磁盘IO、数据库表空间的使用情况，有异常立即通知相关人员