MongoDB 并发机制

并发机制

以下列出各版本的新特性

MongoDB 2.2 并发机制

https://docs.mongodb.com/v2.2/faq/concurrency/

1、锁类型

使用 readers-writer 锁机制，允许同时有多个读操作，但只能有一个写操作；一个写操作会阻塞其它读、写操作；写操作优先级高于读操作

2、锁粒度

2.2 版本之前只有一个实例级别的全局锁。从 2.2 版本开始大多操作实现了数据库级别的锁粒度，少量操作仍用到实例级的全局锁

3、查看锁状态

i. db.serverStatus()

Locks：https://docs.mongodb.com/v2.2/reference/server-status/#server-status-locks

"locks" : {
"." : {
"timeLockedMicros" : {
"R" : <num>,
"W" : <num>
},
"timeAcquiringMicros" : {
"R" : <num>,
"W" : <num>
}
},
...
"<database>" : {
"timeLockedMicros" : {
"r" : <num>,
"w" : <num>
},
"timeAcquiringMicros" : {
"r" : <num>,
"w" : <num>
}
}
},

Global Lock：https://docs.mongodb.com/v2.2/reference/server-status/#server-status-global-lock

"globalLock" : {
"totalTime" : <num>,
"lockTime" : <num>,
"currentQueue" : {
"total" : <num>,
"readers" : <num>,
"writers" : <num>
},
"activeClients" : {
"total" : <num>,
"readers" : <num>,
"writers" : <num>
}
},

ii. db.currentOp()

Current Operation：https://docs.mongodb.com/v2.2/reference/current-op/

{
"inprog": [
{
"opid" : 3434473,
"active" : <boolean>,
"secs_running" : 0,
"op" : "<operation>",
"ns" : "<database>.<collection>",
"query" : {
},
"client" : "<host>:<outgoing>",
"desc" : "conn57683",
"threadId" : "0x7f04a637b700",
"connectionId" : 57683,
"locks" : {
"^" : "w",
"^local" : "W",
"^<database>" : "W"
},
"waitingForLock" : false,
"msg": "<string>"
"numYields" : 0,
"progress" : {
"done" : <number>,
"total" : <number>
}
"lockStats" : {
"timeLockedMicros" : {
"R" : NumberLong(),
"W" : NumberLong(),
"r" : NumberLong(),
"w" : NumberLong()
},
"timeAcquiringMicros" : {
"R" : NumberLong(),
"W" : NumberLong(),
"r" : NumberLong(),
"w" : NumberLong()
}
}
},
]
}

iii. mongotop

–locks 参数：https://docs.mongodb.com/v2.2/reference/mongotop/#cmdoption–locks

# mongotop22 -h localhost:28022 --locks
connected to: localhost:28022

db       total        read       write      2016-09-20T04:05:12
local         0ms         0ms         0ms
admin         0ms         0ms         0ms
.         0ms         0ms         0ms

...

iv. mongostat and/or

locked db 字段：https://docs.mongodb.com/v2.2/reference/mongostat/#fields

# mongostat22 -h localhost:28022
connected to: localhost:28022
insert  query update delete getmore command flushes mapped  vsize    res faults  locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time
0      0      0      0       0       1       0     0m   128m    28m      0 local:0.0%          0       0|0     0|0    62b     1k     1   12:08:07
0      0      0      0       0       1       0     0m   128m    28m      0 local:0.0%          0       0|0     0|0    62b     1k     1   12:08:08
...

v. the MongoDB Management Service (MMS)

4、锁的退让（yield）

当出现需要等待慢速 IO 操作时，临时释放掉锁，等 IO 完成后再重新添加锁

目的是将长时间操作占用的大锁拆分为多个细粒度的小锁，不会一直占用锁

2.0 基于时间分片 yield

2.2 优化算法，基于磁盘访问预测：增加了内存数据跟踪，能够预测出要读取的数据是否在内存中

5、产生数据库锁的操作

Operation	Lock Type
Issue a query	Read lock
Get more data from a cursor	Read lock
Insert data	Write lock
Remove data	Write lock
Update data	Write lock
Map-reduce	Read lock and write lock, unless operations are specified as non-atomic. Portions of map-reduce jobs can run concurrently.
Create an index	Building an index in the foreground, which is the default, locks the database for extended periods of time.
db.eval()	Write lock. db.eval() blocks all other JavaScript processes.
eval	Write lock. If used with the nolock lock option, the eval option does not take a write lock and cannot write data to the database.
aggregate()	Read lock

6、产生数据库锁的管理命令

i. 长期写锁

db.collection.ensureIndex(), when issued without setting background to true,

reIndex,

compact,

db.repairDatabase(),

db.createCollection(), when creating a very large (i.e. many * gigabytes) capped collection,

db.collection.validate(), and

db.copyDatabase(). This operation may lock all databases. See Does a MongoDB operation ever lock more than one database?.

ii. 读锁

The db.collection.group() operation takes a read lock and does not allow any other threads to execute JavaScript while it is running.

iii.短期锁

db.collection.dropIndex(),

db.getLastError(),

db.isMaster(),
rs.status() (i.e. replSetGetStatus,)

db.serverStatus(),

db.auth(), and

db.addUser().

7、同时锁多个库的操作

db.copyDatabase() must lock the entire mongod instance at once.

Journaling, which is an internal operation, locks all databases for short intervals. All databases share a single journal.

User authentication locks the admin database as well as the database the user is accessing.

All writes to a replica set’s primary lock both the database receiving the writes and the local database. The lock for the local database allows the mongod to write to the primary’s oplog.

8、分片

每个分片 mongod 相互独立，互不影响

9、主库

写数据同时写 local.oplog，要锁 2 个库

10、备库

备库上批量并行的执行 oplog，在同步数据时不允许读操作

11、JavaSript

一个 mongod 实例同时只能执行一个 JavaScript 操作

一些使用限制：

mapReduce

The JavaScript operations within a mapReduce job are short lived and yield many times during the operation. Portions of the map-reduce operation take database locks for reading, writing data to a temporary collection and writing the final output of the write operation.

group

The group takes a read lock in addition to blocking all other JavaScript execution.

db.eval()

Unless you specify the nolock option, db.eval() takes a write lock in addition to blocking all JavaScript operations.

$where

Only a single query that uses the $where operation can run at a time.

MongoDB 2.4 并发机制

https://docs.mongodb.com/v2.4/faq/concurrency/

1、查看锁状态

i. db.serverStatus()

支持指定输出的字段，如：

db.serverStatus( { repl: 0, indexCounters: 0, locks: 0 } )
db.serverStatus( { workingSet: 1, metrics: 0, locks: 0 } )

2、JavaScript

使用 V8 JavaScript 引擎，支持同时执行多个 JavaScript 操作

MongoDB 2.6 并发机制

https://docs.mongodb.com/v2.6/faq/concurrency/

1、锁的退让（yield）

扫描索引时不 yield lock

2、同时锁多个库的操作

User authentication requires a read lock on the admin database for deployments using 2.6 user credentials. For deployments using the 2.4 schema for user credentials, authentication locks the admin database as well as the database the user is accessing.

用户认证：2.6 锁 admin 一个库，2.4 锁 admin 和 user auth 两个库

3、事务

不支持多文档之间事务

支持对单个文档的原子性操作

https://docs.mongodb.com/v2.6/core/write-operations-atomicity/

4、隔离

https://docs.mongodb.com/v2.6/faq/concurrency/#what-isolation-guarantees-does-mongodb-provide

并发读写操作情况下提供以下隔离保证：

1、单独文档的读写操作是原子性的，保证文档处于一致性的状态。意味着，读不会看到部分更新的文档；索引和集合内容保持一致性；对单一文档的读写是串行的

2、查询谓词的正确性。如 find、update 只会影响符合条件的文档

3、排序的正确性。find、aggregate 的排序顺序不受并发写的影响

多文档操作没有事务性保证，会出现以下情况：

1、Non-point-in-time read operations。同时执行 read 和 update，read 可以看到 update 后的数据，而不是查询那一刻的快照

2、Non-serializable operations。假如有一个读操作和一个写操作，t1 读 d1、t3 写 d1，如果操作是串行的，读操作必须在写操作前执行。再假如 t2 写 d2、t4 读 d2，这又导致写操作必选先于读操作。这就形成了循环依赖，所以读写两个操作不能串行执行

3、Dropped results。读操作过程中更新或删除文档，可能导致读取不到匹配的文档的情况

5、读未提交数据

读出未经客户端确认的写操作的数据

读出即将回滚的数据

上述场景也被称作 read uncommitted

MongoDB 3.0 并发机制

3.0 允许多个客户端同时读写

1、锁类型

支持多粒度锁，支持全局、数据库、集合级别

支持存储引擎实现自己的并发控制机制，可以实现小于集合级别的锁

除了共享锁（S）、排他锁（X），还增加了 IS、IX 意向锁。意向锁表示将用更细粒度的锁来读写资源。在某个粒度上添加锁，在所有更高级别上也要添加意向锁

参考：https://en.wikipedia.org/wiki/Multiple_granularity_locking

To Get	Must Have on all Ancestors
IS or S	IS or IX
IX, SIX or X	IX or SIX

兼容性：

Mode	NL	IS	IX	S	SIX	X
NL	Yes	Yes	Yes	Yes	Yes	Yes
IS	Yes	Yes	Yes	Yes	Yes	No
IX	Yes	Yes	Yes	No	No	No
S	Yes	Yes	No	Yes	No	No
SIX	Yes	Yes	No	No	No	No
X	Yes	No	No	No	No	No

读写锁是公平的，按顺序排队。为了提高吞吐量，当授权一个锁请求，同时也会授权其它所有兼容的锁请求。

2、锁粒度

WiredTiger

大部分读写操作使用乐观锁

只在全局、库、集合级别使用意向锁

一旦存储引擎发现两个操作存在冲突，会引发其中一个写冲突，让 MongoDB 重试该操作

一些全局操作，主要是跨库的短期操作，仍用到全局实例锁

一些操作，比如删除集合，仍用到排他的数据库锁

MMAPv1

3.0 开始使用集合级别的锁机制，之前版本是数据库锁

3、查看锁状态

i. db.currentOp()

{
"inprog": [
{
"desc" : <string>,
"threadId" : <string>,
"connectionId" : <number>,
"opid" : <number>,
"active" : <boolean>,
"secs_running" : <NumberLong()>,
"microsecs_running" : <number>,
"op" : <string>,
"ns" : <string>,
"query" : <document>,
"insert" : <document>,
"planSummary": <string>,
"client" : <string>,
"msg": <string>,
"progress" : {
"done" : <number>,
"total" : <number>
},
"killPending" : <boolean>,
"numYields" : <number>,
"locks" : {
"Global" : <string>,
"MMAPV1Journal" : <string>,
"Database" : <string>,
"Collection" : <string>,
"Metadata" : <string>,
"oplog" : <string>
},
"waitingForLock" : <boolean>,
"lockStats" : {
"Global": {
"acquireCount": {
"r": <NumberLong>,
"w": <NumberLong>,
"R": <NumberLong>,
"W": <NumberLong>
},
"acquireWaitCount": {
"r": <NumberLong>,
"w": <NumberLong>,
"R": <NumberLong>,
"W": <NumberLong>
},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(0),
"R" : NumberLong(0),
"W" : NumberLong(0)
},
"deadlockCount" : {
"r" : NumberLong(0),
"w" : NumberLong(0),
"R" : NumberLong(0),
"W" : NumberLong(0)
}
},
"MMAPV1Journal": {
...
},
"Database" : {
...
},
...
}
},
...
],
"fsyncLock": <boolean>,
"info": <string>
}

https://docs.mongodb.com/v3.0/reference/method/db.currentOp/#output-fields

The locks document reports the type and mode of locks the operation currently holds. The possible lock types are as follows:

Global represents global lock.

MMAPV1Journal represents MMAPv1 storage engine specific lock to synchronize journal writes; for non-MMAPv1 storage engines, the mode for MMAPV1Journal is empty.

Database represents database lock.

Collection represents collection lock.

Metadata represents metadata lock.

oplog represents lock on the oplog.

The possible modes are as follows:

R represents Shared (S) lock.

W represents Exclusive (X) lock.

r represents Intent Shared (IS) lock.

w represents Intent Exclusive (IX) lock.

ii. mongotop

https://docs.mongodb.com/v3.0/reference/program/mongotop/#options

–locks returns an error when called against a mongod instance that does not report lock usage.

iii. mongostat, and/or

https://docs.mongodb.com/v3.0/reference/program/mongostat/#fields

locked

Changed in version 3.0.0: Only appears when mongostat runs against pre-3.0 versions of MongoDB instances.

The percent of time in a global write lock.

4、产生数据库锁的操作

Operation	Lock Type
db.eval() Deprecated since version 3.0.	Write lock. The db.eval() method takes a global write lock while evaluating the JavaScript function. To avoid taking this global write lock, you can use the eval command with nolock: true.
eval Deprecated since version 3.0.	Write lock. By default, eval command takes a global write lock while evaluating the JavaScript function. If used with nolock: true, the eval command does not take a global write lock while evaluating the JavaScript function. However, the logic within the JavaScript function may take write locks for write operations.

5、存储引擎

MMAPv1 存储引擎

https://docs.mongodb.com/v3.0/core/mmapv1/

WiredTiger 存储引擎

https://docs.mongodb.com/v3.0/core/wiredtiger/

提供了文档级别的写并发控制

使用了多版本并发控制（MVCC）。操作开始之时，WiredTiger 提供了一个时间点快照。快照提供了内存数据的一致性视图。

每 60 秒或 2G Journal 发生一次检查点操作

6、事务

https://docs.mongodb.com/v3.0/faq/fundamentals/#does-mongodb-support-transactions

不支持多文档事务

在单个文档上的操作是原子的

MongoDB 3.2 并发机制

1、查看锁状态

i. mongostat, and/or

lr

New in version 3.2.

Only for MMAPv1 Storage Engine. The percentage of read lock acquisitions that had to wait.mongostat displays lr|lw if a lock acquisition waited.

lw

New in version 3.2.

Only for MMAPv1 Storage Engine. The percentage of write lock acquisitions that had to wait.mongostat displays lr|lw if a lock acquisition waited.

lrt

New in version 3.2.

Only for MMAPv1 Storage Engine. The average acquire time, in microseconds, of read lock acquisitions that waited. mongostat displays lrt|lwt if a lock acquisition waited.

lwt

New in version 3.2.

Only for MMAPv1 Storage Engine. The average acquire time, in microseconds, of write lock acquisitions that waited. mongostat displays lrt|lwt if a lock acquisition waited.

2、读未提交数据

https://docs.mongodb.com/manual/release-notes/3.2/#readconcern

新增 readConcern 选项

WiredTiger：

readConcern=majority 只能读取到已被 majority 确认写入磁盘的数据

readConcern=local 可读取未经确认的数据（同旧版本）

MMAPv1：

readConcern=local 只有这一个选项

3、存储引擎

默认为 WiredTiger

参考

关于MongoDB的全局锁 2012-02-03 http://www.linuxidc.com/Linux/2012-02/53133.htm

MongoSF: MongoDB Concurrency Internals in v2.2 2012-05-07 http://www.slideshare.net/mongodb/mongosf-mongodb-concurrency-internals-in-v22

Goodbye global lock - MongoDB 2.0 vs 2.2 2012-05-23 https://blog.serverdensity.com/goodbye-global-lock-mongodb-2-0-vs-2-2/

MongoDB 那些坑 2014-06-22 http://xiewenwei.github.io/blog/2014/06/22/trap-in-mongodb/

To what level does MongoDB lock on writes? (or: what does it mean by “per connection” 2014-06-24 http://stackoverflow.com/questions/17456671/to-what-level-does-mongodb-lock-on-writes-or-what-does-it-mean-by-per-connec

MongoDB V3 & V2 版本锁性能对比测试及锁的基本概况 2015-02-05 http://blog.csdn.net/lantian0802/article/details/43505339

A Technical Introduction to WiredTiger 2015-06-01 https://www.mongodb.com/presentations/a-technical-introduction-to-wiredtiger

MongoDB的并发 2015-11-19 http://blog.csdn.net/liuxingen/article/details/49801059

MongoDB WiredTiger Inernals 2015-12-09 http://www.slideshare.net/NorbertoLeite/mongodb-wiredtiger-inernals

Mongodb中FAQ整理 2015-12-13 http://shift-alt-ctrl.iteye.com/blog/2263285

-eof-