Mongodb Replica Set高可用

概述

对于mongodb的相关介绍，在此不多累赘，如需了解请见:
http://www.mongodb.org/display/DOCS/Home
本文目的是说明如下几个功能：

1.在测试环境构建Replica Set方案

2.在服务器端和客户端测试failover

3.测试天然的读写分离，减轻服务器压力

4.添加权限认证功能

部署Replica Sets 方案

Relica Sets使用的是n个mongod节点，构建具备自动的容错功能(auto-failover),自动恢复的(auto-recovery)的高可用方案,理论上需要三个mongodb实例，在这个测试环境中，我采用的方案是2个mongod+1个arbiter。

(0)基础环境

mongod1 : deploy
ip : 10.12.7.107
port : 27031
$ mkdir -p /data/db/0
./mongod  --dbpath /data/db/0 --port 27031 --replSet myset

mongod2 : doc
ip : 10.12.7.108
port : 27032
$ mkdir -p /data/db/1
./mongod  --dbpath /data/db/1 --port 27032 --replSet myset

mongod3 : deploy
ip : 10.12.7.107
port : 27033
$ mkdir -p /data/db/2
./mongod  --dbpath /data/db/2 --port 27033 --replSet myset

(1)实例化Replica Sets

任选一个mongod节点，mongo shell 登陆进去,执行如下内容：

> config = {_id: 'myset', members: [
{_id: 0, host: '10.12.7.107:27031'},
{_id: 1, host: '10.12.7.108:27032'},
{_id: 2, host: '10.12.7.107:27033', arbiterOnly: true}]}
> rs.initiate(config)
> rs.conf() #查看配置信息
> rs.staus()

到此，Replica Sets配置就算完成了，相关的配置信息保存在local数据库中。

(2)执行流程说明

在同一时刻，每组 Replica Sets 只有一个 Primary，用于接受写操作。而后会异步复制到其他成员数据库中。一旦 primary 死掉，会自动投票选出接任的 primary 来，原服务器恢复后成为普通成员。如果数据尚未从先前的 primary 复制到成员服务器，有可能会丢失数据。

在服务器端和客户端测试failover

(1)服务器端

在 mongo 中向 primary (27031) 插入数据：

> use test
switched to db test
> db.users.insert({name:"terrylc"})

会在mongodb的输出信息中，看到如下信息：

#27031
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.ns, filling with zeroes...
Tue Oct 18 08:44:07 [FileAllocator] done allocating datafile /data/db/0/test.ns, size: 16MB,  took 0.054 secs
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.0, filling with zeroes...
Tue Oct 18 08:44:07 [FileAllocator] done allocating datafile /data/db/0/test.0, size: 64MB,  took 0.153 secs
Tue Oct 18 08:44:07 [FileAllocator] allocating new datafile /data/db/0/test.1, filling with zeroes...
Tue Oct 18 08:44:07 [conn3] building new index on { _id: 1 } for test.users
Tue Oct 18 08:44:07 [conn3] done for 0 records 0secs
Tue Oct 18 08:44:07 [conn3] insert test.users 213ms
Tue Oct 18 08:44:09 [FileAllocator] done allocating datafile /data/db/0/test.1, size: 128MB,  took 1.527 secs

#27032
Tue Oct 18 23:43:02 [FileAllocator] allocating new datafile /data/db/1/test.ns, filling with zeroes...
Tue Oct 18 23:43:02 [FileAllocator] done allocating datafile /data/db/1/test.ns, size: 16MB,  took 0.054 secs
Tue Oct 18 23:43:02 [FileAllocator] allocating new datafile /data/db/1/test.0, filling with zeroes...
Tue Oct 18 23:43:03 [FileAllocator] done allocating datafile /data/db/1/test.0, size: 64MB,  took 0.556 secs
Tue Oct 18 23:43:03 [FileAllocator] allocating new datafile /data/db/1/test.1, filling with zeroes...
Tue Oct 18 23:43:03 [replica set sync] building new index on { _id: 1 } for test.users
Tue Oct 18 23:43:03 [replica set sync] done for 0 records 0secs
Tue Oct 18 23:43:03 [FileAllocator] done allocating datafile /data/db/1/test.1, size: 128MB,  took 0.166 secs

#27033
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is up
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet member 10.12.7.107:27031 PRIMARY
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet info 10.12.7.108:27032 is up
Tue Oct 18 08:42:22 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 RECOVERING
Tue Oct 18 08:42:34 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 SECONDARY

停止27031服务，观察现象：

#27032
Tue Oct 18 23:47:47 [conn2] end connection 10.12.7.107:58587
Tue Oct 18 23:47:47 [replica set sync] replSet syncThread: 10278 dbclient error communicating with server: 10.12.7.107:27031
Tue Oct 18 23:47:48 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Tue Oct 18 23:47:48 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is down (or slow to respond): DBClientBase::findOne: transport error: 10.12.7.107:27031 query: { replSetHeartbeat: "myset", v: 1, pv: 1, checkEmpty: false, from: "10.12.7.108:27032" }
Tue Oct 18 23:47:48 [rs Manager] replSet info electSelf 1
Tue Oct 18 23:47:48 [rs Manager] replSet couldn't elect self, only received -9999 votes
Tue Oct 18 23:47:54 [rs Manager] replSet info electSelf 1
Tue Oct 18 23:47:54 [rs Manager] replSet PRIMARY

#27033
Tue Oct 18 08:48:52 [conn2] end connection 10.12.7.107:43768
Tue Oct 18 08:48:53 [conn3] 10.12.7.108:27032 is trying to elect itself but 10.12.7.107:27031 is already primary and more up-to-date
Tue Oct 18 08:48:54 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Tue Oct 18 08:48:54 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is down (or slow to respond): DBClientBase::findOne: transport error: 10.12.7.107:27031 query: { replSetHeartbeat: "myset", v: 1, pv: 1, checkEmpty: false, from: "10.12.7.107:27033" }
Tue Oct 18 08:48:59 [conn3] replSet info voting yea for 1
Tue Oct 18 08:49:00 [ReplSetHealthPollTask] replSet member 10.12.7.108:27032 PRIMARY
}}}

Mongod 27032 被选为 Primary,在对应的终端下，执行shell mongo

$ ./mongo localhost:27032
> rs.status()
{
"set" : "myset",
"date" : ISODate("2011-10-18T15:53:01Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "10.12.7.107:27031",
"health" : 0,
"state" : 1,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : {
"t" : 1318952647000,
"i" : 1
},
"optimeDate" : ISODate("2011-10-18T15:44:07Z"),
"lastHeartbeat" : ISODate("2011-10-18T15:47:46Z"),
"errmsg" : "socket exception"
},
{
"_id" : 1,
"name" : "10.12.7.108:27032",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1318952647000,
"i" : 1
},
"optimeDate" : ISODate("2011-10-18T15:44:07Z"),
"self" : true
},
{
"_id" : 2,
"name" : "10.12.7.107:27033",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 705,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2011-10-18T15:53:00Z")
}
],
"ok" : 1
}

}}}

查询先前插入的记录正常:

myset:PRIMARY> db.users.findOne()
{ "_id" : ObjectId("4e9d9ec76df8f2f72bd9c45a"), "name" : "terrylc" }

如果重新启动27031对应的mongod，此实例将会变成secondary

Tue Oct 18 23:56:46 [ReplSetHealthPollTask] replSet info 10.12.7.107:27031 is up
Tue Oct 18 23:56:46 [ReplSetHealthPollTask] replSet member 10.12.7.107:27031 SECONDARY
Tue Oct 18 23:56:47 [conn6] I am already primary, 10.12.7.107:27031 can try again once I've stepped down
Tue Oct 18 23:56:48 [initandlisten] connection accepted from 10.12.7.107:44777 #7
Tue Oct 18 23:56:49 [slaveTracking] building new index on { _id: 1 } for local.slaves
Tue Oct 18 23:56:49 [slaveTracking] done for 0 records 0secs
Tue Oct 18 23:57:03 [conn5] end connection 127.0.0.1:58555

(2)客户端测试

从客户端连接 Replica Sets，需要 drivers 支持，本例中，我采用pymongo2.0.1

* 测试功能可用性

目标：当一个主节点出现故障时，备用节点是否可以转化为主节点。

import pymongo
import sys
def testInsert():
flag = 0
while True:
if flag == 4:
print "try to connect server 4 times but failed,so give up"
break
try:
conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"])
db = conn.test
post = {"name1": "terrylc"}
db.users.insert(post)
conn.disconnect()
return sys.exit(0)
except Exception, e:
print e
flag += 1

if __name__ == '__main__':
testInsert()
}}}

结论：

在服务器端的某个实例当掉后，客户端依然能够正常工作。

* 测试大规模写数据完整性

需求：当进行大规模写的时候，如果几个节点在特定时间交叉宕机后，能否确保数据完成性

目标：用python客户端向db中连续插入5000条记录，期望在插入完成后db中增加5000条记录：

def testInsert2(index):
try:
conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"])
db = conn.test
while index <= 5000:
post = {"testTAG": index}
time.sleep(1)
objectId = db.users.save(post, safe=True,w=2)
if objectId is not None:
index += 1
conn.disconnect()
except Exception, e:
print e
print "you index is :" + str(index)
time.sleep(2)
testInsert2(index)

if __name__ == '__main__':
testInsert2(1)
}}}

结论:

在测试过程，分别模拟几台mongodb节点宕机现象，客户端在进行重新连接后能够继续进行写操作，并且最后数据一致且都能查询到。

测试天然的读写分离，减轻服务器压力

其实很简单的方式是在客户端的请求中添加如下请求字段即可：

conn = pymongo.Connection(host=["10.12.7.107:27031", "10.12.7.108:27032", "10.12.7.107:27033"],
slave_okay=True)

slave_okay=True  # 这个是关键，可以将读写分离

添加权限认证功能

Authentication was added in 1.7.5(mongodb)

Replica Sets 的认证跟原本的单台的认证是不同的，最直接的表现在使用replica Sets时，启动实例方式如下：

$ echo "this is my super secret key" > mykey
$ chmod 600 mykey
$ mongod --keyFile mykey # other options...

$ echo "this is my super secret key" > mykey
$ chmod 600 mykey
$ mongod --keyFile mykey # other options...

$ echo "this is my super secret key" > mykey
$ chmod 600 mykey
$ mongod --keyFile mykey # other options...

然后在对应的数据库中添加用户即可。

启动方式不再使用--auth

补充

如果后续需要考虑写服务器的负载问题，可以在现在的组件外层加上Sharding层。

来自http://www.cnblogs.com/terrylc/archive/2011/12/13/2285955.html