公司MongoDB 实施方案

绿岸网络MongoDB
---何旭东（某上市公司运维工程师）
目录
绿岸网络MongoDB. 1
场景介绍... 1
机器功能：... 2
主机名... 2
逻辑图... 3
primary secondary. 4
模型选择... 4
启动... 5
第一台... 5
第二台... 5
第三台... 5
配置... 5
第一台... 5
第二台... 6
第三台... 7
启动路由... 8
分片... 8
添加分片... 9
配置结果... 9
查看当前主库... 11
调整从库可读... 12
模拟故障... 13
主机点宕机... 13
从节点自动提升主节点... 13
主机点恢复... 14
问题... 15
临时调整，不生效... 15
程序实现可读... 17

场景介绍

MongoDB 3.0其实是2.8，上个版本是2.6.2.8有极大的增加，用2.8有点委屈，在MongoDB市场部主导命名3.0. 增加了高性能，可伸缩的存储引擎wiredtiger.性能有了极大的提升，尤其在写操作对硬件资源的利用率。
mongodb最初版本一直到2.6都只支持一种基于内存映射技术的存储引擎即mmap。mongodb 3.0实现对支持集合级锁的存储引擎mmap和支持压缩和文档级锁的存储引擎wiredtiger的支持。
MongoDB官方同步发布了一份YCSB测试报告。
且官方下载资源，确实了没有2.8。以下为官方下载地址 https://www.mongodb.org/downloads?_ga=1.220177757.596614690.1431100647#development

机器功能：

机器	角色名称	端口	功能
	mongod01	28017	'shard1' PRIMARY
	mongoc01	20000	配置节点
	mongos01	28885	路由节点
		41000	启动路由 生成此端口
	mongod02	28017	'shard1' SECONDARY
	mongoc02	20000	配置节点
	mongos02	28885	路由节点
		41000	启动路由 生成此端口
	mongod03	28017	'shard1' arbiter
	mongoc03	20000	配置节点

Shardingoptions:
--configdb arg 1 or 3 comma separated config servers
路由节点只能配置1 或者 3 个！

主机名

192.168.200.87 mongohost1
192.168.200.90 mongohost2
192.168.200.88 mongohost3
192.168.200.87 mongo01
192.168.200.87 mongodb01
192.168.200.87 mongos01
192.168.200.90 mongo02
192.168.200.90 mongodb02
192.168.200.90 mongos02
192.168.200.88 mongo03
192.168.200.88 mongodb03

逻辑图

primary secondary

模型选择

CAP定理（Consistency ，Availability和Partition Tolerance)

–Consistency(一致性)：数据一致更新，所有数据变动都是同步的
–Availability(可用性)：好的响应性能
–PartitionTolerance(分区容错性)：数据可靠性
–分布式系统三选二
官方CP模型
CP–ReplicaSet,设置写入节点数w=ReplicaSet数据节点数，查询开启SlaveOk
AP–ReplicaSet,默认设置w=1，查询开启slaveOk
CA–ReplicaSet,默认设置w=1，查询不开启slaveOk

启动

第一台

mongod -f /etc/shard11.conf
mongod -f /etc/config1.conf
mongos -f /etc/mongos1.conf

第二台

mongod -f /etc/shard12.conf
mongod -f /etc/config2.conf
mongos -f /etc/mongos2.conf

第三台

mongod -f /etc/shard13.conf
mongod -f /etc/config3.conf

配置

第一台

mkdir /usr/local/mongodb/shard11/ -p
mkdir /usr/local/mongodb/config/
mkdir /usr/local/mongodb/logs/

cat > /etc/shard11.conf <<EOF
shardsvr=true
port=28017
dbpath = /usr/local/mongodb/shard11/
logpath = /usr/local/mongodb/logs/shard11.log
logappend=true
fork=true
nojournal=true
rest=true
replSet=shard1
oplogSize=2048
EOF

cat > /etc/config1.conf <<EOF
configsvr=true
port=20000
dbpath = /usr/local/mongodb/config/
logpath =/usr/local/mongodb/logs/config1.log
logappend=true
fork=true
nojournal=true
EOF

cat > /etc/mongos1.conf <<EOF
configdb=mongodb01:20000,mongodb02:20000,mongodb03:20000
chunkSize=100
port=28885
logpath =/usr/local/mongodb/logs/mongos1.log
logappend=true
fork=true
EOF

第二台

mkdir /usr/local/mongodb/shard12/ -p
mkdir /usr/local/mongodb/config/
mkdir /usr/local/mongodb/logs/

cat > /etc/shard12.conf <<EOF
shardsvr=true
port=28017
dbpath = /usr/local/mongodb/shard12/
logpath =/usr/local/mongodb/logs/shard12.log
logappend=true
fork=true
nojournal=true
rest=true
replSet=shard1
oplogSize=2048
EOF

cat > /etc/config1.conf <<EOF
configsvr=true
port=20000
dbpath = /usr/local/mongodb/config/
logpath =/usr/local/mongodb/logs/config2.log
logappend=true
fork=true
nojournal=true
EOF

cat > /etc/mongos2.conf <<EOF
configdb = mongodb01:20000,mongodb02:20000,mongodb03:20000
chunkSize=100
port=28885
logpath =/usr/local/mongodb/logs/mongos2.log
logappend=true
fork=true
EOF

第三台

mkdir /usr/local/mongodb/shard13/ -p
mkdir /usr/local/mongodb/logs/
mkdir /usr/local/mongodb/config/

cat > /etc/shard13.conf <<EOF
shardsvr=true
port=28017
dbpath = /usr/local/mongodb/shard13/
logpath = /usr/local/mongodb/logs/shard13.log
logappend=true
fork=true
nojournal=true
rest=true
replSet=shard1
oplogSize=2048
EOF

cat > /etc/config3.conf <<EOF
configsvr=true
port=20000
dbpath = /usr/local/mongodb/config/
logpath = /usr/local/mongodb/logs/config3.log
logappend=true
fork=true
nojournal=true
EOF

启动路由

参考
mongos --configdbmongohost3:20000,mongohost2:20000,mongohost1:20000 --port 41000 --chunkSize 100--logpath /usr/local/mongodb/logs/mongos.log --logappend --fork

绿岸配置
mongos --configdb mongodb01:20000,mongodb02:20000,mongodb03:20000--port 41000 --chunkSize 100 --logpath /usr/local/mongodb/logs/mongos.log--logappend --fork

分片

以下为参考配置
mongo mongohost1:28017/admin
config = {_id:'shard1', members: [{_id: 0,host: 'mongohost1:28017'},{ "_id" : 1,"host":"mongohost2:28017",},{_id:2, host: 'mongohost3:28017', arbiterOnly : true}]};
rs.initiate(config);

以下为绿岸配置
mongo mongo01:28017/admin
config = {_id:'shard1', members: [{_id: 0,host: 'mongo01:28017'},{ "_id" : 1,"host":" mongo02:28017",},{_id:2, host: ' mongo03:28017', arbiterOnly : true}]};
rs.initiate(config);

添加分片

以下为参考配置
mongo mongohost1:28885/admin
db.runCommand({"addshard" :"shard1/mongohost1:28017,mongohost2:28017"})

以下为绿岸配置
mongo mongos01:28885/admin
db.runCommand({"addshard" :"shard1/mongo01:28017,mongo02:28017"})

配置结果

[root@mongohost1 mongodb]# mongomongohost1:28885/admin
MongoDB shell version: 2.6.10
connecting to: mongohost1:28885/admin
mongos> db.printShardingStatus();
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 4,
"minCompatibleVersion" : 4,
"currentVersion" : 5,
"clusterId" : ObjectId("55adedc8c5488912b58e20aa")
}
shards:
{ "_id" :"shard1", "host" :"shard1/mongohost1:28017,mongohost2:28017" }
databases:
{ "_id" :"admin", "partitioned" : false, "primary" : "config" }

mongos> db.printShardingStatus();
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 4,
"minCompatibleVersion" : 4,
"currentVersion" : 5,
"clusterId" : ObjectId("55adedc8c5488912b58e20aa")
}
shards:
{ "_id" : "shard1", "host" : "shard1/mongohost1:28017,mongohost2:28017"}
databases:
{ "_id" :"admin", "partitioned" : false, "primary" : "config" }

mongos> rs.status()
{
"info" : "mongos",
"ok" : 0,
"errmsg" : "replSetGetStatus is not supported throughmongos"
}
mongos> exit
bye
[root@mongohost1 mongodb]# mongomongohost1:28017/admin
MongoDB shell version: 2.6.10
connecting to: mongohost1:28017/admin
Server has startup warnings:
2015-07-21T14:57:14.934+0800 ** WARNING:--rest is specified without --httpinterface,
2015-07-21T14:57:14.934+0800 ** enabling http interface
2015-07-21T14:57:14.939+0800[initandlisten]
2015-07-21T14:57:14.939+0800[initandlisten] ** WARNING: Readahead for /usr/local/mongodb/shard11/ is set to4096KB
2015-07-21T14:57:14.939+0800[initandlisten] ** We suggestsetting it to 256KB (512 sectors) or less
2015-07-21T14:57:14.939+0800[initandlisten] ** http://dochub.mongodb.org/core/readahead shard1:PRIMARY> rs.status();
{
"set" : "shard1",
"date" : ISODate("2015-07-21T07:13:20Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" :"mongohost1:28017",
"health" : 1,
"state" : 1,
"stateStr" :"PRIMARY",
"uptime" :966,
"optime" :Timestamp(1437462480, 1),
"optimeDate": ISODate("2015-07-21T07:08:00Z"),
"electionTime" : Timestamp(1437462489, 1),
"electionDate" : ISODate("2015-07-21T07:08:09Z"),
"self" : true
},
{
"_id" : 1,
"name" :"mongohost2:28017",
"health" : 1,
"state" : 2,
"stateStr" :"SECONDARY",
"uptime" :319,
"optime" :Timestamp(1437462480, 1),
"optimeDate": ISODate("2015-07-21T07:08:00Z"),
"lastHeartbeat" : ISODate("2015-07-21T07:13:19Z"),
"lastHeartbeatRecv" : ISODate("2015-07-21T07:13:18Z"),
"pingMs" : 0,
"syncingTo" :"mongohost1:28017"
},
{
"_id" : 2,
"name" :"mongohost3:28017",
"health": 1,
"state" : 7,
"stateStr" :"ARBITER",
"uptime" :319,
"lastHeartbeat" : ISODate("2015-07-21T07:13:19Z"),
"lastHeartbeatRecv": ISODate("2015-07-21T07:13:18Z"),
"pingMs" : 0
}
],
"ok" : 1
}
shard1:PRIMARY>

查看当前主库

shard1:PRIMARY> rs.isMaster();
{
"setName" : "shard1",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"hosts" : [
"mongohost1:28017",
"mongohost2:28017"
],
"arbiters" : [
"mongohost3:28017"
],
"primary" : "mongohost1:28017",
"me" : "mongohost1:28017",
"electionId" : ObjectId("55adefda41c78564aaa0ffd8"),
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2015-07-21T07:21:14.600Z"),
"maxWireVersion" : 2,
"minWireVersion" : 0,
"ok" : 1
}
shard1:PRIMARY>

调整从库可读

passive ----节点执行以下内容
testSet:PRIMARY> show dbs;
admin (empty)
local 2.077GB
testSet:PRIMARY> use testdb;
switched to db testdb
testSet:PRIMARY>db.document01.insert({"name":"hxd","sex":"N","age":"22"});

WriteResult({ "nInserted" : 1 })
testSet:PRIMARY> show tables;
document01
system.indexes
testSet:PRIMARY> db.document01.find();
{ "_id" :ObjectId("55a71118442e12ca0d28b50a"), "name" :"hxd", "sex" : "N", "age" :"22" }
testSet:PRIMARY>

standard ----节点执行以下内容
testSet:SECONDARY> show dbs;
admin (empty)
local 2.077GB
testdb 0.078GB
testSet:SECONDARY> use testdb
switched to db testdb
testSet:SECONDARY> show tables;
2015-07-16T10:05:22.978+0800 error: {"$err" : "not master and slaveOk=false", "code" :13435 } at src/mongo/shell/query.js:131
testSet:SECONDARY> rs.slaveOk();
testSet:SECONDARY> show tables;
document01
system.indexes
testSet:SECONDARY> db.document01.find();
{ "_id" :ObjectId("55a71118442e12ca0d28b50a"), "name" :"hxd", "sex" : "N", "age" :"22" }
testSet:SECONDARY>

模拟故障

主机点宕机

[root@mongohost1 mongodb]# ps aux | grepshard11.conf
root 2142 0.4 5.6 2888808 56768 ? Sl 14:57 0:07 mongod -f/etc/shard11.conf
root 5191 0.0 0.0 112640 972 pts/0 S+ 15:25 0:00 grep --color=auto shard11.conf
[root@mongohost1 mongodb]# kill -2 2142

从节点自动提升主节点

shard1:SECONDARY> db.document01.find();
{ "_id" :ObjectId("55adf36ae512c815a6569440"), "name" :"hxd", "sex" : "N", "age" :"22" }
shard1:SECONDARY> rs.status()
{
"set" : "shard1",
"date" : ISODate("2015-07-21T07:25:47Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" :"mongohost1:28017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" :Timestamp(1437463402, 1),
"optimeDate": ISODate("2015-07-21T07:23:22Z"),
"lastHeartbeat" : ISODate("2015-07-21T07:25:47Z"),
"lastHeartbeatRecv" : ISODate("2015-07-21T07:25:39Z"),
"pingMs" : 0
},
{
"_id" : 1,
"name" :"mongohost2:28017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" :1683,
"optime" :Timestamp(1437463402, 1),
"optimeDate": ISODate("2015-07-21T07:23:22Z"),
"electionTime" :Timestamp(1437463543, 1),
"electionDate" : ISODate("2015-07-21T07:25:43Z"),
"self" : true
},
{
"_id" : 2,
"name" :"mongohost3:28017",
"health" : 1,
"state" : 7,
"stateStr" :"ARBITER",
"uptime" :1063,
"lastHeartbeat" : ISODate("2015-07-21T07:25:47Z"),
"lastHeartbeatRecv" :ISODate("2015-07-21T07:25:46Z"),
"pingMs" : 0
}
],
"ok" : 1
}
shard1:PRIMARY>

主机点恢复

主节点宕机状态：
"stateStr" : "(notreachable/healthy)",
恢复主节点后：
"stateStr" :"SECONDARY",

问题

问题：
降级的SECONDARY 为不可读模式。需手动调整。

在已升级成为：mongohost2 PRIMARY 节点插入数据。
shard1:PRIMARY> show dbs;
admin (empty)
local 2.077GB
testdb 0.078GB
shard1:PRIMARY> use testdb;
switched to db testdb
shard1:PRIMARY>db.document01.insert({"name":"lazy","sex":"N","age":"52"});
WriteResult({ "nInserted" : 1 })
shard1:PRIMARY> show tables;
document01
system.indexes
shard1:PRIMARY> db.document01.find();
{ "_id" : ObjectId("55adf36ae512c815a6569440"),"name" : "hxd", "sex" : "N","age" : "22" }
{ "_id" :ObjectId("55adf6ddc036dfbdc18191cc"), "name" :"lazy", "sex" : "N", "age" :"52" }
shard1:PRIMARY> exit
bye
[root@mongohost2 mongodb]#

在已降级成为：mongohost1 SECONDARY 节点查看数据
shard1:SECONDARY> rs.slaveOk();
shard1:SECONDARY> db.document01.find();
{ "_id" :ObjectId("55adf36ae512c815a6569440"), "name" :"hxd", "sex" : "N", "age" :"22" }
{ "_id" :ObjectId("55adf6ddc036dfbdc18191cc"), "name" :"lazy", "sex" : "N", "age" :"52" }
shard1:SECONDARY> exit
bye
[root@mongohost1 mongodb]#

临时调整，不生效

手动配置slaveOk ，为临时。exit退出后，将失去读取权限，问题 求解决。
以下为测试案例：
shard1:SECONDARY> show dbs;
admin (empty)
local 2.077GB
testdb 0.078GB
shard1:SECONDARY> use testdb
switched to db testdb
shard1:SECONDARY> show tables;
2015-07-21T15:52:12.008+0800 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131
shard1:SECONDARY> db.document01.find();
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
shard1:SECONDARY> rs.slaveOk();
shard1:SECONDARY> db.document01.find();
{ "_id" : ObjectId("55adf36ae512c815a6569440"), "name" : "hxd", "sex" : "N", "age" : "22" }
{ "_id" : ObjectId("55adf6ddc036dfbdc18191cc"), "name" : "lazy", "sex" : "N", "age" : "52" }
shard1:SECONDARY> exit
bye
[root@mongohost1 mongodb]# mongo mongohost1:28017/admin 再次登录
MongoDB shell version: 2.6.10
connecting to: mongohost1:28017/admin
Server has startup warnings:
2015-07-21T15:28:05.172+0800 ** WARNING: --rest is specified without --httpinterface,
2015-07-21T15:28:05.172+0800 ** enabling http interface
2015-07-21T15:28:05.176+0800 [initandlisten]
2015-07-21T15:28:05.176+0800 [initandlisten] ** WARNING: Readahead for /usr/local/mongodb/shard11/ is set to 4096KB
2015-07-21T15:28:05.176+0800 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2015-07-21T15:28:05.176+0800 [initandlisten] **
http://dochub.mongodb.org/core/readahead shard1:SECONDARY> use testdb
switched to db testdb
shard1:SECONDARY> db.document01.find();
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
shard1:SECONDARY> rs.slaveOk();
shard1:SECONDARY> show tables;
document01
system.indexes
shard1:SECONDARY> db.document01.find();
{ "_id" : ObjectId("55adf36ae512c815a6569440"), "name" : "hxd", "sex" : "N", "age" : "22" }
{ "_id" : ObjectId("55adf6ddc036dfbdc18191cc"), "name" : "lazy", "sex" : "N", "age" : "52" }
shard1:SECONDARY> exit
bye
[root@mongohost1 mongodb]#

北京-何旭东 2015/7/21 15:57:15

以上为mongohost1 ，在首次配置中为：PRIMARY状态。当mongohost1 数据节点宕机后。mongohost2 成为PRIMARY 状态。
现状： 想对mongohost1 进行读取操作，没有配置生效。
我找一下这个问题的答案，大家也可以想一下。

程序实现可读

就以上截图，重点说明了。1 当首次查询 没有权限，手从操作后，再次查询，没有问题。exit退出后
2 退出后，再次登录查询，发现又没有了 权限。 请求。
此次说明了。rs.slave0k(); 为临时生效，退出后 依然没有查询权限。
16:08:13
2015/7/21 16:08:13

的确如此：slaveOk只对当次的连接生效。每次连接之后都要判断一次当前是否master，不是就重新设置，的确有些不太合理了。
看看老师有什么解决方法

16:08:54
老师-孙玄2015/7/21 16:08:54

如果使用driver，在程序里可以设置永久开启slaveok
老师-孙玄2015/7/21 16:09:11

如果使用shell客户端，的确每次都有设置slaveok
老师-孙玄2015/7/21 16:09:19

不然从库读不了
37537-厦门-陈 2015/7/21 16:09:54

MongoDB为什么要这么设计呀？
2015/7/21 16:09:57

也就是使用driver的话，不管是否连master，都设置一次slaveok?

北京-何旭东 2015/7/21 16:10:04

哦。一会咨询下，我们的这边的大神。谢谢老师回答
2015/7/21 16:10:28

这是一种保护机制：防止误读取了从库，以至于读取到非最新的数据。
2015/7/21 16:10:28

实现
PRIMARY 节点 写入
SECONDARY 节点 读取
老师-孙 2015/7/21 16:15:51

基于paxos算法
老师-孙 2015/7/21 16:16:25

@北京-何旭东 这点replic set集合支持的
老师-孙 2015/7/21 16:16:30

开启slaveok就可以
2015/7/21 16:16:49

但之前程序是直接连接的单个节点，原本连接的这个节点是主库，重新选举后其他节点变成了主库，此节点变成从库。那程序在没有调整设置之前，如何从从库读取数据呢？
2015/7/21 16:17:06

所以，还是经过mongos是最佳的方案。
老师-孙 2015/7/21 16:17:21

对，直接连接数据节点，不行的
16:17:30
2015/7/21 16:17:30

也就是说副本集最好与分片一起，否则，读写分离上有些麻烦？
老师-孙 2015/7/21 16:17:31

经过mongos就没有这个问题了
老师-孙 2015/7/21 16:17:45

不是
老师-孙 2015/7/21 16:18:03

一个replic set 内部是支持读写分离的
老师-孙 2015/7/21 16:18:11

和sharding没有必然关系

本文出自 “晴空” 博客，谢绝转载！