mongo复制集+分片实现海量数据存储

MongoDB Auto-Sharding 解决了海量存储和动态扩容的问题，但离实际生产环境所需的高可
靠、高可用还有些距离，所以有了” Replica Sets + Sharding”的解决方案:
Shard:
使用 Replica Sets，确保每个数据节点都具有备份、自动容错转移、自动恢复能力。
Config:
使用3 个配置服务器，确保元数据完整性
Route:
使用3 个路由进程，实现负载平衡，提高客户端接入性能

基本结构如下：

主机	IP	服务及端口
ServerA	192.168.10.150	mongod shard1_1:27017 mongod shard2_1:27018 mongod config1:20000 mongs1:30000
ServerB	192.168.10.151	mongod shard1_2:27017 mongod shard2_2:27018 mongod config2:20000 mongs2:30000
ServerC	192.168.10.154	mongod shard1_3:27017 mongod shard2_3:27018 mongod config3:20000 mongs3:30000

创建数据目录

serverA:
mkdir -p /data/shardset/{shard1_1,shard2_1,config}/

serverB:
mkdir -p /data/shardset/{shard1_2,shard2_2,config}/

serverC
mkdir -p /data/shardset/{shard1_3,shard2_3,config}/

配置复制集

配置shard1所用到的Replica Sets

serverA:
mongod --shardsvr --replSet shard1 --port 27017 --dbpath /data/shardset/shard1_1 --logpath /data/shardset/shard1_1/shard1_1.log --logappend --fork

serverB:
mongod --shardsvr --replSet shard1 --port 27017 --dbpath /data/shardset/shard1_2 --logpath /data/shardset/shard1_2/shard1_2.log --logappend --fork
serverC:
mongod --shardsvr --replSet shard1 --port 27017 --dbpath /data/shardset/shard1_3 --logpath /data/shardset/shard1_3/shard1_3.log --logappend --fork
用mongo 连接其中一台机器的27017 端口的mongod，初始化Replica Sets“shard1”，执行:
[root@template ~]# mongo --port 27017
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:27017/test
>config = {_id: 'shard1', members: [ {_id: 0, host: '192.168.10.150:27017'},{_id: 1, host: '192.168.10.151:27017'},{_id: 2, host: '192.168.10.154:27017'}]}
{
"_id" : "shard1",
"members" : [
{
"_id" : 0,
"host" : "192.168.10.150:27017"
},
{
"_id" : 1,
"host" : "192.168.10.151:27017"
},
{
"_id" : 2,
"host" : "192.168.10.154:27017"
}
]
}
> rs.initiate(config)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}

配置shard2所用到的Replica Sets

ServerA:
mongod --shardsvr --replSet shard2 --port 27018 --dbpath /data/shardset/shard2_1 --logpath /data/shardset/shard2_1/shard2_1.log --logappend --fork

ServerB:
mongod --shardsvr --replSet shard2 --port 27018 --dbpath /data/shardset/shard2_2 --logpath /data/shardset/shard2_2/shard2_2.log --logappend --fork

ServerC:
mongod --shardsvr --replSet shard2 --port 27018 --dbpath /data/shardset/shard2_3 --logpath /data/shardset/shard2_3/shard2_3.log --logappend --fork

用mongo 连接其中一台机器的27018 端口的mongod，初始化Replica Sets “shard2”，执行:
[root@template ~]# mongo --port 27018
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:27018/test
>config = {_id: 'shard2', members: [ {_id: 0, host: '192.168.10.150:27018'},{_id: 1, host: '192.168.10.151:27018'},{_id: 2, host: '192.168.10.154:27018'}]}
{
"_id" : "shard2",
"members" : [
{
"_id" : 0,
"host" : "192.168.10.150:27018"
},
{
"_id" : 1,
"host" : "192.168.10.151:27018"
},
{
"_id" : 2,
"host" : "192.168.10.154:27018"
}
]
}
> rs.initiate(config)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}

配置3 台Config Server

在Server A、B、C上执行:
mongod --configsvr --dbpath /data/shardset/config --port 20000 --logpath/data/shardset/config/config.log --logappend --fork

配置3 台Route Process

在Server A、B、C上执行:
mongos --configdb 192.168.10.150:20000,192.168.10.151:20000,192.168.10.154:20000 --port 30000 --chunkSize 1 --logpath /data/shardset/mongos.log --logappend --fork

配置Shard Cluster

连接到其中一台机器的端口30000 的mongos 进程，并切换到admin 数据库做以下配置:
[root@template ~]# mongo --port 30000
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:30000/test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb-user mongos>
mongos> use admin
switched to db admin
mongos> db.runCommand({addshard:"shard1/192.168.10.150:27017,192.168.10.151:27017,192.168.10.154:27017"});
{ "shardAdded" : "shard1", "ok" : 1 }
mongos> db.runCommand({addshard:"shard2/192.168.10.150:27018,192.168.10.151:27018,192.168.10.154:27018"});
{ "shardAdded" : "shard2", "ok" : 1 }
mongos>
激活数据库及集合的分片
mongos> db.runCommand({ enablesharding:"test" })
{ "ok" : 1 }
mongos> db.runCommand({ shardcollection: "test.users", key: { _id:1 }})
{ "collectionsharded" : "test.users", "ok" : 1 }
mongos>

查看配置
mongos> use admin
switched to db admin
mongos> db.runCommand({listshards: 1})
{
"shards" : [
{
"_id" : "shard1",
"host" : "shard1/192.168.10.150:27017,192.168.10.151:27017,192.168.10.154:27017"
},
{
"_id" : "shard2",
"host" : "shard2/192.168.10.150:27018,192.168.10.151:27018,192.168.10.154:27018"
}
],
"ok" : 1
}

验证Sharding正常工作

mongos> use test
switched to db test
mongos> for(var i=1;i<=200000;i++) db.users.insert({id:i,addr_1:"Beijing",addr_2:"Shanghai"});
WriteResult({ "nInserted" : 1 })
mongos> db.users.stats()
{
"sharded" : true,
"systemFlags" : 1,
"userFlags" : 1,
"ns" : "test.users",
"count" : 200000,
"numExtents" : 13,
"size" : 22400000,
"storageSize" : 33689600,
"totalIndexSize" : 6908720,
"indexSizes" : {
"_id_" : 6908720
},
"avgObjSize" : 112,
"nindexes" : 1,
"nchunks" : 13,
"shards" : {
"shard1" : {
"ns" : "test.users",
"count" : 147600,
"size" : 16531200,
"avgObjSize" : 112,
"storageSize" : 22507520,
"numExtents" : 7,
"nindexes" : 1,
"lastExtentSize" : 11325440,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 4807488,
"indexSizes" : {
"_id_" : 4807488
},
"ok" : 1
},
"shard2" : {
"ns" : "test.users",
"count" : 52400,
"size" : 5868800,
"avgObjSize" : 112,
"storageSize" : 11182080,
"numExtents" : 6,
"nindexes" : 1,
"lastExtentSize" : 8388608,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 2101232,
"indexSizes" : {
"_id_" : 2101232
},
"ok" : 1
}
},
"ok" : 1
}

ServerA:
[root@template ~]# mongo --port 27017
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:27017/test
shard1:PRIMARY> show dbs
admin (empty)
local 4.076GB
test 0.078GB
shard1:PRIMARY>

ServerB:
[root@template ~]# mongo --port 27017
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:27017/test
shard1:SECONDARY>
shard1:SECONDARY> show dbs
admin (empty)
local 4.076GB
test 0.078GB
shard1:SECONDARY>
默认从库是不可查询的，但是执行如下语句则可以查询。

rs.slaveOk();
或者
db.getMongo().setSlaveOk()
做读写分离时上述语句必不可少

ServerC:
[root@template ~]# mongo --port 27017
MongoDB shell version: 2.6.0
connecting to: 127.0.0.1:27017/test
shard1:SECONDARY> show dbs
admin (empty)
local 4.076GB
test 0.078GB
shard1:SECONDARY>
如果想在线切换主从库,需要更改优先级：
[root@template ~]# mongo --port 27017
需要在主库中设置
conf=rs.conf()
conf.members[0].priority=2
conf.members[1].priority=5
conf.members[2].priority=1
rs.reconfig(conf)

查看节点
db.status()
或者
db.isMaster()

查看数据是否开启分片
use config
db.databases.find()

增加节点
db.remove("192.168.10.155:27017")

删除节点
db.add("192.168.10.154:27017")

增加仲裁节点
rs.addArb("192.168.10.155:27017")
rs.isMaster()
"arbiters" : [
"192.168.10.155:27017"
],

创建用户
use admin
db.createUser(
{
user: "adminUserName",
pwd: "userPassword",
roles:
[
{
roles: "userAdminAnyDatabase",
db: "admin"
}
]
}
)