Redis 集群搭建

一. 下载

https://github.com/antirez/redis/releases

选择

3.0

以上版本，之前版本不支持集群。

$ tar xzvf redis-3.2.6.tar.gz -C /usr/local
$ cd /usr/local/redis-3.2.6
$ make && make install
$ cp src/redis-trib.rb /usr/bin

1.1. 建立集群目录

$ cd /root
$ mkdir cluster
$ cd cluster
$ mkdir 7000 7001 7002 7003 7004 7005

1.2. 修改配置

$ cp /usr/local/redis-3.2.6/redis.conf /root/cluster
$ vim redis.conf

配置如下：

port 7000
daemonize yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

修改后，将

redis.conf

分别拷贝到 7000 7001 7002 7003 7004 7005 目录下。

注意： 拷贝后修改该目录下 redis.conf 中的

port

字段，与目录名对应。

分别进入上面6个目录，启动 redis 实例：

$ cd 700x
$ redis-server ./redis.conf

二. 建立 redis 集群

2.1. 准备

$ sudo yum install ruby rubygems
$ sudo gem install redis

注意： 如果遇到gem无法安装redis情况，可以采取下列方法：

$ gem sources -a https://ruby.taobao.org/ $ sudo gem install redis -V

好，现在准备阶段结束，可以正式建立集群了。

2.2. 建立

redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

create 表示创建一个新的集群。

–replicas 1 表示为集群中的每个主节点创建一个从节点。

其他参数则是实例的地址列表。

以上命令是让 redis-trib 程序创建一个包含三个主节点和三个从节点的集群。

接着 redis-trib 会打印出一份配置，输入

yes

后回车即可。

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
S: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
replicates 543beab779aa57c9c2ac8331373571afe6d7694e
Can I set the above configuration? (type 'yes' to accept):

如果一切正常的会输出下列信息：

>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
S: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots: (0 slots) slave
replicates 543beab779aa57c9c2ac8331373571afe6d7694e
M: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

我们来查看下：

$ redis-cli -c -p 7000
127.0.0.1:7000> CLUSTER NODES
c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003@17003 slave 68770484f2bc9c472f89d04b8862397e234e96e7 0 1515041902579 4 connected
0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005@17005 slave 543beab779aa57c9c2ac8331373571afe6d7694e 0 1515041901000 6 connected
543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002@17002 master - 0 1515041900563 3 connected 10923-16383
b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001@17001 master - 0 1515041902579 2 connected 5461-10922
946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004@17004 slave b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 0 1515041901573 5 connected
68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000@17000 myself,master - 0 1515041901000 1 connected 0-5460

三. 操作集群

CRC16('key') % 16384

的算法将

foo

的 key 放到了

7002

的槽上，redis 也相继跳到了 7002。

127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
127.0.0.1:7002> get foo
"bar"

3.1. 挂掉节点

master 节点挂掉会怎样？

$ ps axu|grep redis
test1     75480  0.1  0.2  50604  7940 ?        Ssl  12:04   0:06 redis-server 127.0.0.1:7000 [cluster]
test1     75487  0.1  0.2  50604  7100 ?        Ssl  12:04   0:05 redis-server 127.0.0.1:7001 [cluster]
test1     75492  0.1  0.1  50604  5100 ?        Ssl  12:04   0:06 redis-server 127.0.0.1:7002 [cluster]
test1     75497  0.1  0.2  50604  6932 ?        Ssl  12:04   0:06 redis-server 127.0.0.1:7003 [cluster]
test1     75502  0.1  0.2  50604  6924 ?        Ssl  12:04   0:05 redis-server 127.0.0.1:7004 [cluster]
test1     75507  0.1  0.1  50604  4924 ?        Ssl  12:05   0:05 redis-server 127.0.0.1:7005 [cluster]
$ kill 75492

看下能否获取 key

$ redis-cli -c -p 7000
127.0.0.1:7000> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7005
"bar"
127.0.0.1:7005> CLUSTER NODES
946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004@17004 slave b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 0 1515043424000 5 connected
0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005@17005 myself,master - 0 1515043424000 7 connected 10923-16383
c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003@17003 slave 68770484f2bc9c472f89d04b8862397e234e96e7 0 1515043424000 4 connected
68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000@17000 master - 0 1515043424847 1 connected 0-5460
543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002@17002 master,fail - 1515043164514 1515043163609 3 disconnected
b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001@17001 master - 0 1515043424545 2 connected 5461-10922

原 master 7002 的 slave 7005 被选为 master 提供服务了。

我们再来启动 7002

test1@test1:~/cluster/7002$ redis-server ./redis.conf
76595:C 04 Jan 13:26:47.191 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
76595:C 04 Jan 13:26:47.191 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=76595, just started
76595:C 04 Jan 13:26:47.191 # Configuration loaded
test1@test1:~/cluster/7002$ ps axu|grep redis
test1     75480  0.1  0.2  50604  7940 ?        Ssl  12:04   0:07 redis-server 127.0.0.1:7000 [cluster]
test1     75487  0.1  0.2  50604  7112 ?        Ssl  12:04   0:07 redis-server 127.0.0.1:7001 [cluster]
test1     75497  0.1  0.2  50604  6932 ?        Ssl  12:04   0:07 redis-server 127.0.0.1:7003 [cluster]
test1     75502  0.1  0.2  50604  6924 ?        Ssl  12:04   0:07 redis-server 127.0.0.1:7004 [cluster]
test1     75507  0.1  0.1  52652  4952 ?        Ssl  12:05   0:07 redis-server 127.0.0.1:7005 [cluster]
test1     76596  0.4  0.2  52652  8224 ?        Ssl  13:26   0:00 redis-server 127.0.0.1:7002 [cluster]

可以看到，

7002

自动成为了

7005

的

slave

$ redis-cli -c -p 7000
127.0.0.1:7000> CLUSTER NODES
c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003@17003 slave 68770484f2bc9c472f89d04b8862397e234e96e7 0 1515043779000 4 connected
0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005@17005 master - 0 1515043779504 7 connected 10923-16383
543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002@17002 slave 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 0 1515043779000 7 connected
b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001@17001 master - 0 1515043779000 2 connected 5461-10922
946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004@17004 slave b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 0 1515043779706 5 connected
68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000@17000 myself,master - 0 1515043778000 1 connected 0-5460

3.2. 加入新节点

我们新加入一个节点 7006，看看发生什么

~cluster/7006$ redis-server ./redis.conf
76704:C 04 Jan 13:39:13.718 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
76704:C 04 Jan 13:39:13.718 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=76704, just started
76704:C 04 Jan 13:39:13.718 # Configuration loaded

$ redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000
>>> Adding node 127.0.0.1:7006 to cluster 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots: (0 slots) slave
replicates 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:7006 to make it join the cluster.
[OK] New node added correctly.

查看新节点

7006

$ redis-trib.rb check 127.0.0.1:7006
>>> Performing Cluster Check (using node 127.0.0.1:7006)
M: 28d79f857ae17e7d3f9d0c8f3b301d276c214471 127.0.0.1:7006
slots: (0 slots) master
0 additional replica(s)
S: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots: (0 slots) slave
replicates 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

新加入的

7006

默认作为了

master

，不过

slot

是 0，也就是说目前的 7006 还无法提供存储。

redis cluster 不会对新加入的节点自动进行迁移，我们需要手动

reshard

。

$ redis-trib.rb reshard 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
M: 28d79f857ae17e7d3f9d0c8f3b301d276c214471 127.0.0.1:7006
slots: (0 slots) master
0 additional replica(s)
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots: (0 slots) slave
replicates 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

算下 slot 的平均值

16384/4 = 4096

，移动 4096 个比较合适。

How many slots do you want to move (from 1 to 16384)? 4096

接收节点的

node ID

，填入

7006

的即可。

What is the receiving node ID? 28d79f857ae17e7d3f9d0c8f3b301d276c214471

可以从特定节点移动一些 slot 给 7006，也可以使用所有节点。

Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:

我这里选择 all，重新调整后节点 7006 的 slot 为

slots:0-1364,5461-6826,10923-12181 (3990 slots)

$ redis-trib.rb check 127.0.0.1:7006
>>> Performing Cluster Check (using node 127.0.0.1:7006)
M: 28d79f857ae17e7d3f9d0c8f3b301d276c214471 127.0.0.1:7006
slots:0-1364,5461-6826,10923-12181 (3990 slots) master
0 additional replica(s)
S: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots: (0 slots) slave
replicates 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:12182-16383 (4202 slots) master
1 additional replica(s)
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:1365-5460 (4096 slots) master
1 additional replica(s)

3.3. 为节点指定 master

假设有新节点 7007，想为它指定 master 7005，该如何操作？

需要通过

--master-id

来指定

master

，具体命令是：

redis-trib.rb add-node --slave --master-id master节点ID new节点 master节点

redis-trib.rb add-node --slave --master-id 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7007 127.0.0.1:7005
>>> Send CLUSTER MEET to node 127.0.0.1:7007 to make it join the cluster.
Waiting for the cluster to join...
>>> Configure node as replica of 127.0.0.1:7005.
[OK] New node added correctly.

来验证下：

$ redis-trib.rb check 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 68770484f2bc9c472f89d04b8862397e234e96e7 127.0.0.1:7000
slots:1365-5460 (4096 slots) master
1 additional replica(s)
S: 5741612aaa0cb8f3427b50ff21ca55f08d7c33ce 127.0.0.1:7007
slots: (0 slots) slave
replicates 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
S: c546664cfa67b4cabdd89010b3081624af7c7888 127.0.0.1:7003
slots: (0 slots) slave
replicates 68770484f2bc9c472f89d04b8862397e234e96e7
M: 28d79f857ae17e7d3f9d0c8f3b301d276c214471 127.0.0.1:7006
slots:0-1364,5461-6826,10923-12181 (3990 slots) master
1 additional replica(s)
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:12182-16383 (4202 slots) master
1 additional replica(s)
S: 543beab779aa57c9c2ac8331373571afe6d7694e 127.0.0.1:7002
slots: (0 slots) slave
replicates 28d79f857ae17e7d3f9d0c8f3b301d276c214471
M: b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3 127.0.0.1:7001
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: 946fe907edafbcd929f442e5a692b4d10780c871 127.0.0.1:7004
slots: (0 slots) slave
replicates b18aa47b0b53ee1c4aad55a9d7e54becbf86e6a3
[OK] All nodes agree about slots configuration.

3.4. 移除节点

具体命令是：

redis-trib.rb del-node 127.0.0.1:7000 `<node-id>`

3.4.1 移除 master 节点

假设移除

7005

$ redis-trib.rb del-node 127.0.0.1:7000 0166a342ef0990ee8a5c3acbb6feb0a249e175ad
>>> Removing node 0166a342ef0990ee8a5c3acbb6feb0a249e175ad from cluster 127.0.0.1:7000
[ERR] Node 127.0.0.1:7005 is not empty! Reshard data away and try again.

由于 7005 中有数据，不能直接移除，需要先 reshard

redis-trib.rb reshard 127.0.0.1:7000
M: 0166a342ef0990ee8a5c3acbb6feb0a249e175ad 127.0.0.1:7005
slots:12182-16383 (4202 slots) master
How many slots do you want to move (from 1 to 16384)?

提示，我们要分多少个槽点，由于7005上有4202个槽点，所以这里填写4202

How many slots do you want to move (from 1 to 16384)? 4202

What is the receiving node ID?

提示我们，需要移动到哪个id上，假设7000：

What is the receiving node ID? 68770484f2bc9c472f89d04b8862397e234e96e7
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:

注意这里，从哪个 7005 节点去转移数据到 7000，所以填入 7005 的 ID：

Source node #1:0166a342ef0990ee8a5c3acbb6feb0a249e175ad
Source node #2:done
Do you want to proceed with the proposed reshard plan (yes/no)? yes

迁移结束后就可以通过之前的命令删除 7005 节点了。

3.4.2 移除 slave 节点

移除 slave 要相对容易下，执行前面的命令可直接删除。

四. 参考资料

Redis cluster tutorial

Redis Commands

集群教程