redis cluster搭建

服务器信息：

redis版本 3.2.0

ruby版本 1.8.7.374

10.100.0.29 redis node 1

10.100.0.45 redis node 2

10.100.0.46 redis node 3

10.100.0.47 redis node 4

10.100.0.49 redis node 5

redis cluster 节点配置

protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
daemonize yes
supervised no
pidfile /var/run/redis.pid
loglevel notice
logfile "/var/log/rediscluster.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
cluster-enabled yes           #开启cluster支持
cluster-config-file nodes.conf
cluster-node-timeout 5000

1. 安装redis

# make PREFIX=/data/app/redis-3.2.0 && make install PREFIX=/data/app/redis-3.2.0

# cp -a utils /data/app/redis-3.2.0

# cp -a src /data/app/redis-3.2.0/

# mkdir -p /data/app/redis-3.2.0/conf/

将redis配置文件保存为/data/app/redis-3.2.0/conf/rediscluster.conf
一台机器启动了2个实例，另一个实例使用6380端口

2. 安装ruby redis模块

# yum install redis rubygems -y && gem install redis

3. 启动各节点，查看是否启动正常

# cd /data/app/redis-3.2.0

# bin/redis-server conf/rediscluster.conf

4. 创建节点

# src/redis-trib.rb create --replicas 1 10.100.0.29:6379 10.100.0.29:6380 10.100.0.45:6379 10.100.0.45:6380 10.100.0.46:6379 10.100.0.46:6380

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.100.0.29:6379
10.100.0.46:6379
10.100.0.45:6379
Adding replica 10.100.0.46:6380 to 10.100.0.29:6379
Adding replica 10.100.0.29:6380 to 10.100.0.46:6379
Adding replica 10.100.0.45:6380 to 10.100.0.45:6379
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots:10923-16383 (5461 slots) master
S: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
M: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) master
replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots:10923-16383 (5461 slots) master
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots: (0 slots) master
replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
M: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) master
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

5. 查看cluster状态

# src/redis-trib.rb info 10.100.0.29:6379

10.100.0.29:6379 (484c437d...) -> 0 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 0 keys | 5462 slots | 1 slaves.
10.100.0.45:6379 (12952b61...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

# src/redis-trib.rb check 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
S: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots: (0 slots) slave
replicates 12952b61c854a54112897a5bea35b8ddc6f04390
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

6. 测试

1、master停机漂移

停止10.100.0.45:6379,查看集群状态

经过查看，在经过一段时间后10.100.0.45:6380代替10.100.0.45:6379成为master

[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
[ERR] Sorry, can't connect to node 10.100.0.45:6379
*** WARNING: 10.100.0.45:6380 claims to be slave of unknown node ID 12952b61c854a54112897a5bea35b8ddc6f04390.
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 5 keys in 2 masters.
0.00 keys per slot on average.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 0 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.

2、 启动宕机的10.100.0.45:6379

原先宕机的redis实例成为了slave

[root@localhost redis-3.2.0]# src/redis-trib.rb check 10.100.0.29:6379
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots: (0 slots) slave
replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.

redis cluster扩展

将node 4 ，5上的4个实例加入集群

# src/redis-trib.rb add-node 10.100.0.47:6379 10.100.0.45:6379 #待加入的节点后跟已有节点

>>> Adding node 10.100.0.47:6379 to cluster 10.100.0.45:6379
>>> Performing Cluster Check (using node 10.100.0.45:6379)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots: (0 slots) slave
replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.100.0.47:6379 to make it join the cluster.
[OK] New node added correctly.

同理加入10.100.0.47:6380后查看集群状态

# src/redis-trib.rb check 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
slots: (0 slots) master
0 additional replica(s)
M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
slots: (0 slots) master
0 additional replica(s)
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots: (0 slots) slave
replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

发现2个节点都为master，另外加入的2个节点需要为slave

使用redis-trib.rb add-node --slave --master-id 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.49:6379 10.100.0.45:6379 # 指定成为某个master的slave

# src/redis-trib.rb add-node --slave --master-id 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.49:6379 10.100.0.45:6379

# src/redis-trib.rb add-node --slave --master-id 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.49:6380 10.100.0.45:6379

再次检查cluster状态

# src/redis-trib.rb info 10.100.0.29:6379

10.100.0.29:6379 (484c437d...) -> 4 keys | 5461 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 5461 slots | 1 slaves.
10.100.0.47:6379 (76d6d37d...) -> 0 keys | 0 slots | 1 slaves.
10.100.0.47:6380 (5ebaf183...) -> 0 keys | 0 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 10 keys in 5 masters.
0.00 keys per slot on average.

redis cluster负载均衡

自动rebalance

# src/redis-trib.rb rebalance 10.100.0.29:6379

额 负载平衡的时候会报错，不让移动

>>> Performing Cluster Check (using node 10.100.0.29:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 3 nodes. Total weight = 3
Moving 1003 slots from 10.100.0.45:6380 to 10.100.0.29:6379
###########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Moving 994 slots from 10.100.0.45:6380 to 10.100.0.46:6379
##################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

手动移动

# src/redis-trib.rb reshard 10.100.0.29:6379

>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:999-5460 (4462 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
S: 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a 10.100.0.49:6380
slots: (0 slots) slave
replicates 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:11922-16383 (4462 slots) master
1 additional replica(s)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots: (0 slots) slave
replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
S: 69a64bd82f5037fdad55d25c0b224845f297d521 10.100.0.49:6379
slots: (0 slots) slave
replicates 5ebaf1831f7c8e4c1293c87632968312f371d96a
M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
slots:0-998,5461-6461,10923-11921 (2999 slots) master
1 additional replica(s)
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
slots: (0 slots) master
1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:6462-10922 (4461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 10      #移动多少个slot,但是实际上不会移动那么多slot
What is the receiving node ID? 5ebaf1831f7c8e4c1293c87632968312f371d96a   #移动到哪个node
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all        #选择源node

Ready to move 10 slots.
Source nodes:
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:999-5460 (4462 slots) master
1 additional replica(s)
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:11922-16383 (4462 slots) master
1 additional replica(s)
M: 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 10.100.0.47:6379
slots:0-998,5461-6461,10923-11921 (2999 slots) master
1 additional replica(s)
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:6462-10922 (4461 slots) master
1 additional replica(s)
Destination node:
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
slots: (0 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 999 from 484c437d11e330023eb29268c6f00ebc5af409fd
Moving slot 1000 from 484c437d11e330023eb29268c6f00ebc5af409fd
Moving slot 1001 from 484c437d11e330023eb29268c6f00ebc5af409fd
Moving slot 11922 from 5501efd20bfcad76e193f3b00d0aeba3acdb0715
Moving slot 11923 from 5501efd20bfcad76e193f3b00d0aeba3acdb0715
Moving slot 6462 from 3df73cb44e750cd676e3996c97309e148b628abf
Moving slot 6463 from 3df73cb44e750cd676e3996c97309e148b628abf
Moving slot 0 from 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
Do you want to proceed with the proposed reshard plan (yes/no)? yes    #yes开始move slot
......

移动后查看cluster状态

...
M: 5ebaf1831f7c8e4c1293c87632968312f371d96a 10.100.0.47:6380
slots:0,999-1001,6462-6463,11922-11923 (8 slots) master
1 additional replica(s)
...

redis cluster node删除

删除node 4，5上的4个redis node

使用src/redis-trib.rb reshard 10.100.0.29:6379命令手动移动需要删除的node到指定的node，确保node为空

[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.47:6379 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5
>>> Removing node 76d6d37d6c2c33adacc45cbfbf49e134ec4b1dc5 from cluster 10.100.0.47:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.47:6380 5ebaf1831f7c8e4c1293c87632968312f371d96a
>>> Removing node 5ebaf1831f7c8e4c1293c87632968312f371d96a from cluster 10.100.0.47:6380
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.49:6379 69a64bd82f5037fdad55d25c0b224845f297d521
>>> Removing node 69a64bd82f5037fdad55d25c0b224845f297d521 from cluster 10.100.0.49:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb del-node 10.100.0.49:6380 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a
>>> Removing node 0ee117b8c72529f9b7038c69cfaa1c60cd7f952a from cluster 10.100.0.49:6380
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost redis-3.2.0]# src/redis-trib.rb info 10.100.0.29:6379
10.100.0.29:6379 (484c437d...) -> 4 keys | 4459 slots | 1 slaves.
10.100.0.45:6380 (5501efd2...) -> 5 keys | 7458 slots | 1 slaves.
10.100.0.46:6379 (3df73cb4...) -> 1 keys | 4467 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.
[root@localhost redis-3.2.0]# src/redis-trib.rb check 10.100.0.29:6379
>>> Performing Cluster Check (using node 10.100.0.29:6379)
M: 484c437d11e330023eb29268c6f00ebc5af409fd 10.100.0.29:6379
slots:1002-5460 (4459 slots) master
1 additional replica(s)
S: ee8ca5b7303392831ef753e3c07ccb4c086d82fe 10.100.0.29:6380
slots: (0 slots) slave
replicates 3df73cb44e750cd676e3996c97309e148b628abf
M: 5501efd20bfcad76e193f3b00d0aeba3acdb0715 10.100.0.45:6380
slots:1-998,5461-6461,10923-11921,11924-16383 (7458 slots) master
1 additional replica(s)
S: 12952b61c854a54112897a5bea35b8ddc6f04390 10.100.0.45:6379
slots: (0 slots) slave
replicates 5501efd20bfcad76e193f3b00d0aeba3acdb0715
S: 9ae373556ba70cd0fc24fdcba95c0fee3fd0201e 10.100.0.46:6380
slots: (0 slots) slave
replicates 484c437d11e330023eb29268c6f00ebc5af409fd
M: 3df73cb44e750cd676e3996c97309e148b628abf 10.100.0.46:6379
slots:0,999-1001,6462-10922,11922-11923 (4467 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.