redis cluster 集群搭建

1 cluster模式简介

redis有Redis Sentinel、Redis Cluster两种集群方式，从redis3.0开始才支持Redis Cluster模式，常见部署方式为3主3从。3个从redis与三个主redis一一对应，集群内置数据自动分片机制，集群内部将所有的key经过hash后映射到16384个Slot中，集群中的每个Redis Instance负责其中的一部分的Slot的读写。从下图中可以看到数据集群中的数据走向。

 

2 环境搭建

2.1 部署环境

由于环境有限，在同一台机器上搭建伪集群，部署如下：

master	slave
192.168.8.16:7000	192.168.8.16:7003
192.168.8.16:7001	192.168.8.16:7004
192.168.8.16:7002	192.168.8.16:7005

 

2.2 redis 安装：

[code]# wget http://download.redis.io/releases/redis-3.2.6.tar.gz

# tar xzf redis-3.2.6.tar.gz

# cd redis-3.2.6

# make

# make isntall

2.3 配置并启动redis

1、在redis-3.2.6目录下新建cluster/7000、cluster/7001、cluster/7002、cluster/7003、cluster/7004、cluster/7005目录

2、将redis-3.2.6目录下的redis.conf拷贝到cluster/7000目录下，配置做如下调整：

        bind 192.168.8.16 127.0.0.1
        port 7000
        pidfile /var/run/redis_7000.pid
        appendonly yes
        cluster-enabled yes
        cluster-config-file /root/home/redis-3.2.6/cluster/7000/nodes.conf
        dir /root/home/redis-3.2.6/cluster/7000/
        cluster-node-timeout 15000

3、cluster/7001、cluster/7002、cluster/7003、cluster/7004、cluster/7005目录的redis.conf可以参考上面的方式做调整

4、分别启动6个实例：./redis-server /root/home/redis-3.2.6/cluster/7000/redis.conf

cluster 相关参数说明：

1、cluster-enabled yes  #开启集群
2、cluster-config-file nodes-6379.conf  #集群配置信息文件，由Redis自行更新，不用手动配置。每个节点都有一个集群配置文件用于持久化保存集群信息，需确保与运行中实例的配置文件名    不冲突。
3、cluster-node-timeout 15000  #节点互连超时时间，毫秒为单位
4、cluster-slave-validity-factor 10  #在进行故障转移的时候全部slave都会请求申请为master，但是有些slave可能与master断开连接一段时间了导致数据过于陈旧，不应该被提升为master。该参数就是用来判断slave节点与master断线的时间是否过长。判断方法是：比较slave断开连接的时间和(node-timeout * slave-validity-factor)+ repl-ping-slave-period如果节点超时时间为三十秒, 并且slave-validity-factor为10，假设默认的repl-ping-slave-period是10秒，即如果超过310秒slave将不会尝试进行故障转移
5、cluster-migration-barrier 1  #master的slave数量大于该值，slave才能迁移到其他孤立master上，如这个参数被设为2，那么只有当一个主节点拥有2个可工作的从节点时，它的一个从节点才会尝试迁移。
6、cluster-require-full-coverage yes  #集群全部的slot有节点负责，集群状态才为ok并提供服务。设置为no可以在slot没有全部分配的时候提供服务。不建议打开该配置，这样会造成分区的时候，小分区的master一直在接受写请求，而造成很长时间数据不一致。
7、Appendonly是一种能够提供非常好的持久化的模式，例如使用默认的Fsync方案，Redis能在发生服务器电源故障或操作系统仍然正常运行但Redis进程莫名挂掉的情况下，只丢失1秒的数据

2.4 安装ruby和rubygems

安装ruby

[code][root@localhost local]# tar zxvf  ruby-2.4.0.tar.gz

[root@localhost local]# cd ruby-2.4.0

[root@localhost local]#./configure && make && make install

[root@localhost local]# ruby --version

ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

安装rubygems

[code][root@localhost]# tar zxvf rubygems-3.0.2.tgz

[root@localhost ]# cd rubygems-3.0.2

[root@localhost rubygems-3.0.2]# ruby setup.rb

[root@localhost rubygems-3.0.2]# gem --version

3.0.2
[root@dg1 local]# gem install redis -v 4.0.0.rc1 --pre

2.5 创建集群

./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

这个命令就代表把这些个redis实例创建为一个集群 上面命令中的1代表主节点和从节点的比值是多少，如果12个主节点，6个从节点那么我们的比值就是2，那么我们是3主3从，所以这个比值是1，而且前三个一定是主节点，redis就是这样规定的。

[code][root@localhost src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
/usr/local/lib/ruby/gems/2.4.0/gems/redis-4.0.0.rc1/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: 446ece0004f762cde28d2637ce94718b51b50cf6 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 70806e108250977c0e0b771fad60ff126ff1b4d0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 76bbacde830abfc73ada8635c57cae0426bf370a 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
S: d881644f76e1c164d2c90eacf150020d2133503e 127.0.0.1:7003
replicates 446ece0004f762cde28d2637ce94718b51b50cf6
S: 123250d99c8f85fb3a762fdd7e73e13e01ecf370 127.0.0.1:7004
replicates 70806e108250977c0e0b771fad60ff126ff1b4d0
S: 57003dbce6d92a056488fbcd3b6c0590b5e7d6cd 127.0.0.1:7005
replicates 76bbacde830abfc73ada8635c57cae0426bf370a
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 446ece0004f762cde28d2637ce94718b51b50cf6 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 57003dbce6d92a056488fbcd3b6c0590b5e7d6cd 192.168.8.16:7005
slots: (0 slots) slave
replicates 76bbacde830abfc73ada8635c57cae0426bf370a
S: d881644f76e1c164d2c90eacf150020d2133503e 192.168.8.16:7003
slots: (0 slots) slave
replicates 446ece0004f762cde28d2637ce94718b51b50cf6
M: 70806e108250977c0e0b771fad60ff126ff1b4d0 192.168.8.16:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 123250d99c8f85fb3a762fdd7e73e13e01ecf370 192.168.8.16:7004
slots: (0 slots) slave
replicates 70806e108250977c0e0b771fad60ff126ff1b4d0
M: 76bbacde830abfc73ada8635c57cae0426bf370a 192.168.8.16:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群信息：

[code]127.0.0.1:7001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:39
cluster_my_epoch:34
cluster_stats_messages_sent:242027
cluster_stats_messages_received:151659

127.0.0.1:7001> cluster nodes
d881644f76e1c164d2c90eacf150020d2133503e 192.168.8.16:7003 slave 446ece0004f762cde28d2637ce94718b51b50cf6 0 1551946384970 38 connected
57003dbce6d92a056488fbcd3b6c0590b5e7d6cd 192.168.8.16:7005 slave 76bbacde830abfc73ada8635c57cae0426bf370a 0 1551946386991 39 connected
123250d99c8f85fb3a762fdd7e73e13e01ecf370 192.168.8.16:7004 slave 70806e108250977c0e0b771fad60ff126ff1b4d0 0 1551946383958 34 connected
70806e108250977c0e0b771fad60ff126ff1b4d0 192.168.8.16:7001 myself,master - 0 0 34 connected 5461-10922
76bbacde830abfc73ada8635c57cae0426bf370a 192.168.8.16:7002 master - 0 1551946385978 39 connected 10923-16383
446ece0004f762cde28d2637ce94718b51b50cf6 192.168.8.16:7000 master - 0 1551946382949 38 connected 0-5460

使用集群的方式登录，并作测试：

[code][root@localhost src]# ./redis-cli -c -p 7001
127.0.0.1:7001> set test_50key test50value
-> Redirected to slot [1727] located at 192.168.8.16:7000
OK
192.168.8.16:7000> get test_50key
"test50value"

  Redis Cluster 管理和测试 

 2.6 可能碰到的问题

1、[root@localhost src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

/usr/local/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- redis (LoadError)

from /usr/local/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:54:in `require'

from ./redis-trib.rb:25:in `<main>'

缺少redis库。需要安装 redis-4.0.0.rc1.gem

 

2、[root@localhost src]# gem install -l redis-4.0.0.rc1.gem

ERROR: Could not find a valid gem 'redis-4.0.0.rc1.gem' (>= 0) in any repository

[root@localhost src]# gem sources -l

*** CURRENT SOURCES ***

[root@localhost src]# gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

Error fetching https://gems.ruby-china.org/:

SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert protocol version (https://gems.ruby-china.org/specs.4.8.gz)

如果出现上述错误可以使用http的源

[root@localhost src]# gem sources --add http://gems.ruby-china.com/ --remove https://rubygems.org/

http://gems.ruby-china.com/ added to sources

source https://rubygems.org/ not present in cache

[root@localhost src]# gem sources -l

*** CURRENT SOURCES ***

http://gems.ruby-china.com/

 

3、[root@localhost src]# gem install -l redis-4.0.0.rc1.gem

ERROR: Could not find a valid gem 'redis-4.0.0.rc1.gem' (>= 0) in any repository

需要更新一下源即可

[root@localhost src]# gem sources -u

source cache successfully updated

[root@localhost src]# gem install redis -v 4.0.0.rc1 --pre

Fetching redis-4.0.0.rc1.gem

Successfully installed redis-4.0.0.rc1

Parsing documentation for redis-4.0.0.rc1

Installing ri documentation for redis-4.0.0.rc1

Done installing documentation for redis after 2 seconds

1 gem installed

 

4、Waiting for the cluster to join一直等待解决办法

[root@localhost src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

>>> Creating cluster

/usr/local/lib/ruby/gems/2.4.0/gems/redis-4.0.0.rc1/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated

>>> Performing hash slots allocation on 6 nodes...

Using 3 masters:

127.0.0.1:7000

127.0.0.1:7001

127.0.0.1:7002

Adding replica 127.0.0.1:7003 to 127.0.0.1:7000

Adding replica 127.0.0.1:7004 to 127.0.0.1:7001

Adding replica 127.0.0.1:7005 to 127.0.0.1:7002

M: 9b58ef401db80a4311639e021f10dafd1424a94d 127.0.0.1:7000

slots:0-5460 (5461 slots) master

M: 6b0d5f7436fe4f085313edf68cd56fa3ae9c3852 127.0.0.1:7001

slots:5461-10922 (5462 slots) master

M: 91d6a1595d78aa9719933a0e620d50bd3e723ef5 127.0.0.1:7002

slots:10923-16383 (5461 slots) master

S: 01a0e9d786980ead9d7f737c2d88cf3195c724d8 127.0.0.1:7003

replicates 9b58ef401db80a4311639e021f10dafd1424a94d

S: 9b58ef401db80a4311639e021f10dafd1424a94d 127.0.0.1:7004

replicates 6b0d5f7436fe4f085313edf68cd56fa3ae9c3852

S: 9b58ef401db80a4311639e021f10dafd1424a94d 127.0.0.1:7005

replicates 91d6a1595d78aa9719933a0e620d50bd3e723ef5

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join.......................................................................................................................................................................................

需要将所有集群中的redis.conf中的bind

由 bind 127.0.0.1 修改为 bind 192.168.8.16 127.0.0.1