Reids Cluster 部署

目前，项目用的redis主备，感觉超不爽，很多局限性，特别是在 redis master 宕机之后，维护非常麻烦，寻思着弄一个redis集群，Redis3.0+后终于提供了cluster方案。

架构：Redis Cluster采用无中心结构，每个节点都保存数据和整个集群的状态

每个节点都和其他所有节点连接，这些连接保持活跃

使用gossip协议传播信息以及发现新节点

node不作为client请求的代理，client根据node返回的错误信息重定向请求

数据分布：预分桶

预分好16384个桶，根据 CRC16(key) mod 16384的值，决定将一个key放到哪个桶中

每个Redis物理结点负责一部分桶的管理，当发生Redis节点的增减时，调整桶的分布即可

例如，假设Redis Cluster三个节点A/B/C，则

Node A 包含桶的编号可以为: 0 到 5500.

Node B 包含桶的编号可以为: 5500 到 11000.

Node C包含桶的编号可以为: 11001 到 16384.

当发生Redis节点的增减时，调整桶的分布即可。

预分桶的方案介于“硬Hash”和“一致性Hash”之间，牺牲了一定的灵活性，但相比“一致性Hash“，数据的管理成本大大降低

安装配置

安装编译环境

$yum install gcc gcc-c++ gcc-g77 flex bison autoconf automake bzip2-devel zlib-devel ncurses-devel libjpeg-devel libpng-devel libtiff-devel freetype-devel pam-devel openssl-devel libxml2-devel gettext-devel pcre-devel tcl -y

安装高版本ruby (yum安装的版本过低)

卸载老版本:

$yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs
$wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.1.tar.gz $tar -zxvf ...
$./configure
$make && make install
$gem sources -r https://rubygems.org/ $gem sources -a https://ruby.taobao.org/ $gem sources –l

安装rubygems

$wget http://production.cf.rubygems.org/rubygems/rubygems-2.2.2.tgz $ruby setup.rb

安装redis

$wget http://download.redis.io/releases/redis-3.2.0.tar.gz $make MALLOC=libc
$cd src
$cp redis-server /usr/sbin/
$cp redis-cli /usr/sbin/
$cp redis-trib.rb /usr/sbin/

utils目录下有命令脚本的样例参考。

配置

创建用户和目录:

$adduser -M -U redis
$mkdir /var/lib/redis
$mkdir /etc/redis
$chown redis:redis /var/lib/redis
$chown redis:redis /etc/redis

修改系统配置:

$echo 1 > /proc/sys/vm/overcommit_memory
$vi /etc/sysctl.conf
添加vm.overcommit_memory=1

$echo 511 > /proc/sys/net/core/somaxconn

$echo never > /sys/kernel/mm/transparent_hugepage/enabled
$vi /etc/rc.local
添加echo never > /sys/kernel/mm/transparent_hugepage/enabled

编辑redis配置文件7000~7005
$vi /etc/redis/7000.conf

7000.conf(其它配置一致，port7000~7005)

$bind 127.0.0.1
$port 7000
$pidfile /var/run/redis/7000.pid
$loglevel notice
$logfile /var/log/redis/7000.log
$daemonize yes
$dir /var/lib/redis/7000

$cluster-enabled yes
$cluster-config-file nodes.conf
$cluster-node-timeout 5000
$appendonly yes
$appendfsync everysec

$repl-disable-tcp-nodelay yes

启动文件/etc/init.d/redis-cluster

>

#!/bin/bash

# Source function library.

. /etc/rc.d/init.d/functions

# Settings

PORT=7000
TIMEOUT=2000
NODES=6
REPLICAS=1

# You may want to put the above config parameters into config.sh in order to

# override the defaults without modifying this script.

if [ -a config.sh ]
then
source "config.sh"
fi

# Computed vars

ENDPORT=$((PORT+NODES))

if [ "$1" == "start" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
echo "Starting $PORT"
daemon --user redis "/usr/sbin/redis-server /etc/redis/${PORT}.conf"
PORT=$((PORT+1))
done
exit 0
fi

if [ "$1" == "create" ]
then
HOSTS=""
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
HOSTS="$HOSTS 127.0.0.1:$PORT"
PORT=$((PORT+1))
done
/usr/sbin/redis-trib.rb create --replicas $REPLICAS $HOSTS
exit 0
fi

if [ "$1" == "stop" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
echo "Stopping $PORT"
/usr/sbin/redis-cli -p $PORT shutdown nosave
PORT=$((PORT+1))
done
exit 0
fi

if [ "$1" == "watch" ]
then
while [ 1 ]; do
clear
date
PORTTMP=$PORT
while [ $((PORTTMP < ENDPORT)) != "0" ]; do
/usr/sbin/redis-cli -p $PORTTMP cluster nodes | head -30
PORTTMP=$((PORTTMP+1))
done
sleep 2
done
exit 0
fi

if [ "$1" == "tail" ]
then
INSTANCE=$2
PORT=$((PORT+INSTANCE))
tail -f /var/log/redis/${PORT}.log
exit 0
fi

if [ "$1" == "call" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
/usr/sbin/redis-cli -p $PORT $2 $3 $4 $5 $6 $7 $8 $9
PORT=$((PORT+1))
done
exit 0
fi

if [ "$1" == "clean" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
rm -rf /var/log/redis/${PORT}.log
rm -rf /var/lib/redis/${PORT}/*
PORT=$((PORT+1))
done
exit 0
fi

echo "Usage: $0 [start|create|stop|watch|tail|clean]"
echo "start       -- Launch Redis Cluster instances."
echo "create      -- Create a cluster using redis-trib create."
echo "stop        -- Stop Redis Cluster instances."
echo "watch       -- Show CLUSTER NODES output (first 30 lines) of first node."
echo "tail <id>   -- Run tail -f of instance at base port + ID."
echo "clean       -- Remove all instances data, logs, configs."

一些简单的命令

检查状态

$redis-trib.rb check 127.0.0.1:7000

添加master节点

$redis-trib.rb add-node 127.0.0.1:7006

node:新节点没有包含任何数据， 因为它没有包含任何slot。新加入的加点是一个主节点， 当集群需要将某个从节点升级为新的主节点时， 这个新节点不会被选中

为新节点分配slot

$redis-trib.rb reshard 127.0.0.1:7006

添加slave节点

步骤同上

redis-cli连接上新节点shell，输入命令：cluster replicate 对应master的mode-id

>cluster replicate 2b9ebcbd627ff0fd7a7bbcc5332fb09e72788835

删除slave节点

$redis-trib.rb del-node ip:port '<node-id>'

$redis-trib.rb del-node 127.0.0.1:7103 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Python3.5

Redis-py-cluster