Codis集群部署测试

Codis安装部署2台服务器没有HA版本

项目地址： https://github.com/wandoulabs/codis 中文说明文档： https://github.com/wandoulabs/codis/blob/master/doc/tutorial_zh.md
参考地址：
http://0xffff.me/blog/2014/11/11/codis-de-she-ji-yu-shi-xian-part-2/ http://www.cnblogs.com/xuanzhi201111/p/4425194.html http://blog.csdn.net/freewebsys/article/details/44100919
个人：广州-李惟忠

架构图：




Codis 并不太适合 key 少，但是 value 特别大的应用， 而且你的 key 越少， value 越大，最后就会退化成单个 redis 的模型 （性能还不如 raw redis），所以 Codis 更适合海量 Key,

value比较小 （<= 1 MB) 的应用。

机器与应用列表：
操作系统：CentOS6.5

IP: 172.17.3.133 hostname: codis-133 apps: zookeeper, codis_proxy_1, codis_config, codis_server_master,slave
IP: 172.17.3.134 hostname: codis-134 apps: codis_server_master,slave

备注：由于是仿真测试，一台机器跑多个应用，如生产环境，只需把应用分开部署即可，另外zookeeper需要3台搭建集群环境，还需要2台搭建keepalived+haproxy。

一，安装zookeeper

1，配置hosts文件 (所有机器上配置)
vim /etc/hosts
172.17.3.133 codis-133
172.17.3.134 codis-134

2，安装java环境
ZooKeeper 要求 JAVA 的环境才能运行，并且需要 JAVA6 以上的版本，可以从 SUN 官网上下载，并对 JAVA 环境变量进行设置。
yum -y install java-1.7.0-openjdk-devel
java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
3，安装zookeeper
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz tar zxvf zookeeper-3.4.6.tar.gz
mv zookeeper-3.4.6 /usr/local/zookeeper
mkdir -p /data/zookeeper/{data,logs}
配置zoo.cfg
vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

4，启动zookeeper
/usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@localhost conf]# /usr/local/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: standalone

设置开机启动
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start

二，安装codis集群

1，安装go
设置环境变量
vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=/usr/local/codis
source /etc/profile
下载安装go
cd /usr/local/

wget http://golangtc.com/static/go/go1.4.2.linux-amd64.tar.gz
tar -zxvf go1.4.2.linux-amd64.tar.gz
go version
go version go1.4.2 linux/amd64

2，安装依赖环境
yum groupinstall "Development Tools"

3，安装codis
yum install -y git
go get github.com/wandoulabs/codis #这个需要几分钟下载共30M文件
package github.com/wandoulabs/codis
imports github.com/wandoulabs/codis
imports github.com/wandoulabs/codis: no buildable Go source files in /usr/local/codis/src/github.com/wandoulabs/codis
cd $GOPATH/src/github.com/wandoulabs/codis
[root@localhost codis]# pwd
/usr/local/codis/src/github.com/wandoulabs/codis
#执行编译测试脚本，编译go和reids。
./bootstrap.sh #这个需要十几分钟共下载50M文件
make gotest
# 将编译好后，把bin目录和一些脚本复制过去/usr/local/codis目录下：
mkdir -p /usr/local/codis/{logs,conf,scripts}
mkdir -p /data/codis_server/{logs,conf,data}
cp -rf bin /usr/local/codis/
cp sample/config.ini /usr/local/codis/conf/
cp sample/redis_conf/6381.conf /data/codis_server/conf/
cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/*.sh /usr/local/codis/scripts/
cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/usage.md /usr/local/codis/scripts/
cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.13
cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.21
ln -s /usr/local/codis/bin/redis-cli-2.8.21 /usr/local/codis/bin/redis-cli

4. 配置codis_proxy_1 ( codis-133 机器上配置)
cd /usr/local/codis
vim config.ini
zk=localhost:2181
product=bitauto-codis
proxy_id=proxy_1
net_timeout=5
dashboard_addr=localhost:18087
coordinator=zookeeper

5. 修改配置文件，启动codis-server服务.
cd /data/codis_server/conf/
mv 6381.conf 6379.conf.bak
grep -Ev "^#|^$" 6379.conf.bak >6379.conf
vim 6379.conf
修改如下参数: (生产环境，参数适当进行调整)
daemonize yes
timeout 300
pidfile /var/run/redis_6379.pid
port 6379
logfile "/data/codis_server/logs/codis_6379.log"
save 900 1
save 300 10
save 60 10000
dbfilename 6379.rdb
dir /data/codis_server/data
appendfilename "6379_appendonly.aof"
appendfsync everysec

具体配置文件如下：
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 0
loglevel notice
logfile "/data/codis_server/logs/redis_6379.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename 6379.rdb
dir /data/codis_server/data
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 2gb
maxmemory-policy allkeys-lru
appendonly no
appendfilename "6379_appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

复制6380配置文件
cp 6379.conf 6380.conf
sed -i 's/6379/6380/g' 6380.conf

添加内核参数
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
sysctl -p
内核参数说明如下：
overcommit_memory文件指定了内核针对内存分配的策略，其值可以是0、1、2。
0， 表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。
1， 表示内核允许分配所有的物理内存，而不管当前的内存状态如何。
2， 表示内核允许分配超过所有物理内存和交换空间总和的内存

启动codis-server服务
nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf &
nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf &
[root@localhost conf]#nohup /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf &
[1] 15944
[root@localhost conf]# _._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 2.8.21 (575eed4b/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in stand alone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 15944
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'

[15944] 24 Jun 18:23:22.255 # Server started, Redis version 2.8.21
[15944] 24 Jun 18:23:22.256 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with

Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the

setting after a reboot. Redis must be restarted after THP is disabled.
[15944] 24 Jun 18:23:22.256 * The server is now ready to accept connections on port 6379

6. 查看一下启动流程：
cat usage.md
0. start zookeeper
1. change config items in config.ini
2. ./start_dashboard.sh
3. ./start_redis.sh
4. ./add_group.sh
5. ./initslot.sh
之后可以通过浏览器给group分配solt，如果solt分配错误，可以通过浏览器的Migrate Slot重新分配，这个solt迁移需要一点时间，也可以Auto Rebalance平均分配，分片等操作一定要在开启proxy之前。
6. ./start_proxy.sh
7. ./set_proxy_online.sh
8. open browser to http://localhost:18087/admin

这只是一个参考，有些顺序不是必须的，但启动dashboard前，必须启动zookeeper服务，这是必须的，后面有很多操作，都可以在web页面完成，例如添加/删除组，添加/删除redis实例等

7. 修改脚本，启动 dashboard。( 只需在一台机器上启动即可。172.17.3.133上启动 ，后续大部分操作都可以在面板上操作)
cat /usr/local/codis/scripts/start_dashboard.sh
#!/bin/sh
CODIS_HOME=/usr/local/codis

nohup $CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/dashboard.log dashboard --addr=:18087 --http-log=$CODIS_HOME/logs/requests.log

&>/dev/null &

启动dashboard
cd /usr/local/codis/scripts/
sh start_dashboard.sh

8. 启动codis-server master , slave 实例 ，以上步骤已经启动，不在描述(在第5步，此时还没有分主从，只是分别在3.133和3.134上启动2个实例6379和6380)。

9. 通过管理页面添加组ID，为组添加主从实例，一个组里只能有一个redis-master： http://172.17.3.133:18087（最好用Firefox浏览器或者谷歌浏览器） 登录http://172.17.3.133:18087/admin/,添加2个组，组里面有2个实例，一个主一个从，默认每个组里面第一个实例是主
group_1
172.17.3.133:6379 master
172.17.3.134:6380 slave
group_2
172.17.3.134:6379 master
172.17.3.133:6380 slave

10. 修改脚本，初始化槽 ( 172.17.3.133 机器上配置 ，注意这个需要在添加group后操作，这里的脚本是solt槽1024分成了2半，group1和group2个512个)cat /usr/local/codis/scripts/initslot.sh#!/bin/shCODIS_HOME=/usr/local/codisecho "slots initializing..."$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini slot init -fecho "done"
执行初始化槽脚本：sh initslot.sh slots initializing...{ "msg": "OK", "ret": 0 }done

11.测试一下redis-master和redis-slave是否正常同步数据了：
这个需要用redis-cli测试：
[root@localhost conf]# /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli -h 172.17.3.133 -p 6379
172.17.3.133:6379> ping
PONG
172.17.3.133:6379> set name foo
OK
172.17.3.133:6379> get name
"foo"
172.17.3.133:6379> quit
[root@localhost conf]# /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli -h 172.17.3.134 -p 6380
172.17.3.134:6380> get name
"foo"
172.17.3.134:6380> set test 123
(error) READONLY You can't write against a read only slave.
172.17.3.134:6380> quit
[root@localhost conf]#
也可以通过日志查看
root@localhost conf]# [15825] 24 Jun 18:59:42.717 * SLAVE OF 172.17.3.133:6379 enabled (user request)
[15825] 24 Jun 18:59:43.199 * Connecting to MASTER 172.17.3.133:6379
[15825] 24 Jun 18:59:43.199 * MASTER <-> SLAVE sync started
[15825] 24 Jun 18:59:43.200 * Non blocking connect for SYNC fired the event.
[15825] 24 Jun 18:59:43.202 * Master replied to PING, replication can continue...
[15825] 24 Jun 18:59:43.203 * Partial resynchronization not possible (no cached master)
[15825] 24 Jun 18:59:43.204 * Full resync from master: 65c0b5c071fab1b52892a5efe7fd422bbf84f391:1
[15825] 24 Jun 18:59:43.289 * MASTER <-> SLAVE sync: receiving 18 bytes from master
[15825] 24 Jun 18:59:43.289 * MASTER <-> SLAVE sync: Flushing old data
[15825] 24 Jun 18:59:43.290 * MASTER <-> SLAVE sync: Loading DB in memory
[15825] 24 Jun 18:59:43.290 * MASTER <-> SLAVE sync: Finished with success

12. 配置codis-ha服务，主从自动切换。( 随便找个节点机器上配置即可,此环境中在172.17.3.134机器上配置 )
cd /usr/local
go get github.com/ngaut/codis-ha #ngaut大概3M左右
cd /usr/local/codis/src/github.com/ngaut
cp -r codis-ha /usr/local/
cd /usr/local/codis-ha
go build

创建启动脚本，启动codis-ha服务
vim start_codis_ha.sh
codis-ha --codis-config=172.17.3.133:18087 -log-level="info" --productName=bitauto-codis &> ./logs/codis-ha.log &

mkdir logs
sh start_codis_ha.sh

13. 修改start_proxy.sh，启动codis-proxy服务 ( 在172.17.3.133上配置)
[root@codis-133 scripts]# cat start_proxy.sh
#!/bin/sh

CODIS_HOME=/usr/local/codis

echo "shut down codis_proxy_1..."
$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy offline codis_proxy_1
echo "done"

echo "start new codis_proxy_1..."
nohup $CODIS_HOME/bin/codis-proxy --log-level info -c $CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/codis_proxy_1.log --cpu=4 --addr=0.0.0.0:19000 --http-

addr=0.0.0.0:11000 &
echo "done"

echo "sleep 3s"
sleep 3
tail -n 30 $CODIS_HOME/logs/codis_proxy_1.log

cat set_proxy_online.sh
#!/bin/sh

CODIS_HOME=/usr/local/codis
echo "set codis_proxy_1 online"
$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy online codis_proxy_1
echo "done"

启动codis-proxy
./start_proxy.sh

上线codis_proxy_1
./set_proxy_online.sh

14，最后设置开机启动
172.17.3.133上配置如下
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf
/usr/local/codis/scripts/start_dashboard.sh
/usr/local/codis/scripts/initslot.sh(一次性的，开机不需要启动)
/usr/local/codis/scripts/start_proxy.sh
/usr/local/codis/scripts/set_proxy_online.sh

172.17.3.134上配置如下
vim /etc/rc.local
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf
/usr/local/codis-ha/start_codis_ha.sh

15，附加文件redis服务启动脚本
#!/bin/sh
#Configurations injected by install_server below....

EXEC=/usr/local/codis/bin/codis-server
CLIEXEC=/usr/local/codis/bin/redis-cli
PIDFILE=/var/run/redis_6379.pid
CONF="/data/codis_server/conf/6379.conf"
REDISPORT="6379"
###############
# SysV Init Information
# chkconfig: - 58 74
# description: redis_6379 is the redis daemon.
### BEGIN INIT INFO
# Provides: redis_6379
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Should-Start: $syslog $named
# Should-Stop: $syslog $named
# Short-Description: start and stop redis_6379
# Description: Redis daemon
### END INIT INFO

case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis server..."
$EXEC $CONF
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $REDISPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Redis to shutdown ..."
sleep 1
done
echo "Redis stopped"
fi
;;
status)
PID=$(cat $PIDFILE)
if [ ! -x /proc/${PID} ]
then
echo 'Redis is not running'
else
echo "Redis is running ($PID)"
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Please use start, stop, restart or status as first argument"
;;
esac