Redis是单线程还是双线程？适用场景及经验总结 road
2014-03-23 04:18
615 查看
尊重劳动成功，转载请附上本文链接及作者：jianfeng_soft@163.com （road）
最近（周五）在与朋友聊天时，聊到他们做游戏服务端用到一个排行榜的功能，自然接下来我们就聊到了redis，但由于线程问题，网上普遍看到都是单线程，不管是单线程或但即便是双线程，hashes..hgetall...也会出现cpu瓶颈，对于解决此瓶颈，目前能想到的就是在前面先打一层MC(Midnight
Commander)来减少请求量
个人总结，在试用redis前，首先需要准确定位你是使用redis 的 cache呢还是使用她的存储？
cache实用于前面提到的排行榜，统计短数据量的场景，因为用redis的list存排行榜数据，如果内存不够可以T出。
另外我大致以个人经验简述一下redis的场景和注意事项，希望能给未有过实用经验的小伙伴带来一些帮助，redis 只适合应用于大量写入复杂的数据结构，或者简单的数据结构但要持久化的场景，并且最关键是数据容量必须要小于内存，
因此必须要在使用前做硬件内存容量规划，比如未来的增长预算，当前存入一个hashes kv需要占用多少空间，依此来做预算。
即便做了持久化，内存碎片也很严重，意思就是即便数据被移到磁盘（rdb或aof），所有数据始终都会仍然存在于占用内存 （因此你需要考虑是否需要做持久化？做持久化是选择rdb还是aof？）
rdb： 好处是只会短期内影响读写速度，但是可能丢数据 ；
aof： 每一次写都导致有性能下级，并且恢复时间要长，比mc的延迟更高，特别是做了持久化，即便不做持久化也比mc更延迟。
综合评测redis优缺点
优点：内存操作特别快，高级的数据结构可用，
缺点：磁盘操作慢，偶尔延迟，耗内存
最后附带以下是段落是摘自网络对redis线程（单双之谜）进行贴码透析：
多进程分析：
[html] view
plaincopy
int rdbSaveBackground(char *filename) {
pid_t childpid;
long long start;
if (server.rdb_child_pid != -1) return REDIS_ERR;
server.dirty_before_bgsave = server.dirty;
server.lastbgsave_try = time(NULL);
start = ustime();
if ((childpid = fork()) == 0) {
int retval;
/* Child */
if (server.ipfd > 0) close(server.ipfd);
if (server.sofd > 0) close(server.sofd);
retval = rdbSave(filename);
if (retval == REDIS_OK) {
size_t private_dirty = zmalloc_get_private_dirty();
if (private_dirty) {
redisLog(REDIS_NOTICE,
"RDB: %zu MB of memory used by copy-on-write",
private_dirty/(1024*1024));
}
}
exitFromChild((retval == REDIS_OK) ? 0 : 1);
} else {
/* Parent */
server.stat_fork_time = ustime()-start;
if (childpid == -1) {
server.lastbgsave_status = REDIS_ERR;
redisLog(REDIS_WARNING,"Can't save in background: fork: %s",
strerror(errno));
return REDIS_ERR;
}
redisLog(REDIS_NOTICE,"Background saving started by pid %d",childpid);
server.rdb_save_time_start = time(NULL);
server.rdb_child_pid = childpid;
updateDictResizePolicy();
return REDIS_OK;
}
return REDIS_OK; /* unreached */
}
redis对cache落盘时有save和bgsave两种方式。bgsave将会fork()出一个后台进程。关键在于这一行代码：if ((childpid = fork()) == 0)
而redis的多线程是怎么实现的呢？
在bio.h里有几行关键的代码。
[html] view
plaincopy
/* Background job opcodes */
#define REDIS_BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */
#define REDIS_BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */
#define REDIS_BIO_NUM_OPS 2
这里，redis定义了background IO job数量为2。可以大胆地猜测，background job的数量会随着这个server软件的复杂度而增加，甚至，会把bio提升成为background job层面。
那么，这两个线程是如何生成的呢？
[html] view
plaincopy
/* Initialize the background system, spawning the thread. */
void bioInit(void) {
pthread_attr_t attr;
pthread_t thread;
size_t stacksize;
int j;
/* Initialization of state vars and objects */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
pthread_mutex_init(&bio_mutex[j],NULL);
pthread_cond_init(&bio_condvar[j],NULL);
bio_jobs[j] = listCreate();
bio_pending[j] = 0;
}
/* Set the stack size as by default it may be small in some system */
pthread_attr_init(&attr);
pthread_attr_getstacksize(&attr,&stacksize);
if (!stacksize) stacksize = 1; /* The world is full of Solaris Fixes */
while (stacksize < REDIS_THREAD_STACK_SIZE) stacksize *= 2;
pthread_attr_setstacksize(&attr, stacksize);
/* Ready to spawn our threads. We use the single argument the thread
* function accepts in order to pass the job ID the thread is
* responsible of. */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
redisLog(REDIS_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
}
}
这里的
[html] view
plaincopy
/* Ready to spawn our threads. We use the single argument the thread
* function accepts in order to pass the job ID the thread is
* responsible of. */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
redisLog(REDIS_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
}
创建了REDIS_BIO_NUM_OPS个线程。
这两个线程的入口函数都是bioProcessBackgroundJobs，那reds怎么知道到底是要对应REDIS_BIO_CLOSE_FILE类型还是对应REDIS_BIO_AOF_FSYNC呢？
我们看一下代码。
[html] view
plaincopy
void *bioProcessBackgroundJobs(void *arg) {
struct bio_job *job;
unsigned long type = (unsigned long) arg;
sigset_t sigset;
pthread_detach(pthread_self());
pthread_mutex_lock(&bio_mutex[type]);
/* Block SIGALRM so we are sure that only the main thread will
* receive the watchdog signal. */
sigemptyset(&sigset);
sigaddset(&sigset, SIGALRM);
if (pthread_sigmask(SIG_BLOCK, &sigset, NULL))
redisLog(REDIS_WARNING,
"Warning: can't mask SIGALRM in bio.c thread: %s", strerror(errno));
while(1) {
listNode *ln;
/* The loop always starts with the lock hold. */
if (listLength(bio_jobs[type]) == 0) {
pthread_cond_wait(&bio_condvar[type],&bio_mutex[type]);
continue;
}
/* Pop the job from the queue. */
ln = listFirst(bio_jobs[type]);
job = ln->value;
/* It is now possible to unlock the background system as we know have
* a stand alone job structure to process.*/
pthread_mutex_unlock(&bio_mutex[type]);
/* Process the job accordingly to its type. */
if (type == REDIS_BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == REDIS_BIO_AOF_FSYNC) {
aof_fsync((long)job->arg1);
} else {
redisPanic("Wrong job type in bioProcessBackgroundJobs().");
}
zfree(job);
/* Lock again before reiterating the loop, if there are no longer
* jobs to process we'll block again in pthread_cond_wait(). */
pthread_mutex_lock(&bio_mutex[type]);
listDelNode(bio_jobs[type],ln);
bio_pending[type]--;
}
}
关键就在以下代码。
[html] view
plaincopy
/* Process the job accordingly to its type. */
if (type == REDIS_BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == REDIS_BIO_AOF_FSYNC) {
aof_fsync((long)job->arg1);
} else {
redisPanic("Wrong job type in bioProcessBackgroundJobs().");
}
最近（周五）在与朋友聊天时，聊到他们做游戏服务端用到一个排行榜的功能，自然接下来我们就聊到了redis，但由于线程问题，网上普遍看到都是单线程，不管是单线程或但即便是双线程，hashes..hgetall...也会出现cpu瓶颈，对于解决此瓶颈，目前能想到的就是在前面先打一层MC(Midnight
Commander)来减少请求量
个人总结，在试用redis前，首先需要准确定位你是使用redis 的 cache呢还是使用她的存储？
cache实用于前面提到的排行榜，统计短数据量的场景，因为用redis的list存排行榜数据，如果内存不够可以T出。
另外我大致以个人经验简述一下redis的场景和注意事项，希望能给未有过实用经验的小伙伴带来一些帮助，redis 只适合应用于大量写入复杂的数据结构，或者简单的数据结构但要持久化的场景，并且最关键是数据容量必须要小于内存，
因此必须要在使用前做硬件内存容量规划，比如未来的增长预算，当前存入一个hashes kv需要占用多少空间，依此来做预算。
即便做了持久化，内存碎片也很严重，意思就是即便数据被移到磁盘（rdb或aof），所有数据始终都会仍然存在于占用内存 （因此你需要考虑是否需要做持久化？做持久化是选择rdb还是aof？）
rdb： 好处是只会短期内影响读写速度，但是可能丢数据 ；
aof： 每一次写都导致有性能下级，并且恢复时间要长，比mc的延迟更高，特别是做了持久化，即便不做持久化也比mc更延迟。
综合评测redis优缺点
优点：内存操作特别快，高级的数据结构可用，
缺点：磁盘操作慢，偶尔延迟，耗内存
最后附带以下是段落是摘自网络对redis线程（单双之谜）进行贴码透析：
多进程分析：
[html] view
plaincopy
int rdbSaveBackground(char *filename) {
pid_t childpid;
long long start;
if (server.rdb_child_pid != -1) return REDIS_ERR;
server.dirty_before_bgsave = server.dirty;
server.lastbgsave_try = time(NULL);
start = ustime();
if ((childpid = fork()) == 0) {
int retval;
/* Child */
if (server.ipfd > 0) close(server.ipfd);
if (server.sofd > 0) close(server.sofd);
retval = rdbSave(filename);
if (retval == REDIS_OK) {
size_t private_dirty = zmalloc_get_private_dirty();
if (private_dirty) {
redisLog(REDIS_NOTICE,
"RDB: %zu MB of memory used by copy-on-write",
private_dirty/(1024*1024));
}
}
exitFromChild((retval == REDIS_OK) ? 0 : 1);
} else {
/* Parent */
server.stat_fork_time = ustime()-start;
if (childpid == -1) {
server.lastbgsave_status = REDIS_ERR;
redisLog(REDIS_WARNING,"Can't save in background: fork: %s",
strerror(errno));
return REDIS_ERR;
}
redisLog(REDIS_NOTICE,"Background saving started by pid %d",childpid);
server.rdb_save_time_start = time(NULL);
server.rdb_child_pid = childpid;
updateDictResizePolicy();
return REDIS_OK;
}
return REDIS_OK; /* unreached */
}
redis对cache落盘时有save和bgsave两种方式。bgsave将会fork()出一个后台进程。关键在于这一行代码：if ((childpid = fork()) == 0)
而redis的多线程是怎么实现的呢？
在bio.h里有几行关键的代码。
[html] view
plaincopy
/* Background job opcodes */
#define REDIS_BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */
#define REDIS_BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */
#define REDIS_BIO_NUM_OPS 2
这里，redis定义了background IO job数量为2。可以大胆地猜测，background job的数量会随着这个server软件的复杂度而增加，甚至，会把bio提升成为background job层面。
那么，这两个线程是如何生成的呢？
[html] view
plaincopy
/* Initialize the background system, spawning the thread. */
void bioInit(void) {
pthread_attr_t attr;
pthread_t thread;
size_t stacksize;
int j;
/* Initialization of state vars and objects */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
pthread_mutex_init(&bio_mutex[j],NULL);
pthread_cond_init(&bio_condvar[j],NULL);
bio_jobs[j] = listCreate();
bio_pending[j] = 0;
}
/* Set the stack size as by default it may be small in some system */
pthread_attr_init(&attr);
pthread_attr_getstacksize(&attr,&stacksize);
if (!stacksize) stacksize = 1; /* The world is full of Solaris Fixes */
while (stacksize < REDIS_THREAD_STACK_SIZE) stacksize *= 2;
pthread_attr_setstacksize(&attr, stacksize);
/* Ready to spawn our threads. We use the single argument the thread
* function accepts in order to pass the job ID the thread is
* responsible of. */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
redisLog(REDIS_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
}
}
这里的
[html] view
plaincopy
/* Ready to spawn our threads. We use the single argument the thread
* function accepts in order to pass the job ID the thread is
* responsible of. */
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
redisLog(REDIS_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
}
创建了REDIS_BIO_NUM_OPS个线程。
这两个线程的入口函数都是bioProcessBackgroundJobs，那reds怎么知道到底是要对应REDIS_BIO_CLOSE_FILE类型还是对应REDIS_BIO_AOF_FSYNC呢？
我们看一下代码。
[html] view
plaincopy
void *bioProcessBackgroundJobs(void *arg) {
struct bio_job *job;
unsigned long type = (unsigned long) arg;
sigset_t sigset;
pthread_detach(pthread_self());
pthread_mutex_lock(&bio_mutex[type]);
/* Block SIGALRM so we are sure that only the main thread will
* receive the watchdog signal. */
sigemptyset(&sigset);
sigaddset(&sigset, SIGALRM);
if (pthread_sigmask(SIG_BLOCK, &sigset, NULL))
redisLog(REDIS_WARNING,
"Warning: can't mask SIGALRM in bio.c thread: %s", strerror(errno));
while(1) {
listNode *ln;
/* The loop always starts with the lock hold. */
if (listLength(bio_jobs[type]) == 0) {
pthread_cond_wait(&bio_condvar[type],&bio_mutex[type]);
continue;
}
/* Pop the job from the queue. */
ln = listFirst(bio_jobs[type]);
job = ln->value;
/* It is now possible to unlock the background system as we know have
* a stand alone job structure to process.*/
pthread_mutex_unlock(&bio_mutex[type]);
/* Process the job accordingly to its type. */
if (type == REDIS_BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == REDIS_BIO_AOF_FSYNC) {
aof_fsync((long)job->arg1);
} else {
redisPanic("Wrong job type in bioProcessBackgroundJobs().");
}
zfree(job);
/* Lock again before reiterating the loop, if there are no longer
* jobs to process we'll block again in pthread_cond_wait(). */
pthread_mutex_lock(&bio_mutex[type]);
listDelNode(bio_jobs[type],ln);
bio_pending[type]--;
}
}
关键就在以下代码。
[html] view
plaincopy
/* Process the job accordingly to its type. */
if (type == REDIS_BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == REDIS_BIO_AOF_FSYNC) {
aof_fsync((long)job->arg1);
} else {
redisPanic("Wrong job type in bioProcessBackgroundJobs().");
}
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- 【经验总结】C#常用线程同“.NET研究”步方法应用场景和实现原理
- redis 数据类型详解 以及 redis适用场景场合
- java 常用集合list与Set、Map区别及适用场景总结
- java 常用集合list与Set、Map区别及适用场景总结
- redis 数据类型详解 以及 redis适用场景场合
- redis 数据类型详解 以及 redis适用场景场合
- 国内外三个不同领域巨头分享的Redis实战经验及使用场景
- [经验总结]在ATL中通过GIT让工作线程访问连接点
- 统计工作总结——Echars常用图表适用场景分析
- 国内外三个不同领域巨头分享的Redis实战经验及使用场景
- java 常用集合list与Set、Map区别及适用场景总结
- 关于“幽灵架构”的总结：适用场景与方法重载
- [原]Redis使用场景及使用经验
- 互斥锁与自旋锁的区别（自旋锁不是通过休眠使进程阻塞，而是在获取锁之前一直处于忙等。适用场景：锁被持有时间短，并且线程不希望在重新调度上花费太多成本）
- NoSQL数据库：Redis适用场景及产品定位
- NoSQL数据库：Redis适用场景及产品定位（转）
- Redis应用场景总结
- 【Java线程】volatile的适用场景
- 国内外三个不同领域巨头分享的Redis实战经验及使用场景(一)