Redis queue简单封装
2015-08-24 20:25
183 查看
The following article shows how to use redis to build a simple multi-producer, multi-consumer Queue with an interface similar to the python
standardlib queue. With this queue you can easily share data between multiple processes or offload time consumig calculations to multiple worker processes.
To store the data we use the redis list data type. Redis Lists stores simple strings sorted by insertion order.
The following redis commands are used:
rpush Insert an element at the tail of the list
blpop Get an element from the head of the list, block if list is empty
lpop Get an element from the head of the list, return nothing list is empty
llen Return the length of the list
The implementation uses the redis-py library to talk to the server.
Usage:
Now if we have a look at the redis database with the
the expected results:
We can get the item from a different script with:
A subsequent call of
Queue.
The next step would be to an endoder/decoder (e.g python-json) to the Queue so that you are not limited to send strings.
There alredy exists the nice and simple hotqueue library which has the same interface as the above example and provides encoding/decoding.
Other mentionable queue implementations with a redis backend are:
flask-redis A basic Message Queue with Redis for flask.
celery An asynchronous task queue/job queue based on distributed message passing. Much more advanced. Can be used with different storage
backends.
rq Simple python library for queueing jobs and processing them in the background with workers.
resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Used at github. Includes a nice monitoring web interface.
pyres A resque clone in python.
原网页:http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html
standardlib queue. With this queue you can easily share data between multiple processes or offload time consumig calculations to multiple worker processes.
To store the data we use the redis list data type. Redis Lists stores simple strings sorted by insertion order.
The following redis commands are used:
rpush Insert an element at the tail of the list
blpop Get an element from the head of the list, block if list is empty
lpop Get an element from the head of the list, return nothing list is empty
llen Return the length of the list
The implementation uses the redis-py library to talk to the server.
import redis class RedisQueue(object): """Simple Queue with Redis Backend""" def __init__(self, name, namespace='queue', **redis_kwargs): """The default connection parameters are: host='localhost', port=6379, db=0""" self.__db= redis.Redis(**redis_kwargs) self.key = '%s:%s' %(namespace, name) def qsize(self): """Return the approximate size of the queue.""" return self.__db.llen(self.key) def empty(self): """Return True if the queue is empty, False otherwise.""" return self.qsize() == 0 def put(self, item): """Put item into the queue.""" self.__db.rpush(self.key, item) def get(self, block=True, timeout=None): """Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available.""" if block: item = self.__db.blpop(self.key, timeout=timeout) else: item = self.__db.lpop(self.key) if item: item = item[1] return item def get_nowait(self): """Equivalent to get(False).""" return self.get(False)
Usage:
>>> from RedisQueue import RedisQueue >>> q = RedisQueue('test') >>> q.put('hello world')
Now if we have a look at the redis database with the
redis-cliclient it shows
the expected results:
redis 127.0.0.1:6379> keys * 1) "queue:test" redis 127.0.0.1:6379> type queue:test list redis 127.0.0.1:6379> llen queue:test (integer) 1 redis 127.0.0.1:6379> lrange queue:test 0 1 1) "hello world"
We can get the item from a different script with:
>>> from RedisQueue import RedisQueue >>> q = RedisQueue('test') >>> q.get() 'hello world'
A subsequent call of
q.get()will block until anotherone puts a new item into the
Queue.
The next step would be to an endoder/decoder (e.g python-json) to the Queue so that you are not limited to send strings.
There alredy exists the nice and simple hotqueue library which has the same interface as the above example and provides encoding/decoding.
Other mentionable queue implementations with a redis backend are:
flask-redis A basic Message Queue with Redis for flask.
celery An asynchronous task queue/job queue based on distributed message passing. Much more advanced. Can be used with different storage
backends.
rq Simple python library for queueing jobs and processing them in the background with workers.
resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Used at github. Includes a nice monitoring web interface.
pyres A resque clone in python.
原网页:http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html
相关文章推荐
- Redis常用的命令(五)------列表list
- redis持久化
- Redis配置全解
- Redis详细,config配置,win和linux下怎么玩,和mysql数据同步。
- Redis配置文件config详解
- 高可用、开源的Redis缓存集群方案 - 赵士杰的博客 - ITeye技术网站
- redis配置文件说明--redis.conf
- Redis官网下载步骤(含windows版)
- Redis C语言
- Python操作redis
- ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务
- 分析Redis架构设计
- Redis cluster基础介绍
- redis进阶
- Redis 使用 Eval 多个键值自增操作示例
- Redis在PHP中的应用
- ASP.NET MVC 使用Redis共享Session
- 深入理解 php redis pconnect
- Linux安装redis
- Redis分布式锁服务(八)