Simple Python Queue with Redis
2013-03-28 16:19
47 查看
Simple Python Queue with Redis
Posted on June 3, 2012#python#redis#nosql
The following article shows how to use
redis to build a simplemulti-producer, multi-consumer Queue with an interface similar to thepythonstandardlib queue. With this queue you can easily share databetween multiple processes or offload time
consumig calculations to multiple workerprocesses.
To store the data we use the
redis list data type. Redis Lists storessimple strings sorted by insertion order.
The following redis commands are used:
rpush Insert an element at the tail of the list
blpop Get an element from the head of the list, block if list is empty
lpop Get an element from the head of the list, return nothing list is empty
llen Return the length of the list
The implementation uses the
redis-py library to talk to the server.
import redis class RedisQueue(object): """Simple Queue with Redis Backend""" def __init__(self, name, namespace='queue', **redis_kwargs):
"""The default connection parameters are: host='localhost', port=6379, db=0""" self.__db= redis.Redis(**redis_kwargs) self.key = '%s:%s' %(namespace, name) def qsize(self): """Return the approximate size of the queue.""" return self.__db.llen(self.key) def empty(self): """Return True if the queue is empty, False otherwise.""" return self.qsize() == 0 def put(self, item): """Put item into the queue.""" self.__db.rpush(self.key, item) def get(self, block=True, timeout=None): """Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available.""" if block: item = self.__db.blpop(self.key, timeout=timeout) else: item = self.__db.lpop(self.key) if item: item = item[1] return item def get_nowait(self): """Equivalent to get(False).""" return self.get(False)
Usage:
>>> from RedisQueue import RedisQueue >>> q = RedisQueue('test') >>> q.put('hello world')
Now if we have a look at the redis database with the
redis-cliclient it showsthe expected results:
redis 127.0.0.1:6379> keys * 1) "queue:test" redis 127.0.0.1:6379> type queue:test list redis 127.0.0.1:6379> llen queue:test (integer) 1 redis 127.0.0.1:6379> lrange queue:test 0 1 1) "hello world"
We can get the item from a different script with:
>>> from RedisQueue import RedisQueue >>> q = RedisQueue('test') >>> q.get() 'hello world'
A subsequent call of
q.get()will block until anotherone puts a new item intothe Queue.
The next step would be to an endoder/decoder (e.gpython-json) to theQueue so that you are not limited to send strings.
There alredy exists the nice and simple
hotqueue library which has thesame interface as the above example and provides encoding/decoding.
Other mentionable queue implementations with a redis backend are:
flask-redis A basic Message Queue with Redis for flask.
celery An asynchronous task queue/job queue based on distributedmessage passing. Much more advanced. Can be used with different storagebackends.
rq Simple python library for queueing jobs and processing them in the background with workers.
resque is a Redis-backed Ruby library for creating background jobs,placing them on multiple queues, and processing them later. Used at github. Includes a nice
monitoring web interface.
pyres A resque clone in python.
相关文章推荐
- python实现简单的redis 队列 SIMPLE PYTHON QUEUE WITH REDIS
- Python Tricks(七)—— 使用 list 仿真先入先出的队列(FIFOQueue)
- Python 多线程同步的另一种方法 Queue
- Python的http服务(SimpleHTTPServer,BaseHTTPServer,CGIHTTPServer)
- RabbitMQ(python实现)学习之二:Producer发送消息至多个消息队列queue(广播消息)
- python模块学习---Queue
- simple version of stack and queue 's ADT
- python Queue模块
- Python的SimpleHTTPServer
- Python进程间通信Queue实例解析
- python SimpleHTTPServer 快速共享文件
- Create a simple REST web service with Python--转载
- simple email python
- python多进程程序之间交换数据的两种办法--Queue和Pipe
- python题目-----queue by list
- Tech Tip: Really Simple HTTP Server with Python
- PYTHON queue
- python多线程--优先级队列(Queue)
- Quick share by "Python -m SimpleHTTPServer 8000"
- python3学习-Queue模块