Scrapy Redis源码 spider分析

下载的scrapy-redis源码中的spiders.py源码非常分析：

RedisSpider继承了Spider和RedisMixin这两个类，RedisMixin是用来从redis读取url的类。

当我们生成一个Spider继承RedisSpider时，调用setup_redis函数，这个函数会去连接redis数据库，然后会设置signals，一个是当spider空闲的时候(signal)，会调用spider_idle函数，这个函数调用schedule_next_request函数，然后抛出DontCloseSpider异常，保证spider是一直活着的状态；另一个signal是当抓到一个item时，item_scraped函数，这个函数会调用schedule_next_request函数。

from scrapy import Spider, signals
from scrapy.exceptions import DontCloseSpider

from . import connection

class RedisMixin(object):
"""Mixin class to implement reading urls from a redis queue."""
redis_key = None  # use default '<spider>:start_urls'

def setup_redis(self):
"""Setup redis connection and idle signal.

This should be called after the spider has set its crawler object.
"""
if not self.redis_key:
self.redis_key = '%s:start_urls' % self.name

self.server = connection.from_settings(self.crawler.settings)
# idle signal is called when the spider has no requests left,
# that's when we will schedule new requests from redis queue
self.crawler.signals.connect(self.spider_idle, signal=signals.spider_idle)
self.crawler.signals.connect(self.item_scraped, signal=signals.item_scraped)
self.log("Reading URLs from redis list '%s'" % self.redis_key)

def next_request(self):
"""Returns a request to be scheduled or none."""
use_set = self.settings.getbool('REDIS_SET')

if use_set:
url = self.server.spop(self.redis_key)
else:
url = self.server.lpop(self.redis_key)

if url:
return self.make_requests_from_url(url)

def schedule_next_request(self):
"""Schedules a request if available"""
req = self.next_request()
if req:
self.crawler.engine.crawl(req, spider=self)

def spider_idle(self):
"""Schedules a request if available, otherwise waits."""
self.schedule_next_request()
raise DontCloseSpider

def item_scraped(self, *args, **kwargs):
"""Avoids waiting for the spider to  idle before scheduling the next request"""
self.schedule_next_request()

class RedisSpider(RedisMixin, Spider):
"""Spider that reads urls from redis queue when idle."""

def _set_crawler(self, crawler):
super(RedisSpider, self)._set_crawler(crawler)
self.setup_redis()

我们下到的scrapy-redis源码中有自带一个example-project项目，这个项目包含3个spider，分别是dmoz, mycrawler_redis, myspider_redis，

dmoz（class DmozSpider(CrawlSpider)）这个爬虫继承的是CrawlSpider，它是用来说明Redis的持续性，当我们第一次运行dmoz爬虫，然后Ctrl-C停掉之后，再运行dmoz爬虫，之前的爬取记录是保留下来的。

myspider_redis ( class MySpider(RedisSpider) ) 这个爬虫继承了RedisSpider， 它能够支持分布式的抓取，采用的是basic spider，需要parse函数

mycrawler_redis ( class MyCrawler(RedisMixin, CrawlSpider) ) 这个爬虫，继承了RedisMixin和CrawlSpider，能够支持分布式的抓取，采用的是crawlSpider，需要设置Rule规则