调试Scrapy过程中的心得体会

1.大量抓取网页时出现“Memory Error”解决办法：设置一个队列，每当爬虫空闲时才向队列中放入请求，例如：

from scrapy import signals, Spider
from scrapy.xlib.pydispatch import dispatcher

class ExampleSpider(Spider):
name = "example"
start_urls = ['http://www.example.com/']

def __init__(self, *args, **kwargs):
super(ExampleSpider, self).__init__(*args, **kwargs)
# connect the function to the spider_idle signal
dispatcher.connect(self.queue_more_requests, signals.spider_idle)

def queue_more_requests(self, spider):
# this function will run everytime the spider is done processing
# all requests/items (i.e. idle)

# get the next urls from your database/file
urls = self.get_urls_from_somewhere()

# if there are no longer urls to be processed, do nothing and the
# the spider will now finally close
if not urls:
return

# iterate through the urls, create a request, then send them back to
# the crawler, this will get the spider out of its idle state
for url in urls:
req = self.make_requests_from_url(url)
self.crawler.engine.crawl(req, spider)

def parse(self, response):
pass

More info on the spider_idle signal: http://doc.scrapy.org/en/latest/topics/signals.html#spider-idle
More info on debugging memory leaks: http://doc.scrapy.org/en/latest/topics/leaks.html
P.S.还有一种限定爬取深度的方法（貌似在settings.py中？）待研究

2.如果请求的url不存在（404），则不会有response对象返回，爬虫什么也没做

3.编码问题

pubmed_spider.py中

import sys
reload(sys)
#python默认环境编码时ascii
sys.setdefaultencoding("utf-8")

保证抓取到的数据是utf8格式的

pipeline.py中file = codecs.open('/%s.txt' % (item['name']), mode = 'w',encoding='utf-8')将数据以utf8格式存储