爬虫参考资料

开源爬虫软件汇总：http://blog.chinaunix.net/uid-22414998-id-3774291.html

淘宝搜索-定向抓取漫谈 ：http://www.searchtb.com/2011/01/an-introduction-to-crawler.html

淘宝搜索-快速构建实时抓取集群 ：http://www.searchtb.com/2011/07/%E5%BF%AB%E9%80%9F%E6%9E%84%E5%BB%BA%E5%AE%9E%E6%97%B6%E6%8A%93%E5%8F%96%E9%9B%86%E7%BE%A4.html
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining
to monitoring and automated testing.

强大的scrapy爬虫框架（Python）：http://scrapy.org/

Python抓取框架：Scrapy的架构： http://www.biaodianfu.com/scrapy-architecture.html
使用scrapy进行大规模抓取：http://www.yakergong.net/blog/archives/500

Scrapy入门教程：http://www.cnblogs.com/txw1958/archive/2012/07/16/scrapy-tutorial.html

一个scrapy例子：https://github.com/scrapy/dirbot

一个分布式定向抓取集群的简单实现：https://github.com/agathewiky/spider-roach