python 爬虫scrapy框架练习

最近发现python中的scrapy库用来做爬虫是相当的好用，只要你有想法，就能实现一些抓取，本人的测试环境在ubuntu12.04下进行

第一步安装scrapy：

pip install Scrapy

easy_install scrapy

这两条命令完成了scrapy的安装，

下来我们用这些东西实现一个简单的抓取：

命令：scrapy

这个我们能看到下面出现好多的 命令这些命令是连起来用的

wangyu@ubuntu:~$ scrapy
Scrapy 0.12.0.2546 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  fetch         Fetch a URL using the Scrapy downloader
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command

这个命令可以让你看到生成一个文件夹在当前目录下：

scarpy startproject wo

wangyu@ubuntu:~/wo$ tree
.
├── scrapy.cfg
└── wo
    ├── __init__.py
    ├── items.py
    ├── pipelines.py
    ├── settings.py
    └── spiders
        └── __init__.py

2 directories, 6 files

我们可以通过这棵树看到我们当前的wo文件夹下有什么

我们先用vim来打开items.py

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/topics/items.html 
from scrapy.item import Item, Field

class WoItem(Item):
    # define the fields for your item here like:
    # name = Field()
    pass
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~

这个是当前的空文件，我们要进行一下编辑：

1 # Define here the models for your scraped items
  2 #-*-coding:utf-8-*-
  3 # See documentation in:
  4 # http://doc.scrapy.org/topics/items.html   5 
  6 from scrapy.item import Item, Field
  7 
  8 class WoItem(Item):
  9     # define the fields for your item here like:
 10      title = Field()
 11      link  = Field()
 12 #定义两个一个title，一个link
 13                                                                                       
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~                                                                                         
~

我添加了两句，一句是中文注释，在就是下面的类里面的东西

要建立一个Spider，你必须为scrapy.spider.BaseSpider创建一个子类，并确定三个主要的、强制的属性

name必须唯一

start_urls 爬虫开始爬的地方

parse()：爬虫的方法，调用时候传入从每一个URL传回的Response对象作为参数，response将会是parse方法的唯一的一个参数,

1 from scrapy.spider import BaseSpider   
  2 
  3 class wo(BaseSpider):
  4     name = "wo"
  5     allowed_wo =["jandan.net"]
  6     start_urls=[
  7             "http://jandan.net/new",
  8             "http://jandan.net/fml"
  9             ]
 10     def parse(self,response):
 11         filename=response.url.split("/"    )[-2]
 12         open(filename,'wb').write(respo    nse.body)
 13 
~                                          
~                                          
~                                          
~                                          
~

这个中我们以简单为例。

这时我们运行：scrapy crawl wo

wangyu@ubuntu:~/wo$ scrapy crawl wo
2013-09-21 20:53:40+0800 [scrapy] INFO: Scrapy 0.12.0.2546 started (bot: wo)
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Enabled extensions: TelnetConsole, SpiderContext, WebService, CoreStats, MemoryUsage, CloseSpider
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, DownloaderStats
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-21 20:53:40+0800 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-21 20:53:40+0800 [wo] INFO: Spider opened
2013-09-21 20:53:40+0800 [wo] DEBUG: Crawled (200) <GET http://jandan.net/fml> (referer: None)
2013-09-21 20:53:40+0800 [wo] DEBUG: Crawled (200) <GET http://jandan.net/new> (referer: None)
2013-09-21 20:53:40+0800 [wo] INFO: Closing spider (finished)
2013-09-21 20:53:40+0800 [wo] INFO: Spider closed (finished)

这个就是我们的运行结果，我们能看到这些东西，我们返回wo这个文件夹，会发现一个被名民为jandan的文件，但是打开发现全是代码。不能达到真的直接吧一个网页下载下来，我们在改改这些代码：

1 from scrapy.spider import BaseSpider
  2 
  3 class wo(BaseSpider):
  4     name = "wo"
  5     allowed_wo =["jiandan.net"]
  6     start_urls=[
  7              "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
  8                      "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
  9                     "http://jandan.net/ooxx"                                       
 10 
 11             ]
 12     def parse(self,response):
 13         filename=response.url.split("/")[-2]
 14         open(filename,'wb').write(response.body)
 15     #这个地方是给你的文件命名，在url最后的倒数地一个“/”之后的词语命名为你的文件名，下面调用写入文件
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~                                                                                      
~

再次运行：

wangyu@ubuntu:~/wo$ scrapy crawl wo

2013-09-21 21:21:16+0800 [scrapy] INFO: Scrapy 0.12.0.2546 started (bot: wo)
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Enabled extensions: TelnetConsole, SpiderContext, WebService, CoreStats, MemoryUsage, CloseSpider
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, DownloaderStats
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-21 21:21:16+0800 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-21 21:21:16+0800 [wo] INFO: Spider opened
2013-09-21 21:21:16+0800 [wo] DEBUG: Crawled (200) <GET http://jandan.net/ooxx> (referer: None)
2013-09-21 21:21:18+0800 [wo] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
2013-09-21 21:21:18+0800 [wo] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)
2013-09-21 21:21:18+0800 [wo] INFO: Closing spider (finished)
2013-09-21 21:21:18+0800 [wo] INFO: Spider closed (finished)

这个时候我能能看见在你这个项目的根文件夹下有三个文件叫做books和Resources和jiandan.net这个时候我们就完成了我们的下爬虫了