爬虫学习(三) Scrapy框架入门与豆瓣电影爬虫

知识点笔记

Scrapy框架知识点简记

官方说明文档：http://scrapychs.readthedocs.io/zh_CN/latest/intro/tutorial.html（写在前面，必须要看的）

Scrapy是Python开发的一个快速web抓取的框架。需要提前安装一些库才可以安装Scrapy库。(1.lxml 2.zope.interface 3.Twisted 4.pyOpenSSL 5.pywin32 6.Scrapy)

Scrapy生成项目的方法：在cmd中用命令：scrapy startproject XXX

如下图所示：



在项目中引用库的方法如下所示：

from scrapy.spiders import CrawlSpider

from scrapy .http import Request

from scrapy.selector import Selector

from movie.items import MovieItem

其生成的项目结构如图所示。在spiders目录下编写我们的爬虫文件



这里需要说明的是，在pycharm中运行scrapy框架需要一些设置，所以创建了main文件，其内部代码如图所示。（这里参考了其他相关博客，见链接http://blog.csdn.net/ck4438707/article/details/52076220）



scrapy各个文件功能，见官方说明文档不再赘述。

实例（豆瓣电影top250爬虫）

目标：

url: http://movie.douban.com/top2502.

爬取内容：电影名称，信息，评分以及语录

保存为CSV文件

主程序（spider下面的内容）：

#-*-coding:utf-8 -*-
from scrapy.spiders import CrawlSpider
from scrapy .http import Request
from scrapy.selector import Selector
from movie.items import MovieItem

class test(CrawlSpider):
name = "douban"
redis_key='douban:start_urls'
start_urls=['http://movie.douban.com/top250']
url='https://movie.douban.com/top250'

def parse(self,response):
#print response.body
item = MovieItem()
selector = Selector(response)
Movies = selector.xpath('//div[@class="info"]')
for eachMovie in Movies:
title = eachMovie.xpath('div[@class="hd"]/a/span/text()').extract()
fullTitle = ''
for each in title:
fullTitle += each
movieInfo = eachMovie.xpath('div[@class="bd"]/p/text()').extract()
star = eachMovie.xpath('div[@class="bd"]/div[@class="star"]/span[@class="rating_num"]/text()').extract()[0]
quote = eachMovie.xpath('div[@class="bd"]/p[@class="quote"]/span/text()').extract()
if quote:
quote = quote[0]
else:
quote = ''
item['title'] = fullTitle
item['movieInfo'] = ';'.join(movieInfo)
item['star'] = star
item['quote'] = quote
yield item
nextlink = selector.xpath('//span[@class="next"]/link/@href').extract()
if nextlink:
nextlink = nextlink[0]
print nextlink
yield Request(self.url + nextlink, callback=self.parse)

items.py

from scrapy import Item, Field
class MovieItem(Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = Field()
movieInfo = Field()
star = Field()
quote = Field()

seting.py

# -*- coding: utf-8 -*-

# Scrapy settings for movie project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html #     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html #     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 
BOT_NAME = 'movie'

SPIDER_MODULES = ['movie.spiders']
NEWSPIDER_MODULE = 'movie.spiders'
USER_AGENT='Mozilla/5.0 (Windows NT 10.0; Win64; x64)'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'movie (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = {
#    'movie.middlewares.MovieSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = {
#    'movie.middlewares.MyCustomDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html #EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See http://scr 8d61
apy.readthedocs.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'movie.pipelines.MoviePipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
FEED_URI=u'file:///G:/movie.csv'
FEED_FORMAT='CSV'