Scrapy爬虫实战项目【001】 - 抓取猫眼电影TOP100

爬取猫眼电影TOP100

参考来源：静觅丨崔庆才的个人博客 https://cuiqingcai.com/5534.html

目的：使用Scrapy爬取猫眼电影TOP100并保存至MONGODB数据库

目标网址：http://maoyan.com/board/4?offset=0

分析/知识点：

1. 爬取难度：
   a. 入门级，网页结构简单，静态HTML，少量JS，不涉及AJAX；
   b. 处理分页需要用到正则；

2. MONGODB的update语句使用：
   a. update语句：具备查重/插入新数据功能，以title为查重标准

def process_item(self, item, spider):
self.db['movies'].update({'title': item['title']}, {'$set': item}, upsert=True) #注意upsert=True，更新并插入
return item

实际步骤：

1) 创建Scrapy项目/maoyan(spider)

Terminal: > scrapy startproject maoyan_movie
Terminal: > scrapy genspider maoyan maoyan.com/board/4?offset=

2) 配置settings.py文件

# MONGODB配置
MONGO_URI = 'localhost'
MONGO_DB = 'maoyan_movie'
...

# 启用MongoPipeline
ITEM_PIPELINES = {
'maoyan_movie.pipelines.MongoPipeline': 300,
}

3) 编写items.py文件

from scrapy import Item, Field

class MovieItem(Item):
title = Field()     #电影标题
actors = Field()    #演员
releasetime = Field()   #上映时间
cover_img = Field()     #缩略图
detail_page = Field()   #电影详情页url
score = Field()         #电影评分

4) 编写pipelines.py文件
根据Scrapy官方文档修改：https://doc.scrapy.org/en/latest/topics/item-pipeline.html?highlight=mongo

import pymongo

class MongoPipeline(object):
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db

@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DB')
)

def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]

def close_spider(self, spider):
self.client.close()

# !! 更新MONGODB，使用UPDATE方法，查重功能(以title字段进行判断)
def process_item(self, item, spider):
self.db['movies'].update({'title': item['title']}, {'$set': item}, upsert=True)
return item

5) 编写spiders > maoyan.py文件
注意：
a) 使用scrapy的css selector进行节点解析；
b) 获取电影缩略图url时，注意需要根据网页源代码写css选择器，和审查元素中看到的不同；

item['cover_img'] = movie.css('a.image-link img.board-img::attr(data-src)').extract_first()

c) 获取下一页节点时，直接使用xpath或css难以直接获得，需要使用正则匹配；

next = response.xpath('.').re_first(r'href="(.*?)">下一页</a>')

d) 完整代码如下：

from scrapy import Spider, Request
from maoyan_movie.items import MovieItem

class MaoyanSpider(Spider):
name = 'maoyan'
allowed_domains = ['maoyan.com/board/4?offset=']
start_urls = ['http://maoyan.com/board/4?offset=']

# 每部电影详情页的基本前缀url
base_url = 'http://maoyan.com'

# 下一页前缀url
next_base_url = 'http://maoyan.com/board/4'

def parse(self, response):
if response:
# 获取每页所有电影的节点
movies = response.css('dl.board-wrapper dd')  # 获取所有电影相关节点，切记！！不能加上extract()
item = MovieItem()
for movie in movies:
item['title'] = movie.css('p.name a::text').extract_first()
item['actors'] = movie.css('p.star::text').extract_first().strip()
item['releasetime'] = movie.css('p.releasetime::text').extract_first().strip()
item['score'] = movie.css('i.integer::text').extract_first() + movie.css(
'i.fraction::text').extract_first()
item['detail_page'] = self.base_url + movie.css('p.name a::attr(href)').extract_first()
item['cover_img'] = movie.css(
'a.image-link img.board-img::attr(data-src)').extract_first()  # 注意：需要根据网页源码写css选择器，和审查元素中的不同，估计是受JS影响
yield item

# 处理下一页
next = response.xpath('.').re_first(r'href="(.*?)">下一页</a>')
if next:
next_url = self.next_base_url + next
yield Request(url=next_url, callback=self.parse, dont_filter=True)

6) 运行结果

小结

1. 入门级项目，熟悉了Scrapy的基本使用流程；
2. Scrapy的css/xpath选择器、正则写法需要进一步熟悉；
3. pymongo的update语句，需要进一步熟练掌握。