将数据保存到MongoDB,MySQL和图片下载

先来看一下流程图
数据持久化是最后一道步骤,由pipeline完成,我们可以在这里将爬取到的有效的信息保存到数据库,或是将需要的图片下载到本地.温习一下将数据保存到MongoDB,MySQL和图片下载的步骤.

准备工作

• 数据库:

MySQL事先建表,用来保存信息,而MongoDB则无需事先创建集合
在 items.py 中定义字段,例如:

class News163Item(scrapy.Item):
title = scrapy.Field()
href = scrapy.Field()
content = scrapy.Field()
pic_url = scrapy.Field()

• 在 setting.py 中:

注册管道类,设置权重

ITEM_PIPELINES = {
'news163.pipelines.ImgPipeLine': 300,
'news163.pipelines.MongoPipeline': 301,
'news163.pipelines.MysqlPipeline': 302,
}

提供连接数据库所需的信息

MONGO_URI = 'localhost'   #  MongoDB
MONGO_DB = 'wynews'

MYSQL_HOST = 'localhost'    # MySQL
MYSQL_PORT = 3306
MYSQL_DATABASE = 'news'
MYSQL_USER = 'root'
MYSQL_PWD = ''

图片路径

IMAGES_STORE = './mvs'

OK,准备工作完成先切入正题

书写管道类

写入MongoDB

1. init()方法： 初始化属性host和库名
2. 类方法from_crawler():获取配置内的 MONGO_URI 和 MONGO_DB ， 给属性赋值
3. open_spider():方法:连接数据库
4. process_item():方法:拦截item，插入数据
5. close_spider():关闭连接

例如：

import pymongo
class MongoPipeline(object):
def __init__(self, mongo_uri, mongo_db):
#初始化属性
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db

@classmethod
def from_crawler(cls, crawler):
#给属性赋值
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DB')
)

def open_spider(self, spider):
#连接数据库
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]

def process_item(self, item, spider):
self.db['n163'].insert(dict(item))  #方括号内是集合名
return item

def close_spider(self, spider):
self.client.close()

写入MySQL

1. init():初始化属性:host,端口,数据库,用户名,密码
2. 类方法from_crawler():给属性赋值
3. open_spider():连接数据库,获取cursor对象
4. process_item():拦截item,写SQL语句并执行插入数据
5. close_spider():关闭连接

例子:

from pymysql import connect
class MysqlPipeline(object):
def __init__(self, mysql_host, mysql_port, mysql_database, mysql_user, mysql_pwd):
self.mysql_host = mysql_host
self.mysql_port = mysql_port
self.mysql_database = mysql_database
self.mysql_user = mysql_user
self.mysql_pwd = mysql_pwd
@classmethod
def from_crawler(cls, crawler):
return cls(
mysql_host=crawler.settings.get('MYSQL_HOST'),
mysql_port=crawler.settings.get('MYSQL_PORT'),
mysql_database=crawler.settings.get('MYSQL_DATABASE'),
mysql_user=crawler.settings.get('MYSQL_USER'),
mysql_pwd=crawler.settings.get('MYSQL_PWD')
)

def open_spider(self, spider):
self.conn = connect(host=self.mysql_host, port=self.mysql_port, database=self.mysql_database,
user=self.mysql_user,
password=self.mysql_pwd, charset='utf8')
self.cur = self.conn.cursor()

def process_item(self, item, spider):
data = dict(item)
keys = ','.join(data.keys())
values = ','.join(['%s'] * len(data))
sql = 'insert into %s (%s) values (%s)' % ('news', keys, values)
self.cur.execute(sql, tuple(data.values()))
self.conn.commit()
return item

def close_spider(self, spider):
self.cur.close()
self.conn.close()

在process_item()构造的SQL语句不会受具体字段个数的影响

def process_item(self, item, spider):
data = dict(item)
keys = ','.join(data.keys())
values = ','.join(['%s'] * len(data))   # 按键值对个数生成占位符放在'()'中
sql = 'insert into %s (%s) values (%s)' % ('news', keys, values)    # 将表名,字段名和键值对个数*占位符 替换sql中的占位符
self.cur.execute(sql, tuple(data.values()))   # 拿出item中的所有值,替换占位符
self.conn.commit()
return item

下载图片

继承使用Scrapy中自带的管道类:ImagesPipeline

1. get_media_requests():拦截item,从中拿到与下载图片有关的信息并发出请求
2. file_path():写图片名(在setting.py中以指定下载路径),ps:windows文件名中不能有下列符

“?”、“、”、“╲”、“/”、“*”、““”、“”“、“<”、“>”、“|”

from scrapy.pipelines.images import ImagesPipeline
class ImgPipeLine(ImagesPipeline):
def get_media_requests(self, item, info):
yield scrapy.Request(url=item['pic_url'], meta={'title': item['title']})

def file_path(self, request, response=None, info=None):
file_name = request.meta['title'].replace('?', '') + '.jpg'
return file_name

def item_completed(self, results, item, info):
return item

• 点赞
• 收藏
• 分享
• 文章举报