爬虫实例1-爬取新闻列表和发布时间

一、新建工程

scrapy startproject shop

二、Items.py文件代码：

import scrapy class ShopItem(scrapy.Item): title = scrapy.Field() time = scrapy.Field()

三、shopspider.py文件爬虫代码

# -*-coding:UTF-8-*-import scrapyfrom shop.items import ShopItem class shopSpider(scrapy.Spider): name = "shop" allowed_domains = ["news.xxxxxxx.xx.cn"] start_urls = ["http://news.xxxxx.xxx.cn/hunan/"] def parse(self,response):item = ShopItem()item['title'] = response.xpath("//div[@class='txttotwe2']/ul/li/a/text()").extract()item['time'] = response.xpath("//div[@class='txttotwe2']/ul/li/font/text()").extract()yield item

四、pipelines.py文件代码（打印出内容）：注意：如果在shopspider.py文件中打印出内容则显示的是unicode编码，而在pipelines.py打印出来的信息则是正常的显示内容。

class ShopPipeline(object): def process_item(self, item, spider): count=len(item['title']) print 'news count: ' ,count for i in range(0,count): print 'biaoti: '+item['title'][i] print 'shijian: '+item['time'][i] return item

五、爬取显示的结果：

root@kali:~/shop# scrapy crawl shop --nolognews count: 40biaoti: xxx建成国家食品安全示范城市shijian: (2017-06-16)biaoti: xxxx考试开始报名…………………… …………………..