Python爬虫包 BeautifulSoup 学习(三) 实例
2016-12-22 21:46
549 查看
一步一步构建一个爬虫实例,抓取糗事百科的段子
先不用beautifulsoup包来进行解析
第一步,访问网址并抓取源码
第二步,利用正则表达式提取信息
首先先观察源码中,你需要的内容的位置以及如何识别
然后用正则表达式去识别读取
注意正则表达式中的 . 是不能匹配\n的,所以需要设置一下匹配模式。
第三步,修正数据并保存到文件中
第四步,将多个页面下的内容都抓取下来
使用BeautifulSoup对源码进行解析
这是用BeautifulSoup去抓取书本以及其价格的代码
可以通过对比得出到bs4对标签的读取以及标签内容的读取
(因为我自己也没有学到这一部分,目前只能依葫芦画瓢地写)
先不用beautifulsoup包来进行解析
第一步,访问网址并抓取源码
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 16:16:08 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 20:17:13 import urllib import urllib2 import re import os if __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() print content.decode('utf-8')
第二步,利用正则表达式提取信息
首先先观察源码中,你需要的内容的位置以及如何识别
然后用正则表达式去识别读取
注意正则表达式中的 . 是不能匹配\n的,所以需要设置一下匹配模式。
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 16:16:08 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 20:17:13 import urllib import urllib2 import re import os if __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) items = re.findall(regex, content) # 提取数据 # 注意换行符,设置 . 能够匹配换行符 for item in items: print item
第三步,修正数据并保存到文件中
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 16:16:08 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 21:41:32 import urllib import urllib2 import re import os if __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) items = re.findall(regex, content) # 提取数据 # 注意换行符,设置 . 能够匹配换行符 path = './qiubai' if not os.path.exists(path): os.makedirs(path) count = 1 for item in items: #整理数据,去掉\n,将<br/>换成\n item = item.replace('\n', '').replace('<br/>', '\n') filepath = path + '/' + str(count) + '.txt' f = open(filepath, 'w') f.write(item) f.close() count += 1
第四步,将多个页面下的内容都抓取下来
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 16:16:08 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 20:17:13 import urllib import urllib2 import re import os if __name__ == '__main__': # 访问网址并抓取源码 path = './qiubai' if not os.path.exists(path): os.makedirs(path) user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) count = 1 for cnt in range(1, 35): print '第' + str(cnt) + '轮' url = 'http://www.qiushibaike.com/textnew/page/' + str(cnt) + '/?s=4941357' try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() # print content # 提取数据 # 注意换行符,设置 . 能够匹配换行符 items = re.findall(regex, content) # 保存信息 for item in items: # print item #整理数据,去掉\n,将<br/>换成\n item = item.replace('\n', '').replace('<br/>', '\n') filepath = path + '/' + str(count) + '.txt' f = open(filepath, 'w') f.write(item) f.close() count += 1 print '完成'
使用BeautifulSoup对源码进行解析
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 16:16:08 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 21:34:02 import urllib import urllib2 import re import os from bs4 import BeautifulSoup if __name__ == '__main__': url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) # print response.read() soup_packetpage = BeautifulSoup(response, 'lxml') items = soup_packetpage.find_all("div", class_="content") for item in items: try: content = item.span.string except AttributeError as e: print e exit() if content: print content + "\n"
这是用BeautifulSoup去抓取书本以及其价格的代码
可以通过对比得出到bs4对标签的读取以及标签内容的读取
(因为我自己也没有学到这一部分,目前只能依葫芦画瓢地写)
# -*- coding: utf-8 -*- # @Author: HaonanWu # @Date: 2016-12-22 20:37:38 # @Last Modified by: HaonanWu # @Last Modified time: 2016-12-22 21:27:30 import urllib2 import urllib import re from bs4 import BeautifulSoup url = "https://www.packtpub.com/all" try: html = urllib2.urlopen(url) except urllib2.HTTPError as e: print e exit() soup_packtpage = BeautifulSoup(html, 'lxml') all_book_title = soup_packtpage.find_all("div", class_="book-block-title") price_regexp = re.compile(u"\s+\$\s\d+\.\d+") for book_title in all_book_title: try: print "Book's name is " + book_title.string.strip() except AttributeError as e: print e exit() book_price = book_title.find_next(text=price_regexp) try: print "Book's price is "+ book_price.strip() except AttributeError as e: print e exit() print ""
相关文章推荐
- Python爬虫包BeautifulSoup学习实例(五)
- Python爬虫包 BeautifulSoup 学习(五) 实例
- python3.4学习笔记(十三) 网络爬虫实例代码,使用pyspider抓取多牛投资吧里面的文章信息,抓取政府网新闻内容
- python爬虫实践学习实例之mysql数据库操作
- 【Python3.6爬虫学习记录】(二)使用BeautifulSoup爬取简单静态网页文章
- Python爬虫包 BeautifulSoup 学习(六) 递归抓取
- Python学习笔记 第二部分 - 正则表达式 与 爬虫小实例(抓取豆瓣电影中评分大于等于8分的影片)
- Python爬虫包 BeautifulSoup 学习(一) 简介与安装
- python爬虫【记录】BeautifulSoup 的用法遍历学习
- python学习(6):python爬虫之requests和BeautifulSoup的使用
- Python爬虫实例——基于BeautifulSoup和requests实现
- python爬虫实践学习实例之模拟登录
- python学习之爬虫:BeautifulSoup
- Python爬虫包 BeautifulSoup 学习(十一) CSS 选择器
- python3个人爬虫之:BeautifulSoup学习心得
- Python---BeautifulSoup 简单的爬虫实例
- Python爬虫包 BeautifulSoup 学习(四) bs基本对象与函数
- Python爬虫包 BeautifulSoup 学习(二) 异常处理
- Python爬虫包 BeautifulSoup 学习(八) parent等应用
- Python爬虫实例(3)-用BeautifulSoup爬取多个可翻页网页上的多张照片