《从零开始学Python网络爬虫》CH6

Case1 爬取PEXELS图片

# -*- coding: utf-8 -*-

import requests
import time
from bs4 import BeautifulSoup
import os

folder = 'Beauty'
if not os.path.exists(folder):
os.makedirs(folder)

n = 0
header = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}
url = 'https://www.pexels.com/search/beauty/'
response = requests.get(url, headers = header)
html = response.text
soup = BeautifulSoup(html, 'html.parser')
srcs = soup.select('article > a > img')

for src in srcs:
img = src.get('src')
print(img)
print(n)
res = requests.get(img, headers = header)
f = open(folder + '/' + str(n) + '.jpg', 'wb')
f.write(res.content)
f.close()
time.sleep(0.1)
n += 1

print('done')

Case2 爬取糗事百科网的用户地址信息

这里写代码片