爬虫学习笔记01-爬去猫眼电影TOP100榜单

爬虫学习笔记01-爬取猫眼电影TOP100榜单

#! python3

import requests
import csv
from bs4 import BeautifulSoup

url = 'http://maoyan.com/board/4?offset='

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
}

class maoyan:
def __init__(self, url):
self.url = url
self.csvfile = open('data.csv', 'w', newline='')
self.fieldnames = ['排名', '电影名', '演员', '上映时间', '评分']
self.writer = csv.DictWriter(self.csvfile, fieldnames=self.fieldnames)
self.writer.writeheader()
self.data = []

def set_url(self, num):
result = self.url + str(num)
return result

def parse_one_page(self, url):
try:
res = requests.get(url, headers=headers)
res.encoding = 'utf-8'
soup = BeautifulSoup(res.text, 'lxml')
dds = soup.select('dd')
if not dds:
return
for dd in dds:
index = dd.select(".board-index")[0].text
title = dd.select(".name")[0].text
actors = dd.select('.star')[0].text.strip()[3:]
time = dd.select('.releasetime')[0].text[5:]
score = dd.select('.score')[0].text
info = {
self.fieldnames[0]: index,
self.fieldnames[1]: title,
self.fieldnames[2]: actors,
self.fieldnames[3]: time,
self.fieldnames[4]: score
}
self.data.append(info)
except:
print('无法打开网页')

def write_to_csv(self):
for i in range(10):
url = self.set_url(i * 10)
self.parse_one_page(url)
if self.data:
self.writer.writerows(self.data)
self.csvfile.close()

if __name__ == '__main__':
m = maoyan(url)
m.write_to_csv()

文本方式打开，部分结果如下：

排名,电影名,演员,上映时间,评分
1,霸王别姬,"张国荣,张丰毅,巩俐",1993-01-01,9.6
2,肖申克的救赎,"蒂姆·罗宾斯,摩根·弗里曼,鲍勃·冈顿",1994-10-14(美国),9.5
3,罗马假日,"格利高里·派克,奥黛丽·赫本,埃迪·艾伯特",1953-09-02(美国),9.1
4,这个杀手不太冷,"让·雷诺,加里·奥德曼,娜塔莉·波特曼",1994-09-14(法国),9.5
5,教父,"马龙·白兰度,阿尔·帕西诺,詹姆斯·肯恩",1972-03-24(美国),9.3
6,泰坦尼克号,"莱昂纳多·迪卡普里奥,凯特·温丝莱特,比利·赞恩",1998-04-03,9.5
7,唐伯虎点秋香,"周星驰,巩俐,郑佩佩",1993-07-01(中国香港),9.2
8,千与千寻,"柊瑠美,入野自由,夏木真理",2001-07-20(日本),9.3
9,魂断蓝桥,"费雯·丽,罗伯特·泰勒,露塞尔·沃特森",1940-05-17(美国),9.2
10,乱世佳人,"费雯·丽,克拉克·盖博,奥利维娅·德哈维兰",1939-12-15(美国),9.1
11,天空之城,"寺田农,鹫尾真知子,龟山助清",1992,9.1
12,喜剧之王,"周星驰,莫文蔚,张柏芝",1999-02-13(中国香港),9.2
13,大闹天宫,"邱岳峰,毕克,富润生",1965-12-31,9.0
14,辛德勒的名单,"连姆·尼森,拉尔夫·费因斯,本·金斯利",1993-12-15(美国),9.2
15,音乐之声,"朱莉·安德鲁斯,克里斯托弗·普卢默,埃琳诺·帕克",1965-03-02(美国),9.0
......

Excel打开，部分结果如下图所示：