您的位置：首页 > 理论基础 > 计算机网络

Python网络爬虫与信息提取-Day7-基于bs4库的HTML内容遍历方法

2017-08-26 13:46 761 查看

HTML基本格式
具有树形结构的文本信息
<>…</>构成了所属关系，形成了标签的树形结构

1.标签树的下行遍历

属性	说明
.contents	子节点的列表，将<tag>所有儿子节点存入列表
.children	子节点的迭代类型，与.contents类似，用于循环遍历儿子节点
.descendants	子孙节点的迭代类型，包含所有子孙节点，用于循环遍历

BeautifulSoup类型是标签树的根节点

>>> soup.head
<head><title>This is a python demo page</title></head>
>>> soup.head.contents
[<title>This is a python demo page</title>]
>>> soup.body.contents
['\n', The demo python introduces several python courses., '\n', Python is a wonderful general-purpose programming language. You can learn Python from novice to professional
by tracking the following courses:

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>., '\n']
>>> len(soup.body.contents)
5
>>> soup.body.contents[1]
The demo python introduces several python courses.

遍历儿子节点：

for child in soup.body.children:
print(child)

遍历子孙节点：

for child in soup.body.descendants:
print(child)

2.标签树的上行遍历

属性	说明
.parent	节点的父亲标签
.parents	节点先辈标签的迭代类型，用于循环遍历先辈节点

>>> soup.title.parent
<head><title>This is a python demo page</title></head>
>>> soup.html.parent
<html><head><title>This is a python demo page</title></head>
<body>
The demo python introduces several python courses.
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.
</body></html>
>>> soup.parent
>>>

>>> soup = BeautifulSoup(demo,"html.parser")
>>> for parent in soup.a.parents:
if parent is None:
print(parent)
else:
print(parent.name)

p
body
html
[document]
>>>

遍历所有先辈节点，包括soup本身，所以要区别判断

3.标签树的平行遍历

属性	说明
.next_sibling	返回按照HTML文本顺序的下一个平行节点标签
.previous_sibling	返回按照HTML文本顺序的上一个平行节点标签
.next_siblings	迭代类型，返回按照HTML文本顺序的后续所有平行节点标签
.previous_siblings	迭代类型，返回按照HTML文本顺序的前续所有平行节点

平行遍历发生在同一个父节点下的各节点间

>>> soup = BeautifulSoup(demo,"html.parser")
>>> soup.a.next_sibling
' and '
>>> soup.a.next_sibling.next_sibling
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
>>> soup.a.previous_sibling
'Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n'
>>> soup.a.previous_sibling.previous_sibling
>>> soup.a.parent
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.
>>>

遍历后续节点：

for sibling in soup.a.next_sibling:
print(sibling)

遍历前续节点：

for sibling in soup.a.previous_sibling:
print(sibling)

总结：

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签： python 网络爬虫

相关文章推荐

新的分享

章节导航