【读书笔记】《Python自然语言处理》第1章 语言处理与Python

1.1 语言计算：文本和词汇

入门

>>>import nltk

>>>nltk.download()

>>>from nltk.book import *

>>>text1

搜索文本

>>>text1.concordance("monstrous")

>>>text1.similar("monstrous")

>>>text2.common_contexts(["monstrous","very"])

>>>text4.dispersion_plot(["citizens","democracy","freedom","duties","America"])

>>>text3.generate()

计数词汇

>>>len(text3)

>>>sorted(set(text3))

>>>from __feature__ import division
>>>len(text3) / len(set(text3))

>>>text3.count("smote")

1.2 近观Python：将文本当做词链表

1.3 计算语言：简单的统计

频率分布

>>>fdist1 = FreqDist(text1)
>>>vocabulary1 = fdist1.keys()
>>>vocabulary1[:50]

>>>fdist1.plot(50,cumulative=True)

>>>fdist1.hapaxes()

细粒度选择词

>>>V = set(text1)
>>>long_words = [w for w in V if len(w) > 15]
>>>sorted(long_words)

词语搭配和双连词

>>>list(bigrams(['more', 'is', 'said', 'than', 'done']))

collocations()

>>text4.collocations()

计算其他东西

>>>fdist = FreqDist([len(w) for w in text1])
>>>fdist.keys()

>>>fdist.items()

>>>fdist.max()
>>>fdist.freq(3)

fdist = FreqDist(samples)

fdist.inc(samples)

fdist['monstrous']

fdist.freq('monstrous')

fdist.N()

fdist.keys()

for sample in fdist:

fdist.max()

fdist.tabulate()

fdist.plot()

fdist.plot(cumulative=True)

fdist1<fdist2

1.4 回到Python：决策与控制

1.5 自动理解自然语言

词义消岐

指代消解 anaphora resolution

自动生成语言

babelize_shell()

人机对话系统

>>>import nltk
>>>nltk.chat.chatbots()

文本的含义