Python 识别文本编码

摘要: python chardet 识别文本编码的原理

[code=plain]>>> import chardet
>>> f = open('songs.txt','r')
>>> result = chardet.detect(f.read())
>>> result
{'confidence': 0.99, 'encoding': 'utf-8'}

一次性读取全部文件，可能会浪费点内存（chardet的探测会在搜集到足够数据之后停止，文件比较大时，就比较浪费）

chardet 按照词频统计

[code=plain]def description_of(file, name='stdin'):
"""Return a string describing the probable encoding of a file."""
u = UniversalDetector()
for line in file:
u.feed(line)
u.close()
result = u.result
if result['encoding']:
return '%s: %s with confidence %s' % (name,
result['encoding'],
result['confidence'])
else:
return '%s: no result' % name

逐行读取

探测大量重复数据的时间比不重复数据的时间可能要小点。

官网原理解读：

Some character sequences pop up all the time, while other sequences make no sense. A person fluent in English who opens a newspaper and finds “txzqJv 2!dasd0a QqdKjvz” will instantly recognize that that isn’t English (even though it is composed entirely of English letters). By studying lots of “typical” text, a computer algorithm can simulate this kind of fluency and make an educated guess about a text’s language.

In other words, encoding detection is really language detection, combined with knowledge of which languages tend to use which character encodings.

对于某种文本，某些字符串组合经常出现，其他一些字符组合不会出现，依靠研究每种文本的特殊类型的字符组合 便可以从统计学上识别出文本的编码，另一个角度上来说，识别编码就是识别语言类型。（每种语言都有自己最适合的编码）