解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
2016-11-06 22:53
507 查看
我的爬虫程序
import urllib.request import http.cookiejar import codecs import io import sys sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030') #head: dict of header def makeMyOpener(head = { 'Connection':'keep-alive', 'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Language':'zh-CN,zh;q=0.8,en;q=0.6', 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36' }): cj = http.cookiejar.CookieJar() opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) header = [] for key, value in head.items(): elem = (key, value) header.append(elem) opener.addheaders = header return opener oper = makeMyOpener() uop = oper.open('https://www.baidu.com/', timeout = 1000) data = uop.read() if data[:3] == codecs.BOM_UTF8: data = data[3:] print(data.decode('utf-8'))
今天在用python写爬虫程序时,总是出现如下的类似错误:
UnicodeEncodeError:'gbk' codec can't encode character '\xa9' in position 0:illegal multibyte sequence
这个错误,网上给出的解释是: 某些软件如notepad,在保存一个以UTF-8编码的文件时, 会在文件开始的地方插入三个不可见的字符(0xEF 0xBB 0xBF,即BOM)。 因此我们在读取时需要自己去掉这些字符,python中的codecs module定义了这个常量:
import io import sys #这里判断下,把三个不可见的字符去除掉 if data[:3] == codecs.BOM_UTF8: data = data[3:] print(data.decode('utf-8'))
运行后,虽然这个错不报了,结果又报了另外一个错误!
UnicodeEncodeError:'gbk' codec can't encode character '\xa0' in position 0:illegal multibyte sequence
这下我不淡定了,继续去网上找,最后参考这篇文章找到了答案:
http://blog.csdn.net/jim7424994/article/details/22675759#
一开始我以为是百度采用gbk编码,而我把它转成utf-8时,有些字符转换不成功造成的! 看完后,才知道,原来是python的print()方法的问题。在python中, print()方法在Win7的默认编码是gbk,它在打印时,并不是所有的字符都支持的。 而且这个问题一般也就是在cmd中才会有。 在cmd中是改变标准输出编码:
import io import sys sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8')
但是呢,改成utf-8在cmd中中文又会乱码,所以改成
import io import sys #改变标准输出的默认编码 sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')
——————————转载内容———————————-
从网上抓了一些字节流,想打印出来结果发生了一下错误:
UnicodeEncodeError: 'gbk' codec can't encode character '\xbb' in position 8530: illegal multibyte sequence
代码
import urllib.request res=urllib.request.urlopen('http://www.baidu.com') htmlBytes=res.read() print(htmlBytes.decode('utf-8'))
错误信息让人很困惑,为什么用的是’utf-8’解码,错误信息却提示’gbk’错误呢?
不仅如此,从百度首页的html中发现以下代码:
<meta http-equiv="content-type" content="text/html;charset=utf-8">
这说明网页的确用的是utf-8,为什么会出现Error呢?
在python3里,有几点关于编码的常识
1.字符就是unicode字符,字符串就是unicode字符数组
如果用以下代码测试,
print('a'=='\u0061')
会发现结果为True,足以说明两者的等价关系。
2.str转bytes叫encode,bytes转str叫decode,如上面的代码就是将抓到的字节流给decode成unicode数组
我根据上面的错误信息分析了字节流中出现\xbb的地方,发现有个\xc2\xbb的特殊字符»,我怀疑是它无法被解码。
用以下代码测试后
print(b'\xc2\xbb'.decode('utf-8'))
它果然报错了:UnicodeEncodeError: ‘gbk’ codec can’t encode character ‘\xbb’ in position 0: illegal multibyte sequence
上网找了下utf-8编码表,发现的确特殊字符»的utf-8形式就是c2bb,unicode是’\u00bb’,为什么无法解码呢。。。
仔细看看错误信息,它提示’gbk’无法encode,但是我的代码是utf-8无法decode,压根牛头不对马嘴,终于让我怀疑是print函数出错了。。于是立即有了以下的测试
print('\u00bb')
结果报错了:UnicodeEncodeError: ‘gbk’ codec can’t encode character ‘\xbb’ in position 0: illegal multibyte sequence
原来是print()函数自身有限制,不能完全打印所有的unicode字符。
知道原因后,google了一下解决方法,其实print()函数的局限就是Python默认编码的局限,因为系统是win7的,python的默认编码不是’utf-8’,改一下python的默认编码成’utf-8’就
import io import sys import urllib.request sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8') #改变标准输出的默认编码 res=urllib.request.urlopen('http://www.baidu.com') htmlBytes=res.read() print(htmlBytes.decode('utf-8'))
不报错了,但是居然有好多乱码(英文显示正常,中文则显示乱码)!!又一阵折腾后发现是控制台的问题,具体来说就是我在cmd下运行该脚本会有乱码,而在IDLE下运行却很正常。
由此我推测是cmd不能很好地兼容utf8,而IDLE就可以,甚至在IDLE下运行,连“改变标准输出的默认编码”都不用,因为它默认就是utf8。如果一定要在cmd下运行,那就改一下编码,比如我换成“gb18030”,就能正常显示了:
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030') #改变标准输出的默认编码
最后,附上一些常用的和中文有关的编码的名称,分别赋值给encoding,就可以看到不同的效果了:
——————————
编码名称 | 用途 |
---|---|
utf8 | 所有语言 |
gbk | 简体中文 |
gb2312 | 简体中文 |
gb18030 | 简体中文 |
big5 | 繁体中文 |
big5hkscs | 繁体中文 |
相关文章推荐
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- python编码问题——解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- python编码问题——解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- 解决python3 UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX 标签: pythonco
- UnicodeEncodeError: 'gbk' codec can't encode character '\xbb' in position XXX解决办法
- UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX
- pip install 安装报错:UnicodeEncodeError: 'ascii' codec can't encode characters in position XX的解决办法
- python2.7解决UTF-8 UnicodeEncodeError:'gbk' codec can't encode character u'\xa5' in position