20160228python中文乱码

python抓取中文网页显示乱码

抓取代码：

import urllib2,urllib,cookielib,threading
import os
import re

url = 'http://www.dugukeji.com/' #抓取的url
req = urllib2.Request(url)
response = urllib2.urlopen(req).read()
print response

研究得知源网页为GBK（gb2312）编码，而python打印为utf8编码，所以需要做一下编码转换

修改后的抓取代码：

import urllib2,urllib,cookielib,threading
import os
import re

url = 'http://www.dugukeji.com/'
req = urllib2.Request(url)
response = urllib2.urlopen(req).read()
response = unicode(response,'GBK').encode('UTF-8')
print response

用unicode函数把GBK编码的网页转换为unicode，再用encode编码成UTF-8输出即可

原文链接http://blog.csdn.net/maverick1990/article/details/8880051