Python 编码问题 utf-8 和 unicode

前言：utf-8 和 unicode

UTF-8，1个汉字占3个字节，扩展B区以后的汉字占4个字节。

GBK，1个汉字占2个字节。

UTF-16，通常汉字占两个字节。

Unicode，一个英文等于两个字节，一个中文（含繁体）等于两个字节，范围是 \u4e00-\u9fa5。

UTF-8是Unicode的一个实现？

记得java虚拟机存储代码都是用unicode的，不涉及具体编码格式，只有在读入读出边界处会用具体编码格式来处理。

命令行模式下

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-prompt" style="color: rgb(0, 102, 102); box-sizing: border-box;">>>> </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
<span class="hljs-prompt" style="color: rgb(0, 102, 102); box-sizing: border-box;">>>> </span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
好玩</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

好玩的 unicode 是 \u597d\u73a9

IDE模式下

<code class="hljs go has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"># -*- coding: UTF<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">-8</span> -*-
s = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"好玩"</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> s
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> s.decode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>).encode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>)
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> unicode(s,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>)

<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">len</span>(s)
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">len</span>(s.decode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>).encode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>))
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">print</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">len</span>(unicode(s,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>))</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li></ul>

输出

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">好玩
好玩
好玩
6
6
2</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

utf-8每个汉字占3个字节，所以“好玩”输出了6

一个汉字表示为一个unicode，所以这儿的len表示的是有几个unicode，所以输出了2

“好玩”的unicode为

\u597d\u73a9

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">us = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> us
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> us.encode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>)

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> len(us)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> len(us.encode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>))</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

输出

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">好玩
好玩
2
6</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

疑惑

us = u'\u597d\u73a9'

print us

这儿会输出汉字

好玩

我不知道在IDE里面如何输出

u'\u597d\u73a9'

，在命令行里面倒是可以。

输出的时候会转化为IDE控制台使用的编码格式，所以会输出汉字？

后来找到了解决方法，利用python自带的

repr

函数

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">us = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> repr(us)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> repr(us.encode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"utf-8"</span>))</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

输出

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">u'\u597d\u73a9'</span>
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'\xe5\xa5\xbd\xe7\x8e\xa9'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

可以看出utf-8表示汉字确实用了3个字节。