python爬虫中对含中文的url处理

在练习urllib操作中，遇到了url中含有中文字符的问题。比如http://dotamax.com/，看下源码的话，上方的搜索框的name=p，输入内容点击搜索以后，通过GET方法进行传递，比如我们搜索”意“，url变为http://dotamax.com/search/?q=意。但是url中是不允许出现中文字符的，这时候就改用urllib.parse.quote方法对中文字符进行转换。

url = "http://dotamax.com/"
search = "search/?q=" + urllib.parse.quote("意")
html = urllib.request.urlopen(url + search)

这样就可以正常获取页面了。

需要注意的是不能对整个url调用quote方法。

print(urllib.parse.quote("http://dotamax.com/search/?q=意"))

上面代码输出结果：

http%3A//dotamax.com/search/%3Fq%3D%E6%84%8F

可以看到，' : ', ' ? ', ' = '都被解码，因此需要将最后的中文字符部分调用quote方法后接在后面。

但是还有更方便的方法：

import urllib.parse

b = b'/:?='
print(urllib.parse.quote("http://dotamax.com/search/?q=意", b))

输出结果为：

http://dotamax.com/search/?q=%E6%84%8F

这就是我们想要的结果了。对quote方法是用help命令可以看到如下信息：

Help on function quote in module urllib.parse:

quote(string, safe='/', encoding=None, errors=None)
quote('abc def') -> 'abc%20def'

Each part of a URL, e.g. the path info, the query, etc., has a
different set of reserved characters that must be quoted.

RFC 2396 Uniform Resource Identifiers (URI): Generic Syntax lists
the following reserved characters.

reserved    = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |
"$" | ","

Each of these characters is reserved in some component of a URL,
but not necessarily in all of them.

By default, the quote function is intended for quoting the path
section of a URL.  Thus, it will not encode '/'.  This character
is reserved, but in typical usage the quote function is being
called on a path where the existing slash characters are used as
reserved characters.

string and safe may be either str or bytes objects. encoding must
not be specified if string is a str.

The optional encoding and errors parameters specify how to deal with
non-ASCII characters, as accepted by the str.encode method.
By default, encoding='utf-8' (characters are encoded with UTF-8), and
errors='strict' (unsupported characters raise a UnicodeEncodeError).

None

safe为可以忽略的字符，可以str类型或者bytes类型。

更详细的一些用法可以看这里：
http://www.nowamagic.net/academy/detail/1302863