Python爬虫教程——进阶一之爬虫框架Scrapy安装配置

初级的爬虫我们利用urllib和urllib2库以及正则表达式就可以完成了，不过还有更加强大的工具，爬虫框架Scrapy，这安装过程也是煞费苦心哪，在此整理如下。

Windows 平台：

我的系统是 Win7，首先，你要有Python，我用的是2.7.7版本，Python3相仿，只是一些源文件不同。
官网文档：http://doc.scrapy.org/en/latest/intro/install.html，最权威哒，下面是我的亲身体验过程。

1.安装Python

安装过程我就不多说啦，我的电脑中已经安装了 Python 2.7.7 版本啦，安装完之后记得配置环境变量，比如我的安装在D盘，D:\python2.7.7，就把以下两个路径添加到Path变量中

<code class="hljs tex has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">D:<span class="hljs-command" style="box-sizing: border-box; color: rgb(0, 0, 136);">\python</span>2.7.7;D:<span class="hljs-command" style="box-sizing: border-box; color: rgb(0, 0, 136);">\python</span>2.7.7<span class="hljs-command" style="box-sizing: border-box; color: rgb(0, 0, 136);">\Scripts</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

配置好了之后，在命令行中输入 python –version，如果没有提示错误，则安装成功

2.安装pywin32

在windows下，必须安装pywin32，安装地址：http://sourceforge.net/projects/pywin32/
下载对应版本的pywin32，直接双击安装即可，安装完毕之后验证：



在python命令行下输入
import win32com
如果没有提示错误，则证明安装成功

3.安装pip

pip是用来安装其他必要包的工具，首先下载 get-pip.py
下载好之后，选中该文件所在路径，执行下面的命令

<code class="hljs cs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">python <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">get</span>-pip.py</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

执行命令后便会安装好pip，并且同时，它帮你安装了setuptools
安装完了之后在命令行中执行

<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">pip</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">version</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

如果提示如下，说明就安装成功了，如果提示不是内部或外部命令，那么就检查一下环境变量有没有配置好吧，有两个路径。

4.安装pyOPENSSL

在Windows下，是没有预装pyOPENSSL的，而在Linux下是已经安装好的。
安装地址：https://launchpad.net/pyopenssl

5.安装 lxml

lxml的详细介绍 点我，是一种使用 Python 编写的库，可以迅速、灵活地处理 XML
直接执行如下命令

<code class="hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">pip <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span> lxml</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

就可完成安装，如果提示 Microsoft Visual C++库没安装，则 点我下载支持的库。

6.安装Scrapy

最后就是激动人心的时刻啦，上面的铺垫做好了，我们终于可以享受到胜利的果实啦！
执行如下命令

<code class="hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">pip <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span> Scrapy</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

pip 会另外下载其他依赖的包，这些就不要我们手动安装啦，等待一会，大功告成！

7.验证安装

输入 Scrapy
如果提示如下命令，就证明安装成功啦，如果失败了，请检查上述步骤有何疏漏。

Linux Ubuntu 平台：

Linux 下安装非常简单，只需要执行几条命令几个

1.安装Python

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo apt<span class="hljs-attribute" style="box-sizing: border-box;">-get</span> install python2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.7</span> python2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.7</span><span class="hljs-attribute" style="box-sizing: border-box;">-dev</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

2.安装 pip

首先下载 get-pip.py
下载好之后，选中该文件所在路径，执行下面的命令

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> python get-pip.py</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

3.直接安装 Scrapy

由于 Linux下已经预装了 lxml 和 OPENSSL
如果想验证 lxml ，可以分别输入

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> pip install lxml</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

出现下面的提示这证明已经安装成功

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Requirement already satisfied (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">use</span> --upgrade <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> upgrade): lxml <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> /usr/lib/python2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.7</span>/dist-packages</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

如果想验证 openssl，则直接输入openssl 即可，如果跳转到 OPENSSL 命令行，则安装成功。
接下来直接安装 Scrapy 即可

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> pip install Scrapy</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

安装完毕之后，输入 scrapy
注意，这里linux下不要输入Scrapy，linux依然严格区分大小写的，感谢kamen童鞋提醒。
如果出现如下提示，这证明安装成功

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Usage:
 scrapy <<span class="hljs-command" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">command</span>> [<span class="hljs-title" style="box-sizing: border-box;">options</span>] [<span class="hljs-title" style="box-sizing: border-box;">args</span>]</span>

Available commands:
 bench Run quick benchmark test
 fetch Fetch <span class="hljs-operator" style="box-sizing: border-box;">a</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">URL</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">using</span> <span class="hljs-operator" style="box-sizing: border-box;">the</span> Scrapy downloader
 runspider Run <span class="hljs-operator" style="box-sizing: border-box;">a</span> self-contained spider (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">without</span> creating <span class="hljs-operator" style="box-sizing: border-box;">a</span> project)
 settings Get settings values
 <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">shell</span> Interactive scraping console
 startproject Create <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">new</span> project
 <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">version</span> Print Scrapy <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">version</span>
 view Open <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">URL</span> <span class="hljs-operator" style="box-sizing: border-box;">in</span> browser, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> seen <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">by</span> Scrapy

 [ more ] More commands available when run <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">from</span> project <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">directory</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

截图如下



如有问题，欢迎留言！祝各位小伙伴顺利安装！
【转载请注明：静觅 » Python爬虫进阶一之爬虫框架Scrapy安装配置】