安装 Scrapy for Python2.7

1. 简介

Scrapy是基于Python的网络爬虫框架，支持Python2.7 和 Python3.3以上。

2. 安装

安装平台： Mac 10.11

2.1 先决条件

python 2.7

pip

setuptools

lxml

openssl

2.2 安装scrapy

pip install scrapy

2.3 重新安装six

Mac自带的six库需要重新安装

sudo rm -rf /Library/Python/2.7/site-packages/six*
sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*
sudo pip install six

2.4 重新安装twisted

安装scrapy时自动安装的twisted框架版本有问题，需要重新安装

pip install twisted==13.1.0

3. 使用

现在就能使用scrapy了

$ scrapy
Scrapy 1.3.3 - project: tutorial

Usage:
scrapy <command> [options] [args]

Available commands:
bench         Run quick benchmark test
check         Check spider contracts
commands
crawl         Run a spider
edit          Edit spider
fetch         Fetch a URL using the Scrapy downloader
genspider     Generate new spider using pre-defined templates
list          List available spiders
parse         Parse URL (using its spider) and print the results
runspider     Run a self-contained spider (without creating a project)
settings      Get settings values
shell         Interactive scraping console
startproject  Create new project
version       Print Scrapy version
view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command

4. 相关链接

中文教程： http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/install.html#intro-install