【转】CentOS 6.5安装pyspider过程记录

原文地址：http://blog.sina.com.cn/s/blog_48c95a190102wczx.html

 

1、根据pyspider官方推荐的安装方法，使用pip命令直接安装pyspider # sudo pip install pyspider Installing collected packages: click, itsdangerous, Werkzeug, Flask, chardet, cssselect, lxml, pyquery, ordereddict, backports.ssl-match-hostname, singledispatch, certifi, backports-abc, tornado, Flask-Login, u-msgpack-python, wsgidav, pyspider   Running setup.py install for click   Running setup.py install for itsdangerous   Running setup.py install for chardet   Running setup.py install for lxml     Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-rpK3h8/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-mmscE4-record/install-record.txt --single-version-externally-managed --compile:     Building lxml version 3.6.0.     Building without Cython.     ERROR: /bin/sh: xslt-config: command not found     ** make sure the development packages of libxml2 and libxslt are installed **     Using build configuration of libxslt     running install     running build     running build_py     creating build     creating build/lib.linux-x86_64-2.6     creating build/lib.linux-x86_64-2.6/lxml     copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/cssselect.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/__init__.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/builder.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/sax.py -> build/lib.linux-x86_64-2.6/lxml     creating build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-2.6/lxml/includes     creating build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/defs.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/builder.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/clean.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/diff.py -> build/lib.linux-x86_64-2.6/lxml/html     creating build/lib.linux-x86_64-2.6/lxml/isoschematron     copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-2.6/lxml/isoschematron     copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng     copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     running build_ext     building 'lxml.etree' extension     creating build/temp.linux-x86_64-2.6     creating build/temp.linux-x86_64-2.6/src     creating build/temp.linux-x86_64-2.6/src/lxml     gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w     src/lxml/lxml.etree.c:82:20: error: Python.h: No such file or directory     src/lxml/lxml.etree.c:84:6: error: #error Python headers needed to compile C extensions, please install development version of Python.     Compile failed: command 'gcc' failed with exit status 1     creating tmp     cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitlSjq1o.c -o tmp/xmlXPathInitlSjq1o.o     /tmp/xmlXPathInitlSjq1o.c:1:26: error: libxml/xpath.h: No such file or directory     *********************************************************************************     Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?     *********************************************************************************     error: command 'gcc' failed with exit status 1   根据错误提示可知，导致安装出错的原因是libxml2和libxslt这两个库没有安装，而且找不到xslt-config这个可执行程序，解决方法如下： # sudo yum install libxml2   # sudo yum install libxslt        # sudo yum install libxslt-devel     2、继续使用pip命令安装pyspider # sudo pip install pyspider ... Running setup.py install for lxml     Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-EQanqI/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WzvLSh-record/install-record.txt --single-version-externally-managed --compile:     Building lxml version 3.6.0.     Building without Cython.     Using build configuration of libxslt 1.1.26     Building against libxml2/libxslt in the following directory: /usr/lib64     running install     running build     running build_py     creating build     creating build/lib.linux-x86_64-2.6     creating build/lib.linux-x86_64-2.6/lxml     copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/cssselect.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/__init__.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/builder.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/sax.py -> build/lib.linux-x86_64-2.6/lxml     creating build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-2.6/lxml/includes     creating build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/defs.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/builder.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/clean.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-2.6/lxml/html     copying src/lxml/html/diff.py -> build/lib.linux-x86_64-2.6/lxml/html     creating build/lib.linux-x86_64-2.6/lxml/isoschematron     copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-2.6/lxml/isoschematron     copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-2.6/lxml     copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-2.6/lxml/includes     copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng     copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl     creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1     running build_ext     building 'lxml.etree' extension     creating build/temp.linux-x86_64-2.6     creating build/temp.linux-x86_64-2.6/src     creating build/temp.linux-x86_64-2.6/src/lxml     gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w     src/lxml/lxml.etree.c:82:20: error: Python.h: No such file or directory     src/lxml/lxml.etree.c:84:6: error: #error Python headers needed to compile C extensions, please install development version of Python.     Compile failed: command 'gcc' failed with exit status 1     creating tmp     cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitd0Umi6.c -o tmp/xmlXPathInitd0Umi6.o     cc tmp/xmlXPathInitd0Umi6.o -L/usr/lib64 -lxml2 -o a.out     error: command 'gcc' failed with exit status 1   根据错误提示可知，导致安装出错的原因是Python.h: No such file or directory，即找不到Python.h这个头文件，解决方法如下： # sudo yum install python-devel 至此，再次执行sudo pip install pyspider就可以完成pyspider的安装。   3、启动pyspider，验证安装是否成功 # pyspider Traceback (most recent call last):   File "/usr/bin/pyspider", line 5, in     from pkg_resources import load_entry_point   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in     working_set.require(__requires__)   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require     needed = self.resolve(parse_requirements(requirements))   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve     raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pyquery   根据错误提示可知，在执行pyspider时找不到pyquery，但实际上在安装pyspider时pyquery已经安装了，通过pip freeze也可以看到，导致该问题的原因不详，解决方法如下： # sudo pip install -U setuptools   4、安装pyspider所需使用的驱动 通过单纯的pyspider命令来启动pyspider时，默认会在当前目录下创建data目录，其中存放了sqlite数据库文件，该情况下无需安装任务数据库connector。如果要以分布式集群的方式来运行pyspider的话，就需要使用mysql、mongodb或postgresql作为后台数据库，使用rabbitmq、redis、beanstalk或kombu作为消息队列。官方推荐的安装方式是采用pip install pyspider[all]命令来安装所有驱动，但该方法一方面会安装一些无用的驱动，另一方面极有可能导致安装失败。因此如果已经确定后台数据库软件和消息队列软件的话，那么选择安装需要的驱动即可。例如采用mongodb作为后台数据库，采用redis作为消息队列的话，就可以通过如下命令安装： # sudo pip install pymongo # sudo pip install redis