爬虫服务器环境部署

 安装python
首先要升级python版本到2.7.12
1.下载安装包
wget  https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tar.xz

xz -d  Python-2.7.12.tar.xz
tar -xvf Python-2.7.12.tar.xz
cd Python-2.7.12

./configure

make

make install

2.备份老python,建立软链接

mv /usr/bin/python /usr/bin/python2.7.5

ln -s /usr/local/bin/python2.7 /usr/bin/python

 

3.升级python以后yum的使用可能会受到影响，需要进行一下设置

vi /usr/libexec/urlgrabber-ext-down

vi /usr/bin/yum  

找到第一行#!/usr/bin/python 改为 #!/usr/bin/python2.7.5

 

安装 scrapy

1.首先准备环境

yum install gcc libffi-devel openssl-devel libxml2 libxslt-devel libxml2-devel python-devel -y

 

2.安装easy_install

yum install python-setuptools

 
3.安装pip

easy_install pip
 
4.安装 lxml

easy_install lxml
 
5.安装scrapy

pip install scrapy
 
安装selenium
pip install -U selenium

 

 安装 fontconfig

yum install fontconfig