使用phantomjs抓取ITC和android market的安装统计数据 | 蓝色的华

使用phantomjs抓取ITC和android market的安装统计数据

同事用python抓各大andriod市场的页面分析软件下载量，发现andriod　market的数据是通过ajax加载的，而且貌似加密过了，没法直接解析。后来俺发现了phantomjs这个命令行的webkit,在可以在命令行渲染网页，这样不论数据怎么加载，怎么加密，对于标准的浏览器都无能为力了。软件的原理跟以前介绍的一个命令行web截图工具一样,内嵌一个qt4的webkit，然后渲染到xvfb虚拟出的xserver上。不同是这个提供了js的api，用起来方便，可以用来做爬虫，站点监控，服务端截图。

使用方法：

第一步,安装phantomjs

Mac os & windows:

直接下载.dmg或.exe安装包即可:http://code.google.com/p/phantomjs/downloads/list

mac下安装完闭，可执行文件的路径:/Applications/phantomjs.app/Contents/MacOS/phantomjs

ubuntu:

sudo add-apt-repository ppa:jerome-etienne/neoip
sudo apt-get update
sudo apt-get install phantomjs

sudo add-apt-repository ppa:jerome-etienne/neoip<br /> sudo apt-get update<br /> sudo apt-get install phantomjsCopy Code

centos 5.3:

折腾开始了。。由于phantomjs的linux版本是通过pyqt4实现的，所以装起来比较麻烦

首先我们需要安装qt4.7.而yum默认安装的是4.1

rpm -ivh http://software.freivald.com/centos/software.freivald.com-1.0.0-1.noarch.rpm yum update fontconfig fontconfig-devel
yum install qt4 qt4-devel
#如果已经安装过qt4则执行 yum update qt4 qt4-devel

rpm -ivh http://software.freivald.com/centos/software.freivald.com-1.0.0-1.noarch.rpm<br /> yum update fontconfig fontconfig-devel<br /> yum install qt4 qt4-devel<br /> #如果已经安装过qt4则执行 yum update qt4 qt4-develCopy Code

安装Xvfb

yum install xorg-x11-server-Xvfb xorg-x11-server-Xorg xorg-x11-fonts*

yum install xorg-x11-server-Xvfb xorg-x11-server-Xorg xorg-x11-fonts*Copy Code

安装python 2.7,自带的为2.4，没法用

wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2 tar jxvf Python-2.7.2.tar.bz2
cd Python-2.7.2
./configure --prefix=/opt/python27
make
make install
cd ..

wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2<br /> tar jxvf Python-2.7.2.tar.bz2<br /> cd Python-2.7.2<br /> ./configure --prefix=/opt/python27<br /> make<br /> make install<br /> cd ..Copy Code

安装setup tools

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e tar zxvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
/opt/python27/bin/python setup.py install
cd ..

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e<br /> tar zxvf setuptools-0.6c11.tar.gz<br /> cd setuptools-0.6c11<br /> /opt/python27/bin/python setup.py install<br /> cd ..Copy Code

安装sip

wget http://www.riverbankcomputing.com/static/Downloads/sip4/sip-4.12.4.tar.gz tar zxvf sip-4.12.4.tar.gz
cd sip-4.12.4
/opt/python27/bin/python configure.py
make
make install
cd ..

wget http://www.riverbankcomputing.com/static/Downloads/sip4/sip-4.12.4.tar.gz<br /> tar zxvf sip-4.12.4.tar.gz<br /> cd sip-4.12.4<br /> /opt/python27/bin/python configure.py<br /> make<br /> make install<br /> cd ..Copy Code

安装pyqt4

wget http://www.riverbankcomputing.com/static/Downloads/PyQt4/PyQt-x11-gpl-4.8.5.tar.gz tar zxvf PyQt-x11-gpl-4.8.5.tar.gz
cd PyQt-x11-gpl-4.8.5
/opt/python27/bin/python configure.py -q /usr/lib/qt4/bin/qmake
#对于64位系统
#/opt/python27/bin/python configure.py -q /usr/lib64/qt4/bin/qmake
make
make install
cd ..

wget http://www.riverbankcomputing.com/static/Downloads/PyQt4/PyQt-x11-gpl-4.8.5.tar.gz<br /> tar zxvf PyQt-x11-gpl-4.8.5.tar.gz<br /> cd PyQt-x11-gpl-4.8.5<br /> /opt/python27/bin/python configure.py -q /usr/lib/qt4/bin/qmake<br /> #对于64位系统<br /> #/opt/python27/bin/python configure.py -q /usr/lib64/qt4/bin/qmake<br /> make<br /> make install<br /> cd ..Copy Code

最后安装pyphantomjs

mkdir pyphantomjs
cd pyphantomjs
wget http://phantomjs.googlecode.com/files/pyphantomjs-1.2.0-source.zip unzip pyphantomjs-1.2.0-source.zip
/opt/python27/bin/python setup.py install

mkdir pyphantomjs<br /> cd pyphantomjs<br /> wget http://phantomjs.googlecode.com/files/pyphantomjs-1.2.0-source.zip<br /> unzip pyphantomjs-1.2.0-source.zip<br /> /opt/python27/bin/python setup.py installCopy Code

折腾到这里，pyphantomjs已经安装到了/opt/python27/bin/pyphantomjs

直接执行/opt/python27/bin/pyphantomjs –help,会发现报错

sip.setapi('QString', 2)
ValueError: API 'QString' has already been set to version 1

解决方法

编辑/opt/python27/bin/pyphantomjs,在开始追加几句

#!/opt/python27/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'PyPhantomJS==1.2.0','console_scripts','pyphantomjs'
#fix start
import sip
sip.setapi('QString', 2)
sip.setapi('QVariant', 2)
#fix end
__requires__ = 'PyPhantomJS==1.2.0'

#!/opt/python27/bin/python<br /> # EASY-INSTALL-ENTRY-SCRIPT: 'PyPhantomJS==1.2.0','console_scripts','pyphantomjs'<br /> #fix start<br /> import sip<br /> sip.setapi('QString', 2)<br /> sip.setapi('QVariant', 2)<br /> #fix end<br /> __requires__ = 'PyPhantomJS==1.2.0'Copy Code

第二步：

下载我们用来解析数据的js脚本:android_itc_daliy_report

修改里面的变量:

TIMEOUT = 120;//脚本执行超时时间
ACCOUNT = '';//登陆账号
PASSWORD = '';//密码

第三步:执行抓取脚本

对于Mac os:

#抓取andriod market的安装总量
/Applications/phantomjs.app/Contents/MacOS/phantomjs --load-images=no AndroidMarketDailyReport.js
#抓取ITC的每天的安装量,需要制定日期，而且日期必须是web页面里的日期选择框里有的日期
/Applications/phantomjs.app/Contents/MacOS/phantomjs --load-images=no ITCDailyReport.js 09/06/2011

#抓取andriod market的安装总量<br /> /Applications/phantomjs.app/Contents/MacOS/phantomjs --load-images=no AndroidMarketDailyReport.js<br /> #抓取ITC的每天的安装量,需要制定日期，而且日期必须是web页面里的日期选择框里有的日期<br /> /Applications/phantomjs.app/Contents/MacOS/phantomjs --load-images=no ITCDailyReport.js 09/06/2011Copy Code

对于Centos:

#首先保证Xvfb已经启动
Xvfb :0 -screen 0 1024x768x24 &
#抓取andriod market的安装量
DISPLAY=:0 /opt/python27/bin/pyphantomjs --load-images=no --ignore-ssl-errors=yes AndroidMarketDailyReport.js
#抓取ITC的每天的安装量,需要制定日期，而且日期必须是web页面里的日期选择框里有的日期
DISPLAY=:0 /opt/python27/bin/pyphantomjs --load-images=no --ignore-ssl-errors=yes ITCDailyReport.js 09/06/2011

#首先保证Xvfb已经启动<br /> Xvfb :0 -screen 0 1024x768x24 &<br /> #抓取andriod market的安装量<br /> DISPLAY=:0 /opt/python27/bin/pyphantomjs --load-images=no --ignore-ssl-errors=yes AndroidMarketDailyReport.js<br /> #抓取ITC的每天的安装量,需要制定日期，而且日期必须是web页面里的日期选择框里有的日期<br /> DISPLAY=:0 /opt/python27/bin/pyphantomjs --load-images=no --ignore-ssl-errors=yes ITCDailyReport.js 09/06/2011Copy Code

获取输出结果,以Mac os为例:

/Applications/phantomjs.app/Contents/MacOS/phantomjs –load-images=no ITCDailyReport.js 09/06/2011|grep REPORT

REPORT: soft_name 0000

REPORT: soft_name 0000

/Applications/phantomjs.app/Contents/MacOS/phantomjs –load-images=no AndroidMarketDailyReport.js |grep REPORT

REPORT: total 0000

REPORT: real 0000

如果没有输出结果，则说明有异常，账号错误，超时，等等。。