[笔记] PhantomJS爬虫小记

PhantomJS这个headless浏览器还是很好用的，配合着v8与qt化的webkit引擎，JavaScript可以很好得去处理得到的页面数据。

去年在整合各个平台搭建搜索引擎的时候，写了一个爬虫脚本，去掉一些认证和特殊功能，一个简单的js就放到github上了：

https://github.com/dna2github/dna2oldmemory/blob/master/dna2poem/phantomjs/spider.js

使用的时候直接：

phantomjs spider.js <url>

这个小爬虫包含了：

－ 资源加载监听

－ 英语分词器和词频统计

－ DOM里<a>的href抓取

这样，当phantomjs加载一个页面的时候，就可以知道访问的所有api，图像，css，js等资源url是什么，加载了多长时间。

分词器和词频统计是为了后面给文章进行大概的分类，提高搜索结果的质量。

对于<a>的href抓取，这里没有判断href是不是javascript调用；抓取后有两个用途，一个是建立page rank模型，另一个是可以继续爬取下一个网页。

例如：

phantomjs spider.js https://en.wikipedia.org/wiki/Special:Random 我们可以得到如下结果：

- Sun Mar 27 2016 11:40:50 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST)   <GET: 302> /wiki/Special:Random
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST)   <GET: 200> /wiki/Adele_Wong
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=ext.cite.styles%7Cext.gadget.DRN-wizard%2CReferenceTooltips%2Ccharinsert%2Cfeatured-articles-links%2CrefToolbar%2Cswitcher%2Cteahouse%7Cext.tmh.thumbnail.styles%7Cext.uls.nojs%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.raggett%2CsectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=site&only=styles&skin=vector
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector
- Sun Mar 27 2016 11:40:52 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=tbOYKAbm
url:  https://en.wikipedia.org/wiki/Adele_Wong = frequencies ==========================================
total: 546
in 18
wong 16
film 13
she 8
singapore 8
at 7
of 7
festival 6
s 6
this 6
page 5
for 5
also 5
which 5
lead 5
as 5
links 5
chinese 5
wikipedia 5
2009 5
= word stream ==========================================
 adele wong from wikipedia free encyclopedia this chinese name family name wong this article may require cleanup to meet wikipedia s quality standards specific problem some of links not working anymore more citation needed as
well please help improve this article if you july 2013 adele wong wong in 2008 music video knowing 懂 chinese name 王 惠 珊 pinyin wáng huìsh ā n mandarin ancestry shanghai china born 8 march 1983 singapore occupation actress singer writer screenwriter years active
2006-present adele wong born 8 march 1983 singaporean actress singer writer screenwriter contents 1 biography 2 filmography 3 references 4 external links biography edit born in singapore wong traces her ancestry to shanghai china she graduated from national
university of singapore nus with honors degree in sociology at nus wong active in dance theatre upon graduation wong began her professional career in acting hosting writing she played lead role in damned 1 which part of omnibus feature film which premiered
at osaka asian film festival in 2009 wong also played lead role in garden girls short film which premiered at singapore film festival in 2009 wong also active in television in singapore where she has been seen in several programs she recently starred as female
lead annie in jack neo s first english-language drama series for okto titled happily ever after she also seen in 2009 mediacorp chinese new year television film kuchinta family reunion wong currently filming chinese-language television drama he ain t pesky
he s my brother for mediacorp s channel 8 where she plays lead female character opposite christopher lee premiering in july 2009 wong s feature film debut as female lead role in days which screened at tokyo international film festival singapore international
film festival short listed for new talent award at shanghai international film festival wong also sang theme song for film knowing 懂 2 which written produced by jim lim 林 倛 玉 besides acting wong writes both fiction non-fiction she hosts column on sinema s
website 3 she also recently awarded grant for short film by singapore film commission 4 hanging gardens starring wong go into production in august 2009 filmography edit short films damned garden girls feature films days music videos 懂 knowing serials perfect
deception as evelyn references edit damned in osaka asian film festival 懂 knowing music video sinema column sfc external links edit adele wong at internet movie database days official movie site watch my ass official movie site watch my ass myspace site categories
1983 birthsnational university of singapore alumnisingaporean actressesliving peoplesingaporean people of chinese descent navigation menu not logged intalkcontributionscreate accountlog inarticletalkreadeditview history main page contents featured content
current events random article donate to wikipedia wikipedia store interaction help about wikipedia community portal recent changes contact page tools what links here related changes upload file special pages permanent link page information wikidata item cite
this page print export create book download as pdf printable version languages add links this page last modified on 26 november 2013 at 21 33 text available under creative commons attribution-sharealike license additional terms may apply by using this site
you agree to terms of use privacy policy wikipedia® registered trademark of wikimedia foundation inc non-profit organization privacy policyabout wikipediadisclaimerscontact wikipediadeveloperscookie statementmobile view
= links ================================================
https://en.wikipedia.org/wiki/Adele_Wong#mw-head https://en.wikipedia.org/wiki/Adele_Wong#p-search https://en.wikipedia.org/wiki/Chinese_name https://en.wikipedia.org/wiki/Chinese_surname https://en.wikipedia.org/wiki/Wang_(surname) https://en.wikipedia.org/wiki/Wikipedia:Cleanup https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit https://en.wikipedia.org/wiki/File:Adele_Wong.jpg https://en.wikipedia.org/wiki/Standard_Mandarin https://en.wikipedia.org/wiki/Ancestral_home_(Chinese) https://en.wikipedia.org/wiki/Shanghai https://en.wikipedia.org/wiki/China https://en.wikipedia.org/wiki/Singapore https://en.wikipedia.org/wiki/Adele_Wong#Biography https://en.wikipedia.org/wiki/Adele_Wong#Filmography https://en.wikipedia.org/wiki/Adele_Wong#References https://en.wikipedia.org/wiki/Adele_Wong#External_links https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=1 https://en.wikipedia.org/wiki/National_University_of_Singapore https://en.wikipedia.org/wiki/Adele_Wong#cite_note-1 https://en.wikipedia.org/wiki/Singapore_Film_Festival https://en.wikipedia.org/wiki/Jack_Neo https://en.wikipedia.org/wiki/Okto https://en.wikipedia.org/wiki/Happily_Ever_After_(TV_series) https://en.wikipedia.org/wiki/MediaCorp https://en.wikipedia.org/wiki/MediaCorp https://en.wikipedia.org/wiki/Christopher_Lee https://en.wikipedia.org/w/index.php?title=The_Days_(Singaporean_Film)&action=edit&redlink=1 https://en.wikipedia.org/wiki/Tokyo_International_Film_Festival https://en.wikipedia.org/wiki/Singapore_International_Film_Festival https://en.wikipedia.org/wiki/Shanghai_International_Film_Festival https://en.wikipedia.org/wiki/Adele_Wong#cite_note-2 https://en.wikipedia.org/w/index.php?title=Jim_Lim&action=edit&redlink=1 https://en.wikipedia.org/wiki/Adele_Wong#cite_note-3 https://en.wikipedia.org/wiki/Adele_Wong#cite_note-4 https://en.wikipedia.org/wiki/Hanging_Gardens https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=2 https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=3 https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-1 http://www.oaff.jp/lineup/lineup02.html#meet_02 https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-2 https://www.youtube.com/watch?v=grf-FZqkPao&feature=relate https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-3 http://www.oaff.jp/lineup/lineup02.html#meet_02 https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-4 http://www.sfc.org.sg/main.html https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=4 http://www.imdb.com/name/nm3044125/ https://en.wikipedia.org/wiki/Internet_Movie_Database http://www.thedaysmovie.com/ http://watchmyass.wordpress.com/ https://www.myspace.com/watchmyassmovie https://en.wikipedia.org/wiki/Wikipedia:Persondata https://en.wikipedia.org/wiki/Singapore https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184 https://en.wikipedia.org/wiki/Help:Category https://en.wikipedia.org/wiki/Category:1983_births https://en.wikipedia.org/wiki/Category:National_University_of_Singapore_alumni https://en.wikipedia.org/wiki/Category:Singaporean_actresses https://en.wikipedia.org/wiki/Category:Living_people https://en.wikipedia.org/wiki/Category:Singaporean_people_of_Chinese_descent https://en.wikipedia.org/wiki/Category:Articles_needing_cleanup_from_July_2013 https://en.wikipedia.org/wiki/Category:All_articles_needing_cleanup https://en.wikipedia.org/wiki/Category:Cleanup_tagged_articles_with_a_reason_field_from_July_2013 https://en.wikipedia.org/wiki/Category:Wikipedia_pages_needing_cleanup_from_July_2013 https://en.wikipedia.org/wiki/Category:Use_dmy_dates_from_November_2010 https://en.wikipedia.org/wiki/Special:MyTalk https://en.wikipedia.org/wiki/Special:MyContributions https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong&type=signup https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong https://en.wikipedia.org/wiki/Adele_Wong https://en.wikipedia.org/wiki/Talk:Adele_Wong https://en.wikipedia.org/wiki/Adele_Wong# https://en.wikipedia.org/wiki/Adele_Wong https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=history https://en.wikipedia.org/wiki/Adele_Wong# https://en.wikipedia.org/wiki/Main_Page https://en.wikipedia.org/wiki/Main_Page https://en.wikipedia.org/wiki/Portal:Contents https://en.wikipedia.org/wiki/Portal:Featured_content https://en.wikipedia.org/wiki/Portal:Current_events https://en.wikipedia.org/wiki/Special:Random https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en https://shop.wikimedia.org/ https://en.wikipedia.org/wiki/Help:Contents https://en.wikipedia.org/wiki/Wikipedia:About https://en.wikipedia.org/wiki/Wikipedia:Community_portal https://en.wikipedia.org/wiki/Special:RecentChanges https://en.wikipedia.org/wiki/Wikipedia:Contact_us https://en.wikipedia.org/wiki/Special:WhatLinksHere/Adele_Wong https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Adele_Wong https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard https://en.wikipedia.org/wiki/Special:SpecialPages https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184 https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=info https://www.wikidata.org/wiki/Q4681907 https://en.wikipedia.org/w/index.php?title=Special:CiteThisPage&page=Adele_Wong&id=583442184 https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=book_creator&referer=Adele+Wong https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=render_article&arttitle=Adele+Wong&returnto=Adele+Wong&oldid=583442184&writer=rdf2latex https://en.wikipedia.org/w/index.php?title=Adele_Wong&printable=yes https://en.wikipedia.org/wiki/Adele_Wong# https://www.wikidata.org/wiki/Q4681907#sitelinks-wikipedia https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License https://creativecommons.org/licenses/by-sa/3.0/ https://wikimediafoundation.org/wiki/Terms_of_Use https://wikimediafoundation.org/wiki/Privacy_policy https://www.wikimediafoundation.org/ https://wikimediafoundation.org/wiki/Privacy_policy https://en.wikipedia.org/wiki/Wikipedia:About https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer https://en.wikipedia.org/wiki/Wikipedia:Contact_us https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute https://wikimediafoundation.org/wiki/Cookie_statement https://en.m.wikipedia.org/w/index.php?title=Adele_Wong&mobileaction=toggle_view_mobile https://wikimediafoundation.org/ https://www.mediawiki.org/