java爬虫 之 搜狐新闻爬虫（一）

最近开始学习java爬虫，网上很多教程，自己找的时候花了好久的时间去理解别人的思路。

打算将自己最近的学习进度稍作整理，理清思路。

主要工具使用jsoup：具体用法看http://blog.csdn.net/u012315428/article/details/51135640

下面是获取一个网页中所有的超链接：

package com.sohu;

import org.jsoup.Jsoup;

import java.io.IOException;
import java.util.*;

import org.jsoup.*;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
/*
* author :cj
* 找到所有的超链接
*/
public class FindAllUrl {

public static void main(String[] args) {
// TODO Auto-generated method stub
try {
Document doc = Jsoup.connect("http://news.sohu.com/").get();
Elements links = doc.select("a[href]");
for(Element link : links){
String StrUrl = link.attr("abs:href");
if(StrUrl.startsWith("http://news.sohu.com/"))//以。。。开头的url
System.out.println(StrUrl);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

}

}

运行结果
http://news.sohu.com/ http://news.sohu.com/mindiao/ http://news.sohu.com/scroll/ http://news.sohu.com/mindiao/ http://news.sohu.com/special.shtml http://news.sohu.com/guoneixinwen.shtml http://news.sohu.com/shehuixinwen.shtml http://news.sohu.com/guojixinwen.shtml http://news.sohu.com/matrix/ http://news.sohu.com/newsmaker_list/ http://news.sohu.com/photo/ http://news.sohu.com/wurenji/ http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/20160414/n444127123.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160414/n444193395.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444133304.shtml http://news.sohu.com/20160414/n444199124.shtml http://news.sohu.com/20160413/n444107224.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160413/n444105842.shtml http://news.sohu.com/20160414/n444140620.shtml http://news.sohu.com/20160414/n444126073.shtml http://news.sohu.com/20160413/n444086783.shtml http://news.sohu.com/20160414/n444187234.shtml http://news.sohu.com/20160414/n444193015.shtml http://news.sohu.com/20160414/n444207393.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444193395.shtml