正则表达式——抓取网页中的http及网络图片

**

正则表达式——抓取网页中的http及网络图片

**

最近网上有些朋友问我怎么用JAVA抓起网页中的image地址，我现在就给大家写一个实例，希望大家能通过这个实例学习到一点东西

需要抓取网页中http链接地址首先还需准备一个网站地址，我们就以hao123为例

https://www.hao123.com/



在这个网页中，我们要抓取这里面的href地址首先需要解析这个网站中的内容，把它转换成String字符串

@Test
public void parsing_URL() {
System.out.println(getHTMLText("https://www.hao123.com/","utf-8"));
}

/**
* 获取网页文本
* */
public String getHTMLText(String htmlUrl,String charset){

StringBuilder sb = new StringBuilder();
try {
URL url = new URL(htmlUrl);
BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream(),charset));
String result = "";
while ((result = br.readLine()) != null){
sb.append(result).append("\n");
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

return sb.toString();
}

打印结果如下



然后我们在通过正则表达式来解析网站中存在的href对应的地址

@Test
public void parsing_URL() {
String content = getHTMLText("https://www.hao123.com/","gbk");
String regular = "href=\"([\\w\\s./:]+?)\"";
List<String> list = regularParsingResult(content,regular);
for (String str:list) {
System.out.println(str);
}
}

/**
* 正则解析结果
*/
public List<String> regularParsingResult(String content,String regular){
Pattern pattern = Pattern.compile(regular);
Matcher matcher = pattern.matcher(content);
List<String>  list = new ArrayList<>();
while (matcher.find()){
list.add(matcher.group(1));
}
return list;
}

打印结果如下



现在我们来获取一些网页中的img标签下的图片地址

@Test
public void parsing_URL() {
String content = getHTMLText("https://www.hao123.com/","utf-8");
String regular = "<img[\\s\\S]+?src=\"(.+?)\"";
List<String> list = regularParsingResult(content,regular);
for (String str:list) {
System.out.println(str);
}
}

/**
* 正则解析结果
*/
public List<String> regularParsingResult(String content,String regular){
Pattern pattern = Pattern.compile(regular);
Matcher matcher = pattern.matcher(content);
List<String>  list = new ArrayList<>();
while (matcher.find()){
list.add(matcher.group(1));
}
return list;
}

打印结果



获取网页完整代码

public void getHTMLTextComplete(String htmlUrl) throws InterruptedException {
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_10);
//设置webClient的相关参数
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setCssEnabled(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
//webClient.getOptions().setTimeout(50000);
webClient.getOptions().setThrowExceptionOnScriptError(false);
//模拟浏览器打开一个目标网址
HtmlPage rootPage = webClient.getPage(htmlUrl);
System.out.println("为了获取js执行的数据 线程开始沉睡等待");
Thread.sleep(3000);//主要是这个线程的等待 因为js加载也是需要时间的
System.out.println("线程结束沉睡");
String html = rootPage.asText();
System.out.println(html);
}