爬虫初学

这几天学了下爬虫,也稍微的会了点,做了个爬小说的,这里放个简单的爬取

string html = string.Empty;
try
{
System.Net.ServicePointManager.DefaultConnectionLimit = 50;
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Timeout = 30 * 1000;//等待时间
request.UserAgent = "";//这里填浏览器的就行了
request.ContentType= "text/html;charset=gbk";//转码方式
request.Method = "GET";
Encoding encoding = Encoding.Default;
using (HttpWebResponse r = request.GetResponse() as HttpWebResponse)
{
if (r.StatusCode!=HttpStatusCode.OK)
{
html = "失败";
}
else
{
try
{
StreamReader sr = new StreamReader(r.GetResponseStream(), encoding);
html = sr.ReadToEnd();
sr.Close();
}
catch (Exception ex)
{
html = ex.Message;
}
}
}
}
catch (Exception ex)
{
html = ex.Message;
}
return html;

然后就获得了爬取的数据,数据在html中,然后就要处理这个字符串,用正则表达式处理即可
然后发现多线程爬数据时.偶尔报连接中断,查了很多地方没查到原因,然后发现用这段代码就可以了,估计是多线程的线程太多了.超过了允许请求的上限了

System.Net.ServicePointManager.DefaultConnectionLimit = 50;

• 点赞
• 收藏
• 分享
• 文章举报