python 多线程编程总结（实验多线程判断网址是否在线）

现在做一个针对网址是否在线的判断实验，利用多线程和普通方法来进行对比，以下为代码和代码结果：

一，不使用多线程，代码如下：

#encoding:utf-8

import threading

import urllib2

def online(url = ''):

"""判断网址是否在线"""

req = urllib2.Request(url)

try:

response=urllib2.urlopen(req)

if response.code == 200:

print response.geturl(),' this url is online'

else:

print 'not'

except urllib2.URLError as e:

if hasattr(e, 'reason'):

print url,' We failed to reach a server.'

print 'Reason: ', e.reason

elif hasattr(e, 'code'):

print url,' The server couldn\'t fulfill the request.'

print 'Error code: ', e.code

def main():

url_list = ['http://www.baidu.com','http://www.hitwh.edu.cn','http://www.13.com','http://www.ifeng.com','http://www.sina.com',

'http://www.wewin.com.gr/2','http://www.ifeng.com','http://www.sina.com','http://www.zeeif.com/int/',

'http://www.zeeif.com/websc/verification/',

'http://mjgds.org/classrooms/wp-content/plugins/10421312312/19890907.html',

'http://login-resolution-center-case-475ec2aec1br.propesage-algerie.com/ID',

'http://paypel-login-resolution-center.propesage-algerie.com/ID/',

'http://radiotransilvania.ro/clujarena/rena.php',

'http://kuleteknik.net/wp-includes/lol3.html',

'http://kuleteknik.net/wp-includes/lol2.html'

]

for url in url_list:

#t = threading.Thread(target = online,args = (url,))

#t.start()

online(url)

if __name__ == '__main__':

main()

结果如下：
http://www.baidu.com this url is online
http://www.hitwh.edu.cn this url is online
http://www.13.com We failed to reach a server.

Reason: [Errno 11001] getaddrinfo failed
http://www.ifeng.com this url is online
http://www.sina.com.cn/ this url is online
http://www.wewin.com.gr/2 We failed to reach a server.

Reason: Unauthorized
http://www.ifeng.com this url is online
http://www.sina.com.cn/ this url is online
http://www.zeeif.com/int/ We failed to reach a server.

Reason: Not Found
http://www.zeeif.com/websc/verification/ We failed to reach a server.

Reason: Not Found
http://mjgds.org/classrooms/wp-content/plugins/10421312312/19890907.html We failed to reach a server.

Reason: Internal Server Error
http://login-resolution-center-case-475ec2aec1br.propesage-algerie.com/ID this url is online
http://paypel-login-resolution-center.propesage-algerie.com/ID/ this url is online
http://radiotransilvania.ro/clujarena/rena.php We failed to reach a server.

Reason: Not Found
http://kuleteknik.net/wp-includes/lol3.html this url is online
http://kuleteknik.net/wp-includes/lol2.html this url is online

[Finished in 5.2s]

解释：使用了5.2秒，若判断网址更多，并且其中没有在线的网址更多时，时间会更长

二、使用多线程判断，代码如下：

#encoding:utf-8

import threading

import urllib2

def online(url = ''):

"""判断网址是否在线"""

req = urllib2.Request(url)

try:

response=urllib2.urlopen(req)

if response.code == 200:

print response.geturl(),' this url is online'

else:

print 'not'

except urllib2.URLError as e:

if hasattr(e, 'reason'):

print url,' We failed to reach a server.'

print 'Reason: ', e.reason

elif hasattr(e, 'code'):

print url,' The server couldn\'t fulfill the request.'

print 'Error code: ', e.code

def main():

url_list = ['http://www.baidu.com','http://www.hitwh.edu.cn','http://www.13.com','http://www.ifeng.com','http://www.sina.com',

'http://www.wewin.com.gr/2','http://www.ifeng.com','http://www.sina.com','http://www.zeeif.com/int/',

'http://www.zeeif.com/websc/verification/',

'http://mjgds.org/classrooms/wp-content/plugins/10421312312/19890907.html',

'http://login-resolution-center-case-475ec2aec1br.propesage-algerie.com/ID',

'http://paypel-login-resolution-center.propesage-algerie.com/ID/',

'http://radiotransilvania.ro/clujarena/rena.php',

'http://kuleteknik.net/wp-includes/lol3.html',

'http://kuleteknik.net/wp-includes/lol2.html'

]

for url in url_list:

t = threading.Thread(target = online,args = (url,))

t.start()

#online(url)

if __name__ == '__main__':

main()

结果如下：
http://www.baidu.com this url is online
http://www.ifeng.com this url is online
http://www.13.com We failed to reach a server.

Reason: [Errno 11001] getaddrinfo failed
http://www.ifeng.com this url is online
http://paypel-login-resolution-center.propesage-algerie.com/ID/ this url is online
http://www.hitwh.edu.cn this url is online
http://login-resolution-center-case-475ec2aec1br.propesage-algerie.com/ID this url is online
http://www.sina.com.cn/ this url is online
http://www.sina.com.cn/ this url is online
http://mjgds.org/classrooms/wp-content/plugins/10421312312/19890907.html We failed to reach a server.

Reason: Internal Server Error
http://www.zeeif.com/websc/verification/ We failed to reach a server.

Reason: Not Found
http://www.zeeif.com/int/ We failed to reach a server.

Reason: Not Found
http://kuleteknik.net/wp-includes/lol2.html this url is online
http://kuleteknik.net/wp-includes/lol3.html this url is online
http://www.wewin.com.gr/2 We failed to reach a server.

Reason: Unauthorized
http://radiotransilvania.ro/clujarena/rena.php We failed to reach a server.

Reason: Not Found

[Finished in 1.7s]

解释：每一个网址判断都使用一个线程执行，时间只用了1.7s

总结：

1、当判断的网址多时，数量级达到百万级，多线程的优势会显现的非常大。

2、该多线程代码是为每一个网址创建一个线程，当网址过多时，很显然这个方法不行，所以可以优化该判断代码。

3、当网址存在数据库中时候，如何高效存入数据库，也是很重要的方法。

4、上面判断网址是否在线的函数，个人觉得不是非常正确，因为网址重定向的问题，网址可能不存在，但是重定向后，显示网址还存在，这也是以后改进方法，有改进办法的同学可以跟我留言，共同进步，如果我有方法，也会在博客公开。

更新（2014.10.30）

1、使用pycurl检测url是否在线，效率更高。

2、将其连接数据库，并且将结果存入数据库（自己做的小项目，已经完成）