Python/网络编程/多线程与多进程07

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/qq_39591494/article/details/89295074

Thread类：

  Thread类是threading模块中最常用的功能，通过调用用户指定函数

func

，用于独立生成一个活动线程。调用用户指定函数

func

，可以有两种方法：一是在Thread创建实例对象时，把

func

以参数的形式传递给构造函数；二是通过继承

Thread

类重写

run

方法，调用

func

函数，在

Thread

的子类中，只允许

__init__()

和

run()

方法进行重写。

Thread(group=None, target=None, name=None, args=(), kwargs=None, daemon=None)

• group：用于保留，作为将来的扩展功能，可以忽略该参数，或设置group=none
• target：设置线程需要执行的自定义函数，如target=func设置完成后被run()方法调用; 当target=none时，线程不执行任何操作。
• name：指定需要执行的线程名称。不指定时，该类自动生成一个Thread-N形式的线程名称。
• args：当自定义函数func带有参数时，把参数以元组的形式通过args传递给func
• kwargs：当自定义函数的func带有参数时，把参数以字典的形式通过kwargs传递给func
• daemon：当daemon不是none时，通过设置True&False确定是否守护进程。

Thread类的主要方法：

方法名称	使用描述
start()	线程启动状态（一个线程对象只能调用该方法一次），该方法必须在run()方法前被调用
run()	运行线程，使线程处于活动状态，在run()方法里执行指定的用户自定义函数func，该方法可以在Thread子类被重写
join(timeout=None)	阻塞调用线程。等待调用该方法的线程对象运行，一直到该线程执行终止，阻塞才释放。timeout为可选参数，可以设置阻塞时间（以秒为单位）当timeout参数不存在或者为none时，该操作将阻塞，直到线程终止。该方法运行在run()方法后
name	线程名称，初始名称由构造函数设置
is_alive()	返回线程是否存在
daemon	显示此线程是否为守护程序线程(True)或不(False)的布尔值。这必须在调用start() 之前设置，否则会引发runtimeError。

一旦一个线程对象被创建，其活动必须通过调用线程的start()方法来启动，然后调用run()方法执行指定的用户自定义函数func。join方法在run()方法后执行。这会阻塞调用线程，直到调用join方法的线程运行终止，才能执行后续的程序代码。

线程池

  线程池是一种多线程处理形式，处理过程中将任务添加到队列，然后在创建线程后自动启动这些任务。线程池线程都是后台线程。每个线程都使用默认的堆栈大小，以默认的优先级运行，并处于多线程单元中。如果某个线程在托管代码中空闲（如正在等待某个事件）,则线程池将插入另一个辅助线程来使所有处理器保持繁忙。如果所有线程池线程都始终保持繁忙，但队列中包含挂起的工作，则线程池将在一段时间后创建另一个辅助线程但线程的数目永远不会超过最大值。超过最大值的线程可以排队，但他们要等到其他线程完成后才启动。

关于：

concurrent.futures

:https://devdocs.io/python~3.6/library/concurrent.futures

#!/usr/bin/env python
# -*- coding:utf-8 -*-

from concurrent.futures import ThreadPoolExecutor # 导入

ThreadPoolExecutor 源代码

class ThreadPoolExecutor(_base.Executor):

# Used to assign unique thread names when thread_name_prefix is not supplied.
_counter = itertools.count().__next__

def __init__(self, max_workers=None, thread_name_prefix=''):
"""Initializes a new ThreadPoolExecutor instance.

Args:
max_workers: The maximum number of threads that can be used to
execute the given calls.
thread_name_prefix: An optional name prefix to give our threads.
"""
if max_workers is None:
# Use this number because ThreadPoolExecutor is often
# used to overlap I/O instead of CPU work.
max_workers = (os.cpu_count() or 1) * 5
if max_workers <= 0:
raise ValueError("max_workers must be greater than 0")

self._max_workers = max_workers
self._work_queue = queue.Queue()
self._threads = set()
self._shutdown = False
self._shutdown_lock = threading.Lock()
self._thread_name_prefix = (thread_name_prefix or
("ThreadPoolExecutor-%d" % self._counter()))

max_workers=None

一共有多少个线程可以来执行任务，如果这个线程是

None

max_workers = CPU的核数 * 5

thread_name_prefix

线程的的前缀名称默认Thread-x

submit源码

def submit(self, fn, *args, **kwargs):
with self._shutdown_lock:
if self._shutdown:
raise RuntimeError('cannot schedule new futures after shutdown')

f = _base.Future()
w = _WorkItem(f, fn, args, kwargs)

self._work_queue.put(w)
self._adjust_thread_count()
return f
submit.__doc__ = _base.Executor.submit.__doc__

运行如下：

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests # 导入requests模块
# 从concurrent.futures 导入ThreadPollExecutor and as_completed 模块
from concurrent.futures import ThreadPoolExecutor, as_completed

urls = [
'http://www.baidu.com',
'http://www.taobao.com',
'http://www.jd.com',
'http://www.qq.com',
'http://www.csdn.net',
'http://www.360.com',
'http://edu.51cto.com'
]

def download(url):
"""接收用户传递参数：URL Url来源于urls_list

使用requests.get 获取url，退出并返回url及get status
"""
r = requests.get(url)
return url, r.status_code

def main():
"""as_completed:
An iterator over the given futures that yields each as it completes.
"""
with ThreadPoolExecutor(5) as executor:
# function_name args kw
future = [executor.s
20000
ubmit(download, url) for url in urls]
for i in as_completed(futures):
print(i.result())

if __name__ == "__main__":
main()
>>>
('http://www.baidu.com', 200)
('http://www.jd.com', 200)
('http://www.360.com', 200)
('http://www.qq.com', 200)
('http://www.csdn.net', 200)
('http://www.taobao.com', 200)
('http://edu.51cto.com', 200)

加入log：

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests
import logging
from concurrent.futures import ThreadPoolExecutor, as_completed

logging.basicConfig(
level=logging.DEBUG,
format='%(threadName)-10s:%(message)s'
)

urls = [
'http://www.baidu.com',
'http://www.taobao.com',
'http://www.jd.com',
'http://www.qq.com',
'http://www.csdn.net',
'http://www.360.com',
'http://edu.51cto.com'
]

def download(url):
r = requests.get(url)
return url, r.status_code

def main():
with ThreadPoolExecutor(5) as executor:
# function_name args kw
futures = [executor.submit(download, url) for url in urls]
for i in as_completed(futures):
print(i.result())

if __name__ == "__main__":
main()
>>>
ThreadPoolExecutor-0_1:Starting new HTTP connection (1): www.taobao.com:80
ThreadPoolExecutor-0_4:Starting new HTTP connection (1): www.csdn.net:80
ThreadPoolExecutor-0_3:Starting new HTTP connection (1): www.qq.com:80
ThreadPoolExecutor-0_0:Starting new HTTP connection (1): www.baidu.com:80
ThreadPoolExecutor-0_2:Starting new HTTP connection (1): www.jd.com:80
ThreadPoolExecutor-0_4:http://www.csdn.net:80 "GET / HTTP/1.1" 301 182
ThreadPoolExecutor-0_4:Starting new HTTPS connection (1): www.csdn.net:443
ThreadPoolExecutor-0_0:http://www.baidu.com:80 "GET / HTTP/1.1" 200 None
('http://www.baidu.com', 200)
ThreadPoolExecutor-0_0:Starting new HTTP connection (1): www.360.com:80
ThreadPoolExecutor-0_3:http://www.qq.com:80 "GET / HTTP/1.1" 302 169
ThreadPoolExecutor-0_2:http://www.jd.com:80 "GET / HTTP/1.1" 302 157
ThreadPoolExecutor-0_3:Starting new HTTPS connection (1): www.qq.com:443
ThreadPoolExecutor-0_2:Starting new HTTPS connection (1): www.jd.com:443
ThreadPoolExecutor-0_1:http://www.taobao.com:80 "GET / HTTP/1.1" 302 258
ThreadPoolExecutor-0_0:http://www.360.com:80 "GET / HTTP/1.1" 301 178
ThreadPoolExecutor-0_1:Starting new HTTPS connection (1): www.taobao.com:443
ThreadPoolExecutor-0_0:Starting new HTTPS connection (1): www.360.cn:443
ThreadPoolExecutor-0_2:https://www.jd.com:443 "GET / HTTP/1.1" 200 25679
('http://www.jd.com', 200)
ThreadPoolExecutor-0_2:Starting new HTTP connection (1): edu.51cto.com:80
ThreadPoolExecutor-0_0:https://www.360.cn:443 "GET / HTTP/1.1" 200 None
ThreadPoolExecutor-0_3:https://www.qq.com:443 "GET / HTTP/1.1" 200 None
('http://www.360.com', 200)
('http://www.qq.com', 200)
ThreadPoolExecutor-0_1:https://www.taobao.com:443 "GET / HTTP/1.1" 200 None
ThreadPoolExecutor-0_2:http://edu.51cto.com:80 "GET / HTTP/1.1" 302 None
ThreadPoolExecutor-0_2:Starting new HTTPS connection (1): edu.51cto.com:443
('http://www.taobao.com', 200)
ThreadPoolExecutor-0_4:https://www.csdn.net:443 "GET / HTTP/1.1" 200 None
('http://www.csdn.net', 200)
ThreadPoolExecutor-0_2:https://edu.51cto.com:443 "GET / HTTP/1.1" 200 None
('http://edu.51cto.com', 200)

自定义log前缀名称：

with ThreadPoolExecutor(5, 'yankerp') as executor:

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests
import logging
from concurrent.futures import ThreadPoolExecutor, as_completed

logging.basicConfig(
level=logging.DEBUG,
format='%(threadName)-10s:%(message)s'
)

urls = [
'http://www.baidu.com',
'http://www.taobao.com',
'http://www.jd.com',
'http://www.qq.com',
'http://www.csdn.net',
'http://www.360.com',
'http://edu.51cto.com'
]

def download(url):
r = requests.get(url)
return url, r.status_code

def main():
with ThreadPoolExecutor(5, 'yankerp') as executor:
# function_name args kw
futures = [executor.submit(download, url) for url in urls]
for i in as_completed(futures):
print(i.result())

if __name__ == "__main__":
main()
>>>
yankerp_2 :Starting new HTTP connection (1): www.jd.com:80
yankerp_1 :Starting new HTTP connection (1): www.taobao.com:80
yankerp_0 :Starting new HTTP connection (1): www.baidu.com:80
yankerp_4 :Starting new HTTP connection (1): www.csdn.net:80
yankerp_3 :Starting new HTTP connection (1): www.qq.com:80
yankerp_2 :http://www.jd.com:80 "GET / HTTP/1.1" 302 157
yankerp_2 :Starting new HTTPS connection (1): www.jd.com:443
yankerp_0 :http://www.baidu.com:80 "GET / HTTP/1.1" 200 None
('http://www.baidu.com', 200)
yankerp_0 :Starting new HTTP connection (1): www.360.com:80
yankerp_4 :http://www.csdn.net:80 "GET / HTTP/1.1" 301 182
yankerp_4 :Starting new HTTPS connection (1): www.csdn.net:443
yankerp_3 :http://www.qq.com:80 "GET / HTTP/1.1" 302 169
yankerp_3 :Starting new HTTPS connection (1): www.qq.com:443
yankerp_0 :http://www.360.com:80 "GET / HTTP/1.1" 301 178
yankerp_1 :http://www.taobao.com:80 "GET / HTTP/1.1" 302 258
yankerp_0 :Starting new HTTPS connection (1): www.360.cn:443
yankerp_1 :Starting new HTTPS connection (1): www.taobao.com:443
yankerp_3 :https://www.qq.com:443 "GET / HTTP/1.1" 200 None
yankerp_2 :https://www.jd.com:443 "GET / HTTP/1.1" 200 25722
('http://www.jd.com', 200)
yankerp_0 :https://www.360.cn:443 "GET / HTTP/1.1" 200 None
yankerp_2 :Starting new HTTP connection (1): edu.51cto.com:80
('http://www.360.com', 200)
('http://www.qq.com', 200)
yankerp_1 :https://www.taobao.com:443 "GET / HTTP/1.1" 200 None
('http://www.taobao.com', 200)
yankerp_2 :http://edu.51cto.com:80 "GET / HTTP/1.1" 302 None
yankerp_4 :https://www.csdn.net:443 "GET / HTTP/1.1" 200 None
yankerp_2 :Starting new HTTPS connection (1): edu.51cto.com:443
('http://www.csdn.net', 200)
yankerp_2 :https://edu.51cto.com:443 "GET / HTTP/1.1" 200 None
('http://edu.51cto.com', 200)

加lock及异常

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests
import logging
import threading
from threading import Thread
from concurrent.futures import ThreadPoolExecutor, as_completed

logging.basicConfig(
level=logging.DEBUG,
format='%(threadName)-10s:%(message)s'
)

urls = [
'http://www.baidu.com',
'http://www.taobao.com',
'http://www.jd.com',
'http://www.qq.com',
'http://www.csdn.net',
'http://www.360.com',
'http://edu.51cto.com'
]

lock = threading.Lock()

def download(url):
with lock:
r = requests.get(url)
return url, r.status_code

def main():
with ThreadPoolExecutor(5, 'yankerp') as executor:
# function_name args kw
futures = [executor.submit(download, url) for url in urls]
try:
for i in as_completed(futures):
print(i.result())
except Exception as e:
print(e)

if __name__ == "__main__":
main()
>>>
yankerp_0 :Starting new HTTP connection (1): www.baidu.com:80
yankerp_0 :http://www.baidu.com:80 "GET / HTTP/1.1" 200 None
('http://www.baidu.com', 200)
yankerp_1 :Starting new HTTP connection (1): www.taobao.com:80
yankerp_1 :http://www.taobao.com:80 "GET / HTTP/1.1" 302 258
yankerp_1 :Starting new HTTPS connection (1): www.taobao.com:443
yankerp_1 :https://www.taobao.com:443 "GET / HTTP/1.1" 200 None
('http://www.taobao.com', 200)
yankerp_2 :Starting new HTTP connection (1): www.jd.com:80
yankerp_2 :http://www.jd.com:80 "GET / HTTP/1.1" 302 157
yankerp_2 :Starting new HTTPS connection (1): www.jd.com:443
yankerp_2 :https://www.jd.com:443 "GET / HTTP/1.1" 200 25693
('http://www.jd.com', 200)
yankerp_3 :Starting new HTTP connection (1): www.qq.com:80
yankerp_3 :http://www.qq.com:80 "GET / HTTP/1.1" 302 169
yankerp_3 :Starting new HTTPS connection (1): www.qq.com:443
yankerp_3 :https://www.qq.com:443 "GET / HTTP/1.1" 200 None
('http://www.qq.com', 200)
yankerp_4 :Starting new HTTP connection (1): www.csdn.net:80
yankerp_4 :http://www.csdn.net:80 "GET / HTTP/1.1" 301 182
yankerp_4 :Starting new HTTPS connection (1): www.csdn.net:443
yankerp_4 :https://www.csdn.net:443 "GET / HTTP/1.1" 200 None
('http://www.csdn.net', 200)
yankerp_0 :Starting new HTTP connection (1): www.360.com:80
yankerp_0 :http://www.360.com:80 "GET / HTTP/1.1" 301 178
yankerp_0 :Starting new HTTPS connection (1): www.360.cn:443
yankerp_0 :https://www.360.cn:443 "GET / HTTP/1.1" 200 None
('http://www.360.com', 200)
yankerp_1 :Starting new HTTP connection (1): edu.51cto.com:80
yankerp_1 :http://edu.51cto.com:80 "GET / HTTP/1.1" 302 None
yankerp_1 :Starting new HTTPS connection (1): edu.51cto.com:443
yankerp_1 :https://edu.51cto.com:443 "GET / HTTP/1.1" 200 None
('http://edu.51cto.com', 200)

优化参数：

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests
import logging
import threading
from threading import Thread
from concurrent.futures import ThreadPoolExecutor, as_completed

logging.basicConfig(
level=logging.DEBUG,
format='%(threadName)-10s:%(message)s'
)

urls = [
'http://www.baidu.com',
'http://www.taobao.com',
'http://www.jd.com',
'http://www.qq.com',
'http://www.csdn.net',
'http://www.360.com',
'http://edu.51cto.com'
]

lock = threading.Lock()

def download(url):
with lock:
r = requests.get(url)
return url, r.status_code

def main():
with ThreadPoolExecutor(5, 'yankerp') as executor:
# function_name args kw
# futures = [executor.submit(download, url) for url in urls]
futures = executor.map(download, urls)
try:
for i in futures:
print(i)
except Exception as e:
print(e)

if __name__ == "__main__":
main()
>>>
yankerp_0 :Starting new HTTP connection (1): www.baidu.com:80
yankerp_0 :http://www.baidu.com:80 "GET / HTTP/1.1" 200 None
('http://www.baidu.com', 200)
yankerp_1 :Starting new HTTP connection (1): www.taobao.com:80
yankerp_1 :http://www.taobao.com:80 "GET / HTTP/1.1" 302 258
yankerp_1 :Starting new HTTPS connection (1): www.taobao.com:443
yankerp_1 :https://www.taobao.com:443 "GET / HTTP/1.1" 200 None
('http://www.taobao.com', 200)
yankerp_2 :Starting new HTTP connection (1): www.jd.com:80
yankerp_2 :http://www.jd.com:80 "GET / HTTP/1.1" 302 157
yankerp_2 :Starting new HTTPS connection (1): www.jd.com:443
yankerp_2 :https://www.jd.com:443 "GET / HTTP/1.1" 200 25764
('http://www.jd.com', 200)
yankerp_3 :Starting new HTTP connection (1): www.qq.com:80
yankerp_3 :http://www.qq.com:80 "GET / HTTP/1.1" 302 169
yankerp_3 :Starting new HTTPS connection (1): www.qq.com:443
yankerp_3 :https://www.qq.com:443 "GET / HTTP/1.1" 200 None
('http://www.qq.com', 200)
yankerp_4 :Starting new HTTP connection (1): www.csdn.net:80
yankerp_4 :http://www.csdn.net:80 "GET / HTTP/1.1" 301 182
yankerp_4 :Starting new HTTPS connection (1): www.csdn.net:443
yankerp_4 :https://www.csdn.net:443 "GET / HTTP/1.1" 200 None
('http://www.csdn.net', 200)
yankerp_0 :Starting new HTTP connection (1): www.360.com:80
yankerp_0 :http://www.360.com:80 "GET / HTTP/1.1" 301 178
yankerp_0 :Starting new HTTPS connection (1): www.360.cn:443
yankerp_0 :https://www.360.cn:443 "GET / HTTP/1.1" 200 None
('http://www.360.com', 200)
yankerp_1 :Starting new HTTP connection (1): edu.51cto.com:80
yankerp_1 :http://edu.51cto.com:80 "GET / HTTP/1.1" 302 None
yankerp_1 :Starting new HTTPS connection (1): edu.51cto.com:443
yankerp_1 :https://edu.51cto.com:443 "GET / HTTP/1.1" 200 None
('http://edu.51cto.com', 200)