python 自动化之路 day 09 进程、线程、协程篇
2016-12-11 09:37
671 查看
本节内容
操作系统发展史介绍
进程、与线程区别
python GIL全局解释器锁
线程
语法
join
线程锁之Lock\Rlock\信号量
将线程变为守护进程
Event事件
queue队列
生产者消费者模型
开发一个线程池
进程
语法
进程间通讯
进程池
手工操作
程序员将对应于程序和数据的已穿孔的纸带(或卡片)装入输入机,然后启动输入机把程序和数据输入计算机内存,接着通过控制台开关启动程序针对数据运行;计算完毕,打印机输出计算结果;用户取走结果并卸下纸带(或卡片)后,才让下一个用户上机。
View Code
class
Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.
The lowest valued entries are retrieved first (the lowest valued entry is the one returned by
exception
Exception raised when non-blocking
exception
Exception raised when non-blocking
Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the
Equivalent to
Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the
Equivalent to
Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.
Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each
If a
Raises a
为什么要使用生产者和消费者模式
在线程世界里,生产者就是生产数据的线程,消费者就是消费数据的线程。在多线程开发当中,如果生产者处理速度很快,而消费者处理速度很慢,那么生产者就必须等待消费者处理完,才能继续生产数据。同样的道理,如果消费者的处理能力大于生产者,那么消费者就必须等待生产者。为了解决这个问题于是引入了生产者和消费者模式。
什么是生产者消费者模式
生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯,而通过阻塞队列来进行通讯,所以生产者生产完数据之后不用等待消费者处理,直接扔给阻塞队列,消费者不找生产者要数据,而是直接从阻塞队列里取,阻塞队列就相当于一个缓冲区,平衡了生产者和消费者的处理能力。
下面来学习一个最基本的生产者消费者模型的例子
生产者消费者模型的2个主要作用:
1、程序的解耦
2、异步、提高效率
To show the individual process IDs involved, here is an expanded example:
Queues
使用方法跟threading里的queue差不多
Pipes
The
The two connection objects returned by
Managers
A manager object returned by
A manager returned by
进程同步
Without using the lock output from the different processes is liable to get all mixed up.
进程池中有两个方法:
apply
apply_async
需求:
主机分组
主机信息配置文件用configparser解析
可批量执行命令、发送文件,结果实时返回,执行格式如下
batch_run -h h1,h2,h3 -g web_clusters,db_servers -cmd "df -h"
batch_scp -h h1,h2,h3 -g web_clusters,db_servers -action put -local test.py -remote /tmp/
主机用户名密码、端口可以不同
执行远程命令使用paramiko模块
批量命令需使用multiprocessing并发
操作系统发展史介绍
进程、与线程区别
python GIL全局解释器锁
线程
语法
join
线程锁之Lock\Rlock\信号量
将线程变为守护进程
Event事件
queue队列
生产者消费者模型
开发一个线程池
进程
语法
进程间通讯
进程池
操作系统发展史
手工操作(无操作系统)
1946年第一台计算机诞生--20世纪50年代中期,还未出现操作系统,计算机工作采用手工操作方式。手工操作
程序员将对应于程序和数据的已穿孔的纸带(或卡片)装入输入机,然后启动输入机把程序和数据输入计算机内存,接着通过控制台开关启动程序针对数据运行;计算完毕,打印机输出计算结果;用户取走结果并卸下纸带(或卡片)后,才让下一个用户上机。
#_*_coding:utf-8_*_ __author__ = 'Alex Li' import threading import time import random def door(): door_open_time_counter = 0 while True: if door_swiping_event.is_set(): print("\033[32;1mdoor opening....\033[0m") door_open_time_counter +=1 else: print("\033[31;1mdoor closed...., swipe to open.\033[0m") door_open_time_counter = 0 #清空计时器 door_swiping_event.wait() if door_open_time_counter > 3:#门开了已经3s了,该关了 door_swiping_event.clear() time.sleep(0.5) def staff(n): print("staff [%s] is comming..." % n ) while True: if door_swiping_event.is_set(): print("\033[34;1mdoor is opened, passing.....\033[0m") break else: print("staff [%s] sees door got closed, swipping the card....." % n) print(door_swiping_event.set()) door_swiping_event.set() print("after set ",door_swiping_event.set()) time.sleep(0.5) door_swiping_event = threading.Event() #设置事件 door_thread = threading.Thread(target=door) door_thread.start() for i in range(5): p = threading.Thread(target=staff,args=(i,)) time.sleep(random.randrange(3)) p.start()
View Code
queue队列
queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.class
queue.
Queue(maxsize=0) #先入先出class
queue.
LifoQueue(maxsize=0) #last in fisrt out class
queue.
PriorityQueue(maxsize=0) #存储数据时可设置优先级的队列
Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.
The lowest valued entries are retrieved first (the lowest valued entry is the one returned by
sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form:
(priority_number, data).
exception
queue.
Empty
Exception raised when non-blocking
get()(or
get_nowait()) is called on a
Queueobject which is empty.
exception
queue.
Full
Exception raised when non-blocking
put()(or
put_nowait()) is called on a
Queueobject which is full.
Queue.
qsize()
Queue.
empty() #return True if empty
Queue.
full() # return True if full
Queue.
put(item, block=True, timeout=None)
Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the
Fullexception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the
Fullexception (timeout is ignored in that case).
Queue.
put_nowait(item)
Equivalent to
put(item, False).
Queue.
get(block=True, timeout=None)
Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the
Emptyexception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the
Emptyexception (timeout is ignored in that case).
Queue.
get_nowait()
Equivalent to
get(False).
Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.
Queue.
task_done()
Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each
get()used to fetch a task, a subsequent call to
task_done()tells the queue that the processing on the task is complete.
If a
join()is currently blocking, it will resume when all items have been processed (meaning that a
task_done()call was received for every item that had been
put()into the queue).
Raises a
ValueErrorif called more times than there were items placed in the queue.
Queue.
join() block直到queue被消费完毕
生产者消费者模型
在并发编程中使用生产者和消费者模式能够解决绝大多数并发问题。该模式通过平衡生产线程和消费线程的工作能力来提高程序的整体处理数据的速度。为什么要使用生产者和消费者模式
在线程世界里,生产者就是生产数据的线程,消费者就是消费数据的线程。在多线程开发当中,如果生产者处理速度很快,而消费者处理速度很慢,那么生产者就必须等待消费者处理完,才能继续生产数据。同样的道理,如果消费者的处理能力大于生产者,那么消费者就必须等待生产者。为了解决这个问题于是引入了生产者和消费者模式。
什么是生产者消费者模式
生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯,而通过阻塞队列来进行通讯,所以生产者生产完数据之后不用等待消费者处理,直接扔给阻塞队列,消费者不找生产者要数据,而是直接从阻塞队列里取,阻塞队列就相当于一个缓冲区,平衡了生产者和消费者的处理能力。
下面来学习一个最基本的生产者消费者模型的例子
1、程序的解耦
2、异步、提高效率
多进程multiprocessing
multiprocessingis a package that supports spawning processes using an API similar to the
threadingmodule. The
multiprocessingpackage offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the
multiprocessingmodule allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
进程间通讯
不同进程间内存是不共享的,要想实现两个进程间的数据交换,可以用以下方法:Queues
使用方法跟threading里的queue差不多
The
Pipe()function returns a pair of connection objects connected by a pipe which by default is duplex (two-way). For example:
The two connection objects returned by
Pipe()represent the two ends of the pipe. Each connection object has
send()and
recv()methods (among others). Note that data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time.
Managers
A manager object returned by
Manager()controls a server process which holds Python objects and allows other processes to manipulate them using proxies.
A manager returned by
Manager()will support types
list,
dict,
Namespace,
Lock,
RLock,
Semaphore,
BoundedSemaphore,
Condition,
Event,
Barrier,
Queue,
Valueand
Array. For example,
进程同步
Without using the lock output from the different processes is liable to get all mixed up.
进程池
进程池内部维护一个进程序列,当使用时,则去进程池中获取一个进程,如果进程池序列中没有可供使用的进进程,那么程序就会等待,直到进程池中有可用进程为止。进程池中有两个方法:
apply
apply_async
作业需求:
题目:简单主机批量管理工具需求:
主机分组
主机信息配置文件用configparser解析
可批量执行命令、发送文件,结果实时返回,执行格式如下
batch_run -h h1,h2,h3 -g web_clusters,db_servers -cmd "df -h"
batch_scp -h h1,h2,h3 -g web_clusters,db_servers -action put -local test.py -remote /tmp/
主机用户名密码、端口可以不同
执行远程命令使用paramiko模块
批量命令需使用multiprocessing并发
相关文章推荐
- Python之路,Day9 - 线程、进程、协程和IO多路复用
- python之路 线程、进程、协程、队列、python-memcache、python-redis
- Python学习之路--进程,线程,协程
- Python之路【第七篇】:线程、进程和协程
- Python之路【第七篇】:线程、进程和协程
- 自动化运维Python系列之进程、线程、协程
- python 自动化之路 day 10 协程、异步IO、队列、缓存
- 白话Python 进程,线程,协程
- python 之线程、进程、协程
- python 进程、线程和协程的理解(转)
- Python 中的进程、线程、协程、同步、异步、回调
- python 线程、进程和协程
- Python的进程、线程、协程
- python进程、线程、协程
- Python 中的进程、线程、协程、同步、异步、回调
- 突破python缺陷,实现几种自定义线程池 以及进程、线程、协程的介绍
- Python 中的进程、线程、协程、同步、异步、回调
- Python之路,进程、线程、协程篇
- python进程、线程、协程
- Python 线程、进程和协程