Python学习之进程和并发

从Python2.4 以后，subprocess模块负责衍生出新的进程，和标准输入，标准输出，标准错误输出交互,并监听返回值。Subprocess模块是用来取代一些老的模块，例如os.system, os.spawn, os.popen和popen2

In [34]: subprocess.call('df -h',shell=True)
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
/dev/xvda1                    485M   55M  405M  12% /boot
/dev/xvdb                      99G   21G   74G  22% /data
Out[34]: 0

还可以引入shell变量

In [105]: subprocess.call('du -sh $HOME',shell=True)
34M	/home/caribbean
Out[105]: 0

使用subprocess.call时可以不显示输出。

In [170]: import subprocess

In [171]: ret=subprocess.call("ping -c 5 10.10.41.10",shell=True,stdout=open('/dev/null','w'),stderr=subprocess.STDOUT)

如果不关心shell命令的输出，而只是想执行shell命令，可以使用subprocess.call，如果想捕获shell命令的输出结果，可以使用subprocess.Popen. subprocess.call还会中断等待响应，subprocess.Popen不会

使用Subprocess的返回值

0 成功
1 通用错误
2 误使用shell内置变量
126 调用的命令无法执行

127 命令无法找到

128 无效的参数退出
130 脚本被Ctrl-C 终端

In [250]: subprocess.call("ls /foo",shell=True)
ls: cannot access /foo: No such file or directory
Out[250]: 2

In [252]: ret=subprocess.call("ls /foo",shell=True)
ls: cannot access /foo: No such file or directory

In [253]: if ret==0:
.....:     print "success"
.....: else:
.....:     print "failure"
.....:
.....:
failure

subprocess.call调用命令的返回值可以用在条件语句中

In [382]: if  platform.system() == "Linux":
.....:     ret=subprocess.call('cp /tmp/a.txt /tmp/b.txt',shell=True)
.....:     if ret==0:
.....:         print "Success,the copy was made on %s %s" %(platform.system(),platform.release())
.....:
.....:
Success,the copy was made on Linux 2.6.32-431.el6.x86_64

使用subprocess.Popen捕获输出结果

In [399]:

In [400]: import subprocess

In [401]: p=subprocess.Popen('df -h',shell=True,stdout=subprocess.PIPE)

In [402]: out=p.stdout.readlines()

In [403]: for line in out:
.....:     print line.strip()
.....:
.....:
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
/dev/xvda1                    485M   55M  405M  12% /boot
/dev/xvdb                      99G   21G   73G  23% /data

In [404]:

Readlines()返回的是一个带换行符的列表

In [413]: print out
--------> print(out)
['Filesystem                    Size  Used Avail Use% Mounted on\n', '/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /\n', 'tmpfs                         1.9G     0  1.9G   0% /dev/shm\n', '/dev/xvda1                    485M   55M  405M  12% /boot\n', '/dev/xvdb                      99G   21G   73G  23% /data\n']

In [414]:

In [440]: for line in out:
.....:     print line
.....:
.....:
Filesystem                    Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /

tmpfs                         1.9G     0  1.9G   0% /dev/shm

/dev/xvda1                    485M   55M  405M  12% /boot

/dev/xvdb                      99G   21G   73G  23% /data

In [441]: for line in out:
.....:     print line.strip()
.....:
.....:
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
/dev/xvda1                    485M   55M  405M  12% /boot
/dev/xvdb                      99G   21G   73G  23% /data

In [442]:

subprocess管道工厂

In [513]: def multi(*args):
.....:     for cmd in args:
.....:         p=subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
.....:         out=p.stdout.read()
.....:         print out
.....:
.....:

In [514]: multi("df -h","ls -lh /tmp","tail -f /var/log/system.log")
tail: cannot open `/var/log/system.log' for reading: No such file or directory
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root  6.7G  1.7G  4.6G  28% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
/dev/xvda1                    485M   55M  405M  12% /boot
/dev/xvdb                      99G   22G   73G  23% /data

total 6.5M
-rw-r--r--.  1 root      root       18K May 28 17:55 a.txt
-rw-r--r--.  1 caribbean operation  52K Jul 23 12:00 baidu.html
-rw-r--r--.  1 caribbean operation  18K Aug  1 16:11 b.txt
-rw-r--r--.  1 daemon    daemon    6.4M Aug  1 14:36 php_errors.log
drwxr-xr-x. 11 root      root      4.0K Jul 30 14:40 pip-build-root
-rw-r--r--.  1 root      root         0 Jul 31 14:39 send_msg_s1.txt
-rw-r--r--.  1 root      root         0 Jul 31 14:39 send_msg_s2.txt
-rw-r--r--.  1 root      root         0 Jul 31 14:39 send_msg_s3.txt
-rw-------.  1 root      root         0 Dec 17  2013 yum.log
-rw-------.  1 root      root       877 Jul 25 10:10 yum_save_tx-2014-07-25-10-10iJ0cju.yumtx
-rw-rw-rw-.  1 mysql     mysql     2.4K Jul 11 13:54 yunbaomysql2.txt
-rw-rw-rw-.  1 mysql     mysql     2.4K Jul 11 13:45 yunbaomysql.txt

In [23]: p=subprocess.Popen("wc -c",shell=True,stdin=subprocess.PIPE)

In [24]: p.communicate("charactersinword")
16
Out[24]: (None, None)

和echo charactersinword | wc -c 等价

In [28]: with open('temp.txt','w') as file:
....:     file.write('charactersinword')
....:
....:

In [29]: file=open('temp.txt')

In [30]: f=file.read()

In [31]: p=subprocess.Popen("wc -c",shell=True,stdin=subprocess.PIPE)

In [32]: p.communicate(f)
16

在shell中等价于
$echo charactersinword > temp.txt
$ wc -c < temp.txt

$ cat /etc/passwd|grep 0:0 |cut -d ':' -f 7
等价于

In [75]: p1=subprocess.Popen("cat /etc/passwd",shell=True,stdout=subprocess.PIPE)

In [76]: p2=subprocess.Popen("grep 0:0",shell=True,stdin=p1.stdout,stdout=subprocess.PIPE)

In [77]: p3=subprocess.Popen("cut -d ':' -f 7",shell=True,stdin=p2.stdout,stdout=subprocess.PIPE)

In [78]: print p3.stdout.read()
-------> print(p3.stdout.read())
/bin/bash

等价于

In [114]: import pwd

In [115]: pwd.getpwnam('root')
Out[115]: pwd.struct_passwd(pw_name='root', pw_passwd='x', pw_uid=0, pw_gid=0, pw_gecos='root', pw_dir='/root', pw_shell='/bin/bash')

In [116]: pwd.getpwnam('root')[-1]
Out[116]: '/bin/bash'

Subprocess模块还可以同时发送输入和接收输出，同时监听错误输出。

在IPython中可以使用ed upper.py这样的方式直接切换到vim编辑代码

#/usr/bin/python

import subprocess

p=subprocess.Popen("tr a-z A-Z",shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
output,error=p.communicate("translatetoupper")
print output

In [10]: ed upper.py
Editing... done. Executing edited code...
TRANSLATETOUPPER

Python中的线程

#/usr/bin/python
import threading
import time
count=1
class KissThread(threading.Thread):
def run(self):
global count
print "Thread # %s: Pretending to do stuff" %count
count +=1
time.sleep(2)
print "done with stuff"
for t in range(5):
KissThread().start()

$ python thread1.py
Thread # 1: Pretending to do stuff
Thread # 2: Pretending to do stuff
Thread # 3: Pretending to do stuff
Thread # 4: Pretending to do stuff
Thread # 5: Pretending to do stuff
done with stuff
done with stuff
done with stuff
done with stuff
done with stuff

common.py

#/usr/bin/python
import subprocess
import time

IP_LIST=['google.com',
'yahoo.com',
'yelp.com',
'amazone.com',
'freebase.com',
'baidu.com',
'taobao.com']

cmd_stub='ping -c 5 %s'

def do_ping(addr):
print time.asctime(),"DOING PING FOR", addr
cmd=cmd_stub % (addr,)
return subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)

nothread.py

#/usr/bin/python

from common import IP_LIST,do_ping
import time

z=[]
for ip in IP_LIST:
p=do_ping(ip)
z.append((p,ip))

for p,ip in z:
print time.asctime(),"WAITING FOR",ip
p.wait()
print time.asctime(),ip,"RETURNED",p.returncode

$ python nothread.py
Mon Aug 11 15:01:58 2014 DOING PING FOR google.com
Mon Aug 11 15:01:58 2014 DOING PING FOR yahoo.com
Mon Aug 11 15:01:58 2014 DOING PING FOR yelp.com
Mon Aug 11 15:01:58 2014 DOING PING FOR amazone.com
Mon Aug 11 15:01:58 2014 DOING PING FOR freebase.com
Mon Aug 11 15:01:58 2014 DOING PING FOR baidu.com
Mon Aug 11 15:01:58 2014 DOING PING FOR taobao.com
Mon Aug 11 15:01:58 2014 WAITING FOR google.com
Mon Aug 11 15:02:03 2014 google.com RETURNED 0
Mon Aug 11 15:02:03 2014 WAITING FOR yahoo.com
Mon Aug 11 15:02:03 2014 yahoo.com RETURNED 0
Mon Aug 11 15:02:03 2014 WAITING FOR yelp.com
Mon Aug 11 15:02:03 2014 yelp.com RETURNED 0
Mon Aug 11 15:02:03 2014 WAITING FOR amazone.com
Mon Aug 11 15:02:12 2014 amazone.com RETURNED 1
Mon Aug 11 15:02:12 2014 WAITING FOR freebase.com
Mon Aug 11 15:02:12 2014 freebase.com RETURNED 0
Mon Aug 11 15:02:12 2014 WAITING FOR baidu.com
Mon Aug 11 15:02:12 2014 baidu.com RETURNED 0
Mon Aug 11 15:02:12 2014 WAITING FOR taobao.com
Mon Aug 11 15:02:12 2014 taobao.com RETURNED 0

ping_thread_basic.py

#/usr/bin/python
from threading import Thread
import subprocess
from Queue import Queue

num_threads=3
queue=Queue()
ips=["10.10.41.254","10.10.41.20","10.10.41.42","10.10.41.15"]

def pinger(i,q):
while True:
ip=q.get()
print "Thread %s: Pinging %s" %(i,ip)
ret=subprocess.call("ping -c 1 %s" %ip,shell=True,stdout=open('/dev/null','w'),stderr=subprocess.STDOUT)
if ret==0:
print "%s: is alive" %ip
else:
print "%s: did not respond" %ip
q.task_done()

for i in range(num_threads):
worker=Thread(target=pinger,args=(i,queue))
worker.setDaemon(True)
worker.start()

for ip in ips:
queue.put(ip)

print "Main Thread Waiting"

queue.join()

print "Done"

$ python2.6 ping_thread_basic.py
Main Thread Waiting
Thread 0: Pinging 10.10.41.254
Thread 2: Pinging 10.10.41.20
Thread 1: Pinging 10.10.41.42
10.10.41.254: is alive
Thread 0: Pinging 10.10.41.15
10.10.41.42: is alive
10.10.41.20: is alive
10.10.41.15: is alive
Done

本文出自 “Linux SA John” 博客，请务必保留此出处http://john88wang.blog.51cto.com/2165294/1533651