python进程间共享数据

python的进程间共享数据可以通过multiprocess中的queue,pipe等现成的消息队列传递

也可以通过multiprocessing.manager创建共享的基本数据类型(如list,RLock等内置类型)

但是以上两种方法都只能传递python内置数据类型

如何传递一个自定义类型数据? 比如结构体

方法是通过BaseManager派生子类(文档17.2.2.7.2)

17.2.2.7.2.

Customized managers To create one’s own manager, one creates a subclass of BaseManager and uses the register() classmethod to register new types or callables with the manager class.

For example:

from multiprocessing.managers import BaseManager

class MathsClass:
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y

class MyManager(BaseManager):
pass

MyManager.register('Maths', MathsClass)

if __name__ == '__main__':
with MyManager() as manager:
maths = manager.Maths()
print(maths.add(4, 3))         # prints 7
print(maths.mul(7, 8))         # prints 56

接下来实现一个简单的自定义类

import multiprocessing as MP
from multiprocessing.managers import BaseManager
import os, time, random, logging

class InfoStruct(BaseManager):
def __init__(self):
super().__init__()
self.lock = MP.Lock()
self.i = 0
self.data = list()

def append(self, x):
self.lock.acquire()
self.i += 1
self.data.append(x)
self.lock.release()
return

def get_i(self):
return self.i

def get_data(self):
return self.data

InfoStruct.register("InfoStruct", InfoStruct)

def writer(info_st, n):
logging.warning("start pid=%d  n=%d" % (os.getpid(), n))
for i in range(n):
info_st.append(random.randint(0, 1000))
logging.warning("finished pid=%d" % os.getpid())
return

if __name__ == "__main__":
manager = InfoStruct()
manager.start()
info_st = manager.InfoStruct()

with MP.Pool() as pool:
for i in range(3):
pool.apply_async(func=writer, args=(info_st, (i+1)*2))
pool.close()
pool.join()

print("value of i: ", info_st.get_i())
print("value of list:\n", info_st.get_data())

data1 = info_st.get_data()
print("data id: ", id(data1))
print("data2 id: ", id(info_st.get_data()))

manager.shutdown()
manager.join()

执行以上代码输出结果

WARNING:root:start pid=2476  n=2
WARNING:root:start pid=7320  n=4
WARNING:root:finished pid=2476
WARNING:root:finished pid=7320
WARNING:root:start pid=2476  n=6
WARNING:root:finished pid=2476
value of i:  12
value of list:
[475, 4, 613, 587, 312, 657, 512, 845, 56, 637, 401, 181]
data id:  2076436644744
data2 id:  2076435444872

有两点与单进程环境中的区别需要着重注意

1.无法使用

info_struct.i

的方法直接调用类的子成员,而需要将其封装成接口如

info_struct.get_i()

2.接口返回的都是值而非引用,

可以看到调用两次

get_data()

返回的list是两个info_struct内部self.data的拷贝,

具有不同的id,此时调用

data.append()

等操作修改的是本地的列表,

而不会影响到共享部分的数据,

这也是为什么在1中无法直接调用类的子成员