dispy,asyncoro实现的分布式并行计算框架

dispy,asyncoro实现的分布式并行计算框架

分类： 技术文章2012-09-06
11:18 27031人阅读 评论(1) 收藏 举报

dispy：asyncoro实现的分布式并行计算框架。一个对asyncoro很有说明性的案例。

框架也是非常精简，只有4个组件

dispy.py (client) provides two ways of creating "clusters": JobCluster when only one instance of dispy may run and SharedJobCluster when multiple instances may run (in separate processes). If JobCluster is used, the scheduler contained within dispy.py will
distribute jobs on the server nodes; if SharedJobCluster is used, a separate scheduler (dispyscheduler) must be running.

dispynode.py executes jobs on behalf of dispy. dispynode must be running on each of the (server) nodes that form the cluster.

dispyscheduler.py is needed only when SharedJobCluster is used; this provides a scheduler that can be shared by multiple dispy users.

dispynetrelay.py is needed when nodes are located across different networks; this relays information about nodes on a network to the scheduler. If all the nodes are on same network, there is no need for dispynetrelay - the scheduler and nodes automatically
discover each other.

一般情况下，使用dispy和dispynode就已经足够解决问题了

服务端：

dispynode是服务端组件，它不需要写代码，只是使用参数运行为一个守护进程就OK了，比如：

dispynode.py -c 2 -i 192.168.0.10 -p 51348 -s secret

这个实例会使用2个cpu核心，绑定192.168.0.10:51348地址提供服务，secret是消息加密的共享密钥，更多参数参见http://dispy.sourceforge.net/dispynode.html

客户端：

简单例子：

#!/usr/bin/env python
def compute(n):
import time, socket
time.sleep(n)
host = socket.gethostname()
return (host, n)

if __name__ == '__main__':
import dispy, random
cluster = dispy.JobCluster(compute,nodes=['192.168.0.10', '192.168.3.11'])
jobs = []
for n in range(20):
job = cluster.submit(random.randint(5,20))
job.id = n
jobs.append(job)
# cluster.wait()
for job in jobs:
host, n = job()
print '%s executed job %s at %s with %s' % (host, job.id, job.start_time, n)
# other fields of 'job' that may be useful:
# print job.stdout, job.stderr, job.exception, job.ip_addr, job.start_time, job.end_time
cluster.stats()

JobCluster也可以使用回调来处理结果：

dispy.JobCluster(compute,nodes=['192.168.0.10', '192.168.3.11']，callback=callback)

除了python函数，也可以是调用服务端的程序，比如：

cluster = dispy.JobCluster('/some/program', nodes=['192.168.0.10'])

也可以不写任何代码，而作为一个命令工具来使用：

dispy.py -f /some/file1 -f file2 -a "arg11 arg12" -a "arg21 arg22" -a "arg3" /some/program

详细文档参见 http://dispy.sourceforge.net/index.html

如果嫌celery稍重的话，可以试试dispy