python map/reduce framework
2010-08-20 16:49
381 查看
http://code.google.com/p/octopy/
Inspired by Google's MapReduce and Starfish for Ruby, octo.py is a fast-n-easy MapReduce implementation for Python.
Octo.py doesn't aim to meet all your distributed computing needs, but its simple approach is amendable to a large proportion of parallelizable tasks. If your code has a for-loop, there's a good chance that you can make it distributed with just a few small changes. If you're already using Python's map() and reduce() functions, the changes needed are trivial!
It is not an exact clone of the Big-G's MapReduce, but I'm guessing that you aren't operating a Google-like cluster with a distributed Google File System and can't use a MapReduce clone. Instead, the scope of the project is more akin to Starfish, running on an ad-hoc cluster of computers. The data semantics bears closer resemblance to MapReduce though, except the part about the ordering of intermediate results.
For examples, look at UsageExamples. For detailed usage instructions, take a look at UsersGuide. And if you're interested in modifying the source, take a look at DevelopersGuide.
gearman
disco
http://trac.nchc.org.tw/grid/wiki/jazz/09-04-14#MapReduce
Inspired by Google's MapReduce and Starfish for Ruby, octo.py is a fast-n-easy MapReduce implementation for Python.
Octo.py doesn't aim to meet all your distributed computing needs, but its simple approach is amendable to a large proportion of parallelizable tasks. If your code has a for-loop, there's a good chance that you can make it distributed with just a few small changes. If you're already using Python's map() and reduce() functions, the changes needed are trivial!
It is not an exact clone of the Big-G's MapReduce, but I'm guessing that you aren't operating a Google-like cluster with a distributed Google File System and can't use a MapReduce clone. Instead, the scope of the project is more akin to Starfish, running on an ad-hoc cluster of computers. The data semantics bears closer resemblance to MapReduce though, except the part about the ordering of intermediate results.
For examples, look at UsageExamples. For detailed usage instructions, take a look at UsersGuide. And if you're interested in modifying the source, take a look at DevelopersGuide.
gearman
disco
http://trac.nchc.org.tw/grid/wiki/jazz/09-04-14#MapReduce
相关文章推荐
- python filter, map, 和reduce
- python函数lambda(), filter(), map(), reduce()
- Python特殊语法:filter、map、reduce、lambda(转)
- python filter/map/reduce
- 3.python中map,filter,reduce以及内部实现原理剖析
- Python常用内置函数整理(lambda,reduce,zip,filter,map)
- Python中的map、reduce和filter浅析
- python中map和reduce的应用
- python学习——map/reduce
- Python特殊语法这之:filter、map、reduce、lambda(转自“玩玩python”)
- 使用python实现内置map,filter,reduce,zip函数
- python高阶函数之lambda,map,reduce,filter
- python map filter reduce
- Map和Reduce在Hadoop与Python中有何异同?
- python内置函数lambda、filter、map、reduce
- Python中的map()和reduce()以及一些栗子
- Python自学笔记之函数式编程2——map/reduce
- Python学习4:生成器、lamda表达式及map、reduce、filter函数
- Python高阶函数(Map、Reduce、Filter)
- python Lambda, filter, reduce and map