Storm弹性计算:实时调整Topology并发数
2015-03-25 00:00
309 查看
原创文章,如需转载,请注明出处:http://blog.csdn.net/jmppok/article/details/17143373
Storm计算以topology为单位,topology提交到Storm集群中运行后,通过storm rebalance 命令可对topology进行动态调整。比如增加Topology的worker数,修改Bolt,Spout的并行执行数量 parallelism等,从而实现topology的动态调整,达到弹性计算的目的。(当然调整时要配合监控模块)
可通过storm help rebalance 获取rebalance的帮助信息:
基本上主要有两种用法:
1) storm rebalance topology-name -n new-work-num,
调整指定topology的worknum。
2)storm rebalance topology-name -e component=parallelism
调整指定topology中指定component的并行数量.
Storm计算以topology为单位,topology提交到Storm集群中运行后,通过storm rebalance 命令可对topology进行动态调整。比如增加Topology的worker数,修改Bolt,Spout的并行执行数量 parallelism等,从而实现topology的动态调整,达到弹性计算的目的。(当然调整时要配合监控模块)
可通过storm help rebalance 获取rebalance的帮助信息:
ligh@ubuntu:~/workspace/storm_drpc_test/bin$ storm help rebalance Syntax: [storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*] Sometimes you may wish to spread out where the workers for a topology are running. For example, let's say you have a 10 node cluster running 4 workers per node, and then let's say you add another 10 nodes to the cluster. You may wish to have Storm spread out the workers for the running topology so that each node runs 2 workers. One way to do this is to kill the topology and resubmit it, but Storm provides a "rebalance" command that provides an easier way to do this. Rebalance will first deactivate the topology for the duration of the message timeout (overridable with the -w flag) and then redistribute the workers evenly around the cluster. The topology will then return to its previous state of activation (so a deactivated topology will still be deactivated and an activated topology will go back to being activated). The rebalance command can also be used to change the parallelism of a running topology. Use the -n and -e switches to change the number of workers or number of executors of a component respectively.
基本上主要有两种用法:
1) storm rebalance topology-name -n new-work-num,
调整指定topology的worknum。
2)storm rebalance topology-name -e component=parallelism
调整指定topology中指定component的并行数量.
相关文章推荐
- Storm弹性计算:实时调整Topology并发数
- 实时计算storm流程架构总结
- Storm实时计算系统
- 实时可靠的开源分布式实时计算系统——Storm
- 实时计算之storm
- storm分布式实时计算模式pdf中文扫描完整版
- storm实时计算
- 大数据实时计算工程师/Storm工程师职业学习路线图
- Storm实战_构建大数据实时计算
- 两个例子(来自Storm实战 构建大数据实时计算)
- 从Storm和Spark 学习流式实时分布式计算的设计
- Storm实时计算-新手学习(案例模仿)
- 翻译:Storm Scalable ——Storm弹性计算
- 实时计算平台中的弹性集群资源管理
- Clojure 实战 (5):Storm 实时计算框架
- 学习总结十五:分布式实时计算系统storm简介
- 《storm实战-构建大数据实时计算读书笔记》
- Twitter的开放实时计算系统Storm
- Storm+Kafka实时计算框架搭建