关于Storm的一些疑问解答

Q1： 出现下面的问题怎么解决？

2011-12-26 11:44:21 worker [ERROR] Error on initialization of server mk-worker

java.lang.UnsatisfiedLinkError: /usr/local/lib/libjzmq.so.0.0.0: libzmq.so.1: cannot open shared object file: No such file or directory

at java.lang.ClassLoader$NativeLibrary.load(Native Method)

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1767)

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1692)

at java.lang.Runtime.loadLibrary0(Runtime.java:840)

at java.lang.System.loadLibrary(System.java:1047)

at org.zeromq.ZMQ.<clinit>(ZMQ.java:34)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:186)

at zilch.mq__init.load(Unknown Source)

at zilch.mq__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at clojure.lang.RT.loadClassForName(RT.java:1578)

at clojure.lang.RT.load(RT.java:399)

at clojure.lang.RT.load(RT.java:381)

at clojure.core$load$fn__4511.invoke(core.clj:4905)

at clojure.core$load.doInvoke(core.clj:4904)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at clojure.core$load_one.invoke(core.clj:4729)

at clojure.core$load_lib.doInvoke(core.clj:4766)

at clojure.lang.RestFn.applyTo(RestFn.java:143)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$load_libs.doInvoke(core.clj:4800)

at clojure.lang.RestFn.applyTo(RestFn.java:138)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$require.doInvoke(core.clj:4869)

at clojure.lang.RestFn.invoke(RestFn.java:422)

at backtype.storm.messaging.zmq$loading__4410__auto__.invoke(zmq.clj:1)

at backtype.storm.messaging.zmq__init.load(Unknown Source)

at backtype.storm.messaging.zmq__init.<clinit>(Unknown Source)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at clojure.lang.RT.loadClassForName(RT.java:1578)

at clojure.lang.RT.load(RT.java:399)

at clojure.lang.RT.load(RT.java:381)

at clojure.core$load$fn__4511.invoke(core.clj:4905)

at clojure.core$load.doInvoke(core.clj:4904)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at clojure.core$load_one.invoke(core.clj:4729)

at clojure.core$load_lib.doInvoke(core.clj:4766)

at clojure.lang.RestFn.applyTo(RestFn.java:143)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$load_libs.doInvoke(core.clj:4800)

at clojure.lang.RestFn.applyTo(RestFn.java:138)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$require.doInvoke(core.clj:4869)

at clojure.lang.RestFn.invoke(RestFn.java:409)

at backtype.storm.messaging.loader$mk_zmq_context.doInvoke(loader.clj:8)

at clojure.lang.RestFn.invoke(RestFn.java:437)

at backtype.storm.daemon.worker$fn__3102$exec_fn__858__auto____3103.invoke(worker.clj:111)

at clojure.lang.AFn.applyToHelper(AFn.java:187)

at clojure.lang.AFn.applyTo(AFn.java:151)

at clojure.core$apply.invoke(core.clj:540)

at backtype.storm.daemon.worker$fn__3102$mk_worker__3244.doInvoke(worker.clj:80)

at clojure.lang.RestFn.invoke(RestFn.java:513)

at backtype.storm.daemon.worker$_main.invoke(worker.clj:257)

at clojure.lang.AFn.applyToHelper(AFn.java:174)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.daemon.worker.main(Unknown Source)

2011-12-26 11:44:21 util [INFO] Halting process: ("Error on initialization")

A1：sudo vi /etc/ld.so.conf 在该文件添加一行“/usr/local/lib”， 保存以后，执行“sudo /sbin/ldconfig”就可以fixed这个问题。

Q2：Storm选择使用kestrel而不是zeromq的原因？

A2：ZeroMQ is basically a socket library. If you want to pass data between

topologies, you have two things you need to solve:

1. Discovering where to send the tuples (since Storm can launch the

spout tasks anywhere on the cluster, and their locations can change

over time)

2. Guaranteeing message processing on the second topology

If you use a Kestrel queue (or queues) as the intermediate broker,

that solves #1 because you're using a fixed set of machines that both

topologies know about. It solves #2 because Kestrel persists the

messages and is capable of replaying the messages.

You want to make sure that your spout source can support the out of

order acking that Storm requires for guaranteed message processing. We

use Kestrel because it has this property and is the simplest.

Q3：Storm如果避免发生拥塞？

A3：暂时Storm没有好的方法来控制这个，但是对于保证数据可靠性的topology而言，可以通过TOPOLOGY_MAX_SPOUT_PENDING来对spout进行一些控制；

另外只能靠增加处理速度慢的bolt的并行度或者是减少spout的并行度来调节数据流拥塞问题了。

Q4：关于Storm UI上spout complete latency的一些疑问

. complete latency是如何计算的?

. 对于一个topology而言，理想的complete latency是多少？

. 在实现topology的时候，如果能够保证尽可能小的latency?

A4：

1. The timer is started when the tuple is emitted from the spout and it is

stopped when the tuple is acked. So it measures the time for the entire

tuple tree to be completed.

2. There's no "ideal" time. Every topology has different characteristics.

For example, in our stream processing topologies we care more about

throughput than latency and make use of batching techniques that increase

throughput at the cost of worsened latency. In our distributed RPC

topologies however, we optimize for latency.

3. The completion time will be affected by: a) the processing latency of

the bolts, which you can improve by doing optimization b) the amount of

congestion in the topology, which you can reduce by increasing parallelism.

So if you add up the processing latencies of the bolts and they don't come

close to the complete latency, you should try adding more parallelism.

Q5：为保证数据可靠性，spout acker需要知道哪些东西？

A5：

First, you need to tell Storm whenever you're creating a new link in the tree of tuples.

Second, you need to tell Storm when you have finished processing an individual tuple.

如果你的bolt实现的是IRichBolt接口，这两件事情需要你自己亲自去做；但是如果你继承的是BasicBolt, 则它会偷偷

的帮助你去做这两件事情，所以当你需要unanchored的时候，则不要使用BasicBolt

Tuples emitted to BasicOutputCollector are automatically anchored to the input tuple, and

the input tuple is acked for you automatically when the execute method completes.

Q6：针对Storm DRPC的几个问题？

1) If the drpc server configure as follows, then i need run "./bin/storm drpc" on all of these servers?

> drpc.servers:

> - "10.13.41.104"

> - "10.13.41.64"

> - "10.13.41.98"

2）I need use storm jar summit the topology to storm first, then write a application using DRPCClient to call "execute" ?

3）If i can give more than one hosts to DRPCClient?

DRPCClient client = new DRPCClient("drpc-host", 3772);

A6：

1）Yes, you do.

2) Yes.

3）No, you can't. DRPCClient can definitely be improved (or a higher level one

can be written on top of it).

Q7：关于Storm UI(perftest)的几个问题？ （基于http://groups.google.com/group/storm-user/browse_thread/thread/822be41e58fed298）



1) 为什么topology只定义了一个bolt，但是在storm UI的Bolts下面还有一个Id为-1的，代表什么意思？

2）PrinterBolt并没有调用emit，为什么还会有Emitted呢？

3）Spouts的Emitted为1985940, 而bolt Id 1的Emitted是992980，是因为bolt没有追上spout的emit速度吗？

A7：

1） That's the "acker bolt" which takes care of tracking tuple trees. It's an

integral part of guaranteeing message processing, and is described in the

wiki here:

https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing

In 0.6.0 that bolt was renamed to "__acker"

2）It's emitting ack tuples on its "ack" stream that go to the acker bolt.

3）Notice that the emitted value of spout 1 is just about double the emitted

value of bolt 2. This is because for every tuple spout 1 emits, it sends a tuple to bolt 2

as well as a tuple to the acker bolt (to track the tuple tree). Bolt 2 only

acks tuples so it sends a single ack tuple per received tuple.

Storm Google Group是个提问题的好地方，Storm作者很nice，多上那边逛逛吧！