[Storm] java.io.FileNotFoundException: File '../stormconf.ser' does not exist

问题背景

最近发现刚搭起的Storm集群，没过多久，Supervisor 便悄然死去了一大半。查看死去Supervisor的log，发现java.io.FileNotFoundException: File '../stormconf.ser' does not exist异常。网上给出的答案大多是

将 { storm.local.dir } 目录下的文件清空，重启就好了。

但这是指标不治本，即时重启可以跑起来，可是为什么会出现这个问题，依然不知道。

然后才发现线STORM-130解决了这个问题。该问题的重现场景：

1) Run a storm cluster with atleast 2 supervisors with 4 slots each
2) Deploy a topology that uses 4 workers, topology will be distributed with each supervisor having two workers each
3) kill one of the supervisor lets say supervisor1
4) wait till topology re-balances to occupy 4 workers on supervisor2
5) now bring up supervisor1, It goes through the cycle of cleaning up old topology code
6) nimbus re-balances topology which triggers supervisor.sync-process method
7) sync-process tries to launch a worker for the topology whose code data is delete when the supervisor started causing it throw up following exception

问题原因

上面场景分析提到的 sync-process是supervisor运行的一个函数。Supervisor会在后台运行这两个函数：

synchronize-supervisor

: This is called whenever assignments in Zookeeper change and also every 10 seconds.


Downloads code from Nimbus for topologies assigned to this machine for which it doesn't have the code yet.


Writes into local filesystem what this node is supposed to be running. It writes a map from port -> LocalAssignment. LocalAssignment contains a topology id as well as the list of task ids for that worker.

sync-processes

: Reads from the LFS what

synchronize-supervisor

wrote and compares that to what's actually running on the machine. It then starts/stops worker processes as necessary to synchronize.

从描述中可以看出，synchronized-supervisor 和 sync-process 两个函数是通过 LFS 进行同步。The key reason is "synchronize-supervisor" which responsible for download file and remove file thread and "sync-processes" which responsible for start worker process thread is Asynchronous.

in synchronize-supervisor read assigment information from zk, supervisor download necessary file from nimbus and write local state. In aother thread sync-processes funciton read local state to launch workor process, when the worker process has not start ,synchronize-supervisor function is called again topology's assignment information has changed (cased by rebalance,or worker time out etc) worker assignment to this supervisor has move to another supervisor, synchronize-supervisor remove the unnecessary file (jar file and ser file etc.) , after this, worker launched by " sync-processes" ,ser file was not exsit , this issue occur.

可能解决办法

换一个storm

调整参数

Change "synchronize-supervisor" thread loop time to a longger than 10(default time) sec, such as 30 sec。

supervisor.worker.timeout.secs: 30 -> 5

References:
https://issues.apache.org/jira/browse/STORM-130 http://storm.apache.org/documentation/Lifecycle-of-a-topology.html