spark中updateStateByKey引发StackOverflowError的解决

spark中updateStateByKey引发StackOverflowError的解决

问题描述
写的spark程序, 运行几个小时候总是会出现 StackOverflowError.
程序使用 spark-1.1 运行.
代码的逻辑大概是:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

原因

逐行注释代码, 发现error由updateStateByKey引发.
在每个 updateStateByKey 之后用 foreachRDD 输出 toDebugString, 发现是因为 rdd 的 dependency chain 太长导致的StackOverflowError.

这就比较奇怪, 因为 updateStateByKey 默认会 checkpoint, 而 checkpoint 会切断 dependency chain.
可参考: spark streaming checkpointing
默认的checkpoint机制和interval
spark在代码中有 updateStateByKey 和 reduceByKeyAndWindow设置有 inverse function 的时候, 会自动 checkpoint .
checkpoint 的 interval 必须是 duration 的整数倍, 默认值为满足下面条件的最小值:

interval >= max(duration, Duration(10000))
interval % duration = 0

其中 duration 是 streaming 的 batch duration.
如 spark streaming checkpointing 所讲

checkpointing too infrequently causes the lineage and task sizes to grow which may have detrimental effects

如果将 batch duration 设置为 1 ms, checkpoint 的 interval 则会默认设为 10000 ms, 此时 batch 相对于 interval 来说就是 too infrequently. 这种设置下updateStateByKey很快就会引发 StackOverflowError.
另外需要注意的是, 每个 job 只会 checkpoint 一次. 可看源码:

/**
* Run a function on a given set of partitions in an RDD and pass the results to the given
* handler function. This is the main entry point for all actions in Spark. The allowLocal
* flag specifies whether the scheduler can run the computation on the driver rather than
* shipping it out to the cluster, for short actions like first().
*/
def runJob[T, U: ClassTag](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int],
allowLocal: Boolean,
resultHandler: (Int, U) => Unit) {
if (dagScheduler == null) {
throw new SparkException("SparkContext has been shutdown")
}
val callSite = getCallSite
val cleanedFunc = clean(func)
logInfo("Starting job: " + callSite.shortForm)
dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, allowLocal,
resultHandler, localProperties.get)
progressBar.foreach(_.finishAll())
rdd.doCheckpoint()
}

也就是说, 下面的代码只会 checkpoint 一次. (spark streaming中, 一个 output operation 有一个 job.)

streamC = streamB.updateStateByKey()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

streamC一直没有checkpoint, 从而导致 dependency chain 过长产生 StackOverflowError.
foreachRDD 和 print 的不同
我将代码改为:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamC.foreachRDD()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

发现虽然 streamC 的 checkpoint 已经生效, 但是它的 dependency chain 依旧没有截断. 个人陷入困境.
leader Ning 使用 print 而非 foreachRDD, 发现 dependency chain 有被截断.
foreachRDD 源码:

/**
* Apply a function to each RDD in this DStream. This is an output operator, so
* 'this' DStream will be registered as an output stream and therefore materialized.
*/
def foreachRDD(foreachFunc: (RDD[T], Time) => Unit) {
// because the DStream is reachable from the outer object here, and because
// DStreams can't be serialized with closures, we can't proactively check
// it for serializability and so we pass the optional false to SparkContext.clean
new ForEachDStream(this, context.sparkContext.clean(foreachFunc, false)).register()
}

print 源码:

/**
* Print the first ten elements of each RDD generated in this DStream. This is an output
* operator, so this DStream will be registered as an output stream and there materialized.
*/
def print() {
def foreachFunc = (rdd: RDD[T], time: Time) => {
val first11 = rdd.take(11)
println ("-------------------------------------------")
println ("Time: " + time)
println ("-------------------------------------------")
first11.take(10).foreach(println)
if (first11.size > 10) println("...")
println()
}
new ForEachDStream(this, context.sparkContext.clean(foreachFunc)).register()
}

发现两者都是调用了 ForEachDStream, 区别在于 foreachRDD 设置了 context.sparkContext.clean 的 checkSerializable 为 false.
sparkContext.clean 的源码为:

/**
* Clean a closure to make it ready to serialized and send to tasks
* (removes unreferenced variables in $outer's, updates REPL variables)
* If <tt>checkSerializable</tt> is set, <tt>clean</tt> will also proactively
* check to see if <tt>f</tt> is serializable and throw a <tt>SparkException</tt>
* if not.
*
* @param f the closure to clean
* @param checkSerializable whether or not to immediately check <tt>f</tt> for serializability
* @throws <tt>SparkException<tt> if <tt>checkSerializable</tt> is set but <tt>f</tt> is not
*   serializable
*/
private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean = true): F = {
ClosureCleaner.clean(f, checkSerializable)
f
}

至于 checkSerializable 为 false 导致 rdd 的 dependency chain 不截断的原因, 貌似和 一个KCore算法引发的StackOverflow奇案 相同,
都是 $outer 导致, 这里没有深究.
fix的方法
因此, 将代码调整为:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamC.print()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

虽然 print 会将数据传回 driver, 会有一点影响性能, 但是可以接受.