HBase Error: connection object not serializable
2016-04-05 17:56
453 查看
HBase Error: connection object not serializable
想在spark driver程序中连接HBase数据库,并将数据插入到HBase,但是在spark集群提交运行过程中遇到错误:connection object not serializable详细的错误:
Exception in thread "main" java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable com.sae.model.HbaseHelper Serialization stack: - object not serializable (class: com.sae.model.HbaseHelper, value: com.sae.model.HbaseHelper@27a09971) - field (class: com.sae.demo.KafkaStreamingTest$$anonfun$main$1, name: hbHelper$1, type: class com.sae.model.HbaseHelper) - object (class com.sae.demo.KafkaStreamingTest$$anonfun$main$1, <function1>) - field (class: org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, name: cleanedF$1, type: interface scala.Function1) - object (class org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, <function2>) - writeObject data (class: org.apache.spark.streaming.dstream.DStream) - object (class org.apache.spark.streaming.dstream.ForEachDStream, org.apache.spark.streaming.dstream.ForEachDStream@4bdf) - element of array (index: 0) - array (class [Ljava.lang.Object;, size 16) - field (class: scala.collection.mutable.ArrayBuffer, name: array, type: class [Ljava.lang.Object;) - object (class scala.collection.mutable.ArrayBuffer, ArrayBuffer(org.apache.spark.streaming.dstream.ForEachDStream@4bdf, org.apache.spark.streaming.dstream.ForEachDStream@2b4d4327)) - writeObject data (class: org.apache.spark.streaming.dstream.DStreamCheckpointData) - object (class org.apache.spark.streaming.dstream.DStreamCheckpointData, [ 0 checkpoint files ]) - writeObject data (class: org.apache.spark.streaming.dstream.DStream) - object (class org.apache.spark.streaming.kafka.KafkaInputDStream, org.apache.spark.streaming.kafka.KafkaInputDStream@163042ea) - element of array (index: 0) - array (class [Ljava.lang.Object;, size 16) - field (class: scala.collection.mutable.ArrayBuffer, name: array, type: class [Ljava.lang.Object;) - object (class scala.collection.mutable.ArrayBuffer, ArrayBuffer(org.apache.spark.streaming.kafka.KafkaInputDStream@163042ea)) - writeObject data (class: org.apache.spark.streaming.DStreamGraph) - object (class org.apache.spark.streaming.DStreamGraph, org.apache.spark.streaming.DStreamGraph@2577a95d) - field (class: org.apache.spark.streaming.Checkpoint, name: graph, type: class org.apache.spark.streaming.DStreamGraph) - object (class org.apache.spark.streaming.Checkpoint, org.apache.spark.streaming.Checkpoint@2b4b96a4) at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:557) at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:601) at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:600) at com.sae.demo.KafkaStreamingTest$.main(StreamingDataFromKafka.scala:225) at com.sae.demo.KafkaStreamingTest.main(StreamingDataFromKafka.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
解决办法:
参考官方文档:传送门应该把打开数据库连接的代码放到foreachPartition内部,如:
dstream.foreachRDD { rdd => rdd.foreachPartition { partitionOfRecords => val connection = createNewConnection() partitionOfRecords.foreach(record => connection.send(record)) connection.close() } }
相关文章推荐
- page object模式参考资料
- 手机使用jqueryMobile,提示jquery mobile: Object [object Object] has no method 'jqmData'
- 向下之旅(二十四):kobject与sysfs
- Swift补基础之Selector、条件编译、编译标记、NSObject
- java中的Object...是什么意思详解。
- js post json、object、application/json数据,Nodejs获取json数据
- 谈谈 Object.create ,JS的原型继承
- 【iOS开发技术】对象关联(objc_setAssociatedObject、objc_getAssociatedObject、objc_removeAssociatedObjects)
- Hibernate报错:org.hibernate.ObjectNotFoundException: No row with the given identifier exists 解决办法
- Objective-c 内存管理-MRC 引用计数器
- Objective-c 内存管理-MRC retain&release
- Delphi_07_Delphi_Object_Pascal_基本语法_05_函数参数
- android 子线程通过bundle、handler.sendMessage(message)向主线程传输多个类型数据,包括list<Map<String,Object>>
- Json.net/Newtonsoft 新特性JObject/Linq to Json
- 如何遍历newtonsoft.json的JObject里的JSON数据
- makeObjectsPerformSelector
- @synthesize managedObjectModel = _managedObjectModel;
- object-c中的BOOL类型
- Objective-C中的Hello World
- WaitForMultipleObjects使用详解