spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;
2017-11-28 12:07
801 查看
spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;
起因编写了一个处理两列是否相等的UDF,这两列的数据结构是一样的,但是结构比较复杂,如下:
|-- list: array (nullable = true) | |-- element: map (containsNull = true) | | |-- key: string | | |-- value: array (valueContainsNull = true) | | | |-- element: struct (containsNull = true) | | | | |-- Date: integer (nullable = true) | | | | |-- Name: string (nullable = true) |-- list2: array (nullable = true) | |-- element: map (containsNull = true) | | |-- key: string | | |-- value: array (valueContainsNull = true) | | | |-- element: struct (containsNull = true) | | | | |-- Date: integer (nullable = true) | | | | |-- Name: string (nullable = true)
也就是说Array里嵌着Map,Map里还嵌着一个Array,只能依次去比较,编写的UDF如下:
case class AppList(Date: Int, versionCode: Int, Name:String) def isMapEqual(map1: Map[String, Array[AppList]], map2:Map[String, Array[AppList]]): Boolean = { try{ if (map1.size != map2.size){ return false } else{ for ( x <- map1.keys){ if (map1(x) != map2(x)){ return false } } return true } } catch { case e: Exception => false } } def isListEqual(list1: Array[Map[String, Array[AppList]]], list2:Seq[Map[String, Seq[AppList]]]): Boolean = { try { if (list1.length != list2.length){ return false } else if (list1.length == 0 || list2.length == 0){ return false } else { return isMapEqual(list1(0), list2(0)) } } catch { case e: Exception => false } } val isColumnEqual = udf((list1: Array[Map[String, Array[AppList]]], list2:Array[Map[String, Array[AppList]]]) => { isListEqual(list1, list2) })
然后我就贴到spark-shell里去执行了下面语句:
val dat = df.withColumn("equal", isColumnEqual($"list1", $"list2")) dat.show()
此时就出现了如下错误:
Caused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$1: (array<map<string,array<struct<Date:int,Name:string>>>>, array<map<string,array<struct<Date:int,Name:string>>>>) => boolean) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map; at $anonfun$1.apply(<console>:42) ... 16 more
解决办法
所谓的解决办法,自然是去谷歌了…
在这里看到,说把Array改成Seq就好了,囧,尝试了一下,果然就好了
原因
这里说:
So it looks like the ArrayType on Dataframe "idDF" is really a WrappedArray and not an Array - So the function call to "filterMapKeysWithSet" failed as it expected an Array but got a WrappedArray/ Seq instead (which doesn't implicitly convert to Array in Scala 2.8 and above).
意思是,此Array非Scala中的原生Array,而是封装了一下的Array(有错的一定指出来,我都没写过Scala,慌
参考
https://stackoverflow.com/questions/40199507/scala-collection-mutable-wrappedarrayofref-cannot-be-cast-to-integer
https://stackoverflow.com/questions/40764957/spark-java-lang-classcastexception-scala-collection-mutable-wrappedarrayofref
相关文章推荐
- spark1.5 scala.collection.mutable.WrappedArray$ofRef cannot be cast to ...解决办法
- spark1.5 scala.collection.mutable.WrappedArray$ofRef cannot be cast to ...解决办法
- spark1.5 scala.collection.mutable.WrappedArray$ofRef cannot be cast to ...解决办法
- spark1.5 scala.collection.mutable.WrappedArray$ofRef cannot be cast to ...解决办法
- struts2下载文件时出错提示:java.lang.ClassCastException: java.io.ByteArrayInputStream cannot be cast to java.l
- Gson解析报错 java.lang.ClassCastException: com.google.gson.internal.StringMap cannot be cast to XXX
- struts2下载文件时出现:java.lang.ClassCastException: java.io.ByteArrayInputStream cannot be cast to 错误
- Caused by: java.lang.ClassCastException: org.hibernate.type.StringType cannot be cast to org.hiberna
- struts2 下载问题java.lang.ClassCastException: java.io.ByteArrayInputStream cannot be cast to
- Caused by: java.lang.ClassCastException: com.tencent.mm.sdk.modelmsg.SendMessageToWX$Resp cannot be
- Caused by: java.lang.ClassCastException: org.springframework.web.SpringServletContainerInitializer cannot be cast to javax.servlet.ServletContainerInitializer错误解决办法
- Caused by: java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.reflect.Paramet
- Caused by: java.lang.ClassCastException: android.widget.TextView cannot be cast to android.widget.Bu
- Caused by: java.lang.ClassCastException: org.hibernate.type.StringType cannot be cast to org.hiberna
- struts2下载文件时出现:java.lang.ClassCastException: java.io.ByteArrayInputStream cannot be cast to 错误
- java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.util.Map
- java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
- SparkR读取CSV格式文件错误java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.u
- java.lang.ClassCastException:org.apache.catalina.util.DefaultAnnotationProcessor cannot be cast to org.apache.AnnotationProcesso
- 十、java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to xxx.action.entity.Category