您的位置:首页 > 运维架构

Cannot create encoder for Option of Product type, because Product type is represented as a row

2017-09-14 20:04 471 查看
使用sparksql会遇到下面错误:

Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
java.lang.UnsupportedOperationException: Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:52)
at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:233)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:33)

此问题主要是由于将data[Row]转换成对应的的dataSet类型时,找不到对应的类型转换导致的,需要为对应的类型添加隐式转换,一般添加代码:
implicit val registerKryoEncoder = Encoders.kryo[MyClass]
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签:  spark
相关文章推荐