Cannot create encoder for Option of Product type, because Product type is represented as a row
2017-09-14 20:04
471 查看
使用sparksql会遇到下面错误:
Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
java.lang.UnsupportedOperationException: Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:52)
at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:233)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:33)
此问题主要是由于将data[Row]转换成对应的的dataSet类型时,找不到对应的类型转换导致的,需要为对应的类型添加隐式转换,一般添加代码:
implicit val registerKryoEncoder = Encoders.kryo[MyClass]
Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
java.lang.UnsupportedOperationException: Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases. You can wrap your type with Tuple1 if you do want top level null Product objects, e.g. instead of creating `Dataset[Option[MyClass]]`, you can do something like `val ds: Dataset[Tuple1[MyClass]] = Seq(Tuple1(MyClass(...)), Tuple1(null)).toDS`
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:52)
at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:233)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:33)
此问题主要是由于将data[Row]转换成对应的的dataSet类型时,找不到对应的类型转换导致的,需要为对应的类型添加隐式转换,一般添加代码:
implicit val registerKryoEncoder = Encoders.kryo[MyClass]
相关文章推荐
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- [SAP HANA]Cannot create Delivery Unit as content vendor is not defined for this system
- the application is not licensed to create or modify schema for this type of data
- Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry has been closed.
- Cannot execute as the database principal because the principal "guest" does not exist, this type of principal cannot be impersonated, or you do not have permission.
- SQL Server 对等复制配置失败后无法删除重建,报错:Cannot execute as the database principal because the principal "dbo" does not exist, this type of principal cannot be im
- This error is raised because the column 'type' is reserved for storing the class in case of inheritance
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- [转] EF cannot be tracked because another instance of this type with the same key is already being tracked
- Gradle sync failed: Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorSe
- Gradle sync failed: Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorSe
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- ArcGIS Engine中初始化许可常见问题归纳,the application is not licensed to create or modify schema for this type of data
- PowerShell: 如何解决File **.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please see "get- help about_sig" for more de
- office2013死活没法激活:Cannot Activate because this product is incapable of KMS Activation.
- The application is not licensed to modify or create schema for this type of data 解决办法
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry
- Android Studio: Error:Cannot locate factory for objects of type DefaultGradleConnector, as ConnectorServiceRegistry