您的位置:首页 > 数据库 > MySQL

SPARK SQL - update MySql table using DataFrames and JDBC

2018-01-19 08:52 691 查看
I'm trying to insert and update some data on MySql using Spark SQL DataFrames and JDBC connection.I've succeeded to insert new data using the SaveMode.Append. Is there a way to update the data already existing in MySql Table from Spark SQL?My code to insert is:
myDataFrame.write.mode(SaveMode.Append).jdbc(JDBCurl,mySqlTable,connectionProperties)
If I change to SaveMode.Overwrite it deletes the full table and creates a new one, I'm looking for something like the "ON DUPLICATE KEY UPDATE" available in MySql

It is not possible. As for now (Spark 1.6.0 / 2.2.0 SNAPSHOT) Spark 
DataFrameWriter
 supports only four writing modes:
SaveMode.Overwrite
: overwrite the existing data.
SaveMode.Append
: append the data.
SaveMode.Ignore
: ignore the operation (i.e. no-op).
SaveMode.ErrorIfExists
: default option, throw an exception at runtime.
You can insert manually for example using 
mapPartitions
 (since you want an UPSERT operation should be idempotent and as such easy to implement), write to temporary table and execute upsert manually, or use triggers.In general achieving upsert behavior for batch operations and keeping decent performance is far from trivial. You have to remember that in general case there will be multiple concurrent transactions in place (one per each partition) so you have to ensure that there will no write conflicts (typically by using application specific partitioning) or provide appropriate recovery procedures. In practice it may be better to perform and batch writes to a temporary table and resolve upsert part directly in the database.
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: 
相关文章推荐