ACID Transactions with Spark

Qubole offers a data source on top of Spark Data source V1 APIs, which provides Spark support for Hive ACID transactions.

Note

Hive ACID Data Source for Apache Spark is a Beta feature, and is not enabled for all users by default. Create a ticket with Qubole Support to enable this feature on the QDS account.

With this data source, you can work with Hive ACID V2 tables, both Full ACID tables as well as Insert-Only tables. For details about the supported ACID Datasource JAR files for operations on FULL ACID and Insert-only tables , see Functionality Matrix.

You should contact Qubole Support for the latest binary version and jar for Hive ACID.

The following topics provides the information about the set up, configuration, usage, supported capabilities, and considerations.

• Setting up Hive ACID Data Source for Spark
• Reading from Hive ACID Tables
• Using the Supported Capabilities
• Supported Hive ACID Features for Spark
• Considerations for using Hive ACID Data Source for Spark