Understanding Data Encryption on QDS

Qubole always caches metadata, command results, and the notebook paragraphs-results. This is independent of data on AWS S3. Notebook paragraph-results and commands are stored in Amazon S3 and cached on Qubole servers.

Qubole caches:

  • Metadata for one day that is 24 hours
  • Command results for 7 days
  • Notebook paragraph-results for 30 days

QDS provides encryption mechanisms to protect the data.

Encrypting Cached Data

Create a ticket with Qubole Support to enable encryption of results while fetching them from the object storage, though it might slow down the data retrieving process as QDS would not be caching the results onto cache.

There is no option to disable Metastore caching and read-only Notebooks are always cached with encryption on.

Encrypting Data on Amazon S3

Qubole supports protecting data on Amazon S3 through encryption mechanisms. It supports the server-side and client-side encryption as described in Enabling Data Encryption in QDS (AWS).

Encrypting Ephemeral Data on QDS Clusters

On the QDS clusters, you can encrypt data on Ephemeral HDFS as described in Enabling Encryption on Ephemeral Data in QDS Clusters.

To enable encryption on the ephemeral drives through a Cluster REST API, see security_settings.