Understanding Data Management on QDS

Qubole always caches metadata, command results, and the notebook paragraphs-results. Notebook paragraph-results and commands are stored in Amazon S3 and cached on Qubole serves.

Qubole caches:

  • Metadata for one day that is 24 hours
  • Command results for 7 days
  • Notebook paragraph-results for 30 days

Encrypting Cached Data

The cached data are encrypted and stored in Qubole servers. Create a ticket with Qubole Support to decrypt the encrypted-cached-data.

Encrypting Data on Amazon S3

Qubole supports securing data on Amazon S3 through encryption mechanisms. It supports the server and client side encryption as described below:

Encrypting Data on Ephemeral HDFS

On the QDS UI, encrypting data on Ephemeral HDFS is supported as an option as described in Enable Encryption on Ephemeral HDFS through QDS UI.

To enable encryption on the ephemeral drives through a Cluster REST API, see security_settings.