Understanding the Qubole-Snowflake Integration Workflows

The Qubole-Snowflake integration provides the following workflows that you can execute by using QDS.

Advanced Data Preparation on Snowflake Data only

This workflow involves the following tasks:

  1. Adding the Snowflake data warehouse as a data store by using QDS.
  2. Running the Qubole Dataframe API for Apache Spark to read data from the Snowflake data store from any of the QDS interfaces, such as, QDS Analyze page, notebooks, or API.
  3. Running a Spark command on QDS to prepare data in Apache Spark for data cleansing, blending, and transformation.
  4. Running the Qubole Dataframe API for Apache Spark from QDS to write results back to the Snowflake data store.

The following diagram is the graphical representation of the workflow.

../../../_images/snowflake_usecase1.png

Advanced Data Preparation on Snowflake and Other Data Sources (AWS)

This workflow involves the following tasks:

  1. Setting up Hive metastore with existing data in Amazon S3.
  2. Adding the Snowflake data warehouse as a data store by using QDS.
  3. Running a Spark command to prepare data in S3 for analytics using Apache Spark. Perform data cleansing, blending and transformation.
  4. Running the Qubole Dataframe API for Apache Spark from QDS to write results to the Snowflake data store.

The following diagram is the graphical representation of the workflow.

../../../_images/snowflake_usecase2.png

AI and Machine Learning (ML) on Snowflake Data

This workflow involves the following tasks:

  1. Adding the Snowflake data warehouse as a data store by using QDS.
  2. Running the Qubole Dataframe API for Apache Spark to read data from the Snowflake data store from any of the QDS interfaces, such as, QDS Analyze page, notebooks, or API.
  3. Running a Spark command on QDS for Machine Learning in Apache Spark for feature engineering, model building, training, and validation.

The following diagram is the graphical representation of the workflow.

../../../_images/snowflake_usecase3.png