Overview

The Qubole Data Service (QDS) is accessible via REST APIs.

To write and test applications that use these APIs, you can use any HTTP client in any programming language to interact with Qubole Data Service. The detailed syntax of the calls is described in subsequent sections.

You can also use the QDS Python SDK or the QDS Java SDK to interact with the Qubole Data Service. The Python SDK also provides an easy-to-use command-line interface.

Note

Qubole now supports only HTTPS. All HTTP requests are now redirected to HTTPS. This is aimed at better security for Qubole users.

Access URL

All URLs referenced in the documentation have the following base:

https://api.qubole.com/api/${V}/

where ${V} refers to the version of the API. Valid values of ${V} are currently v1.2, v1.3, and v2 for APIs for QDS-on-AWS. The value of ${V} is only v2 for APIs for QDS-on-Azure, QDS-on-GCP, and QDS-on-OCI.

Qubole supports other access URLs or endpoints that you can use to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.

Here are how features are supported on different API versions and accordingly described in the corresponding topics.

API Versions Supported on QDS-on-AWS

The APIs on QDS-on-AWS are supported as follows:

• Only Cluster API v1.3 is supported on v1.3. The API syntax for v1.2 Cluster API is entirely different and the latest features are not supported on it. The documentation is based on v1.3.

• All other APIs are only supported on v1.2. The documentation only describes the syntax and sample API calls with respect to v1.2.

API Versions Supported on QDS-on-Azure and QDS-on-OCI

APIs for Qubole on Microsoft Azure and APIs for Qubole on Oracle Cloud Infrastructure are only supported on API version 2 that is v2.

cURL is a useful tool for testing out Qubole REST API calls from the command line.

The Qubole API is served over HTTPS and Qubole redirects all HTTP access to HTTPS. Qubole now supports only HTTPS access, which is aimed at better security for Qubole users. When using HTTPS, ensure that the certificates on the client machine are up-to-date.

Authentication

API calls must be authenticated with a Qubole API Token.

Navigate to Control Panel in the QDS UI and click the My Accounts tab on the left pane. Click Show for the account and copy the API token that is displayed.

Set the value of this API token to the AUTH_TOKEN environment variable when running the API examples via curl.

Note

Qubole supports access URLs or endpoints that you can use to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.

API Types

The Qubole REST APIs are broadly divided into the following categories:

• APIs for Qubole on Amazon Web Service:

  • Account API

  • Apps API

  • Automatic Statistics Collection API

  • Cluster API

  • Command API

  • Command Template API

  • Custom Metastore API

  • DbTap API

  • Folder API

  • Groups API

  • Hive Metadata API

  • Notebook API

  • Object Policy API

  • Reports API

  • Roles API

  • Scheduler API

  • Sensor API

  • Users API

• APIs for Qubole on Microsoft Azure

• APIs for Qubole on Oracle Cloud Infrastructure

Account API

These APIs let you to create a new account, edit and delete an existing account, view users, and set Hive bootstrap. See Account API v1.2 for more information.

Apps API

These APIs allow you to create, list, and delete a Spark Job Server App. For more information, see Apps API.

Automatic Statistics Collection API

These APIs allow you to view and kill a command that collects automatic statistics for Hive tables. For more information, see Automatic Statistics Collection API.

Cluster API

These APIs allow you to create a new cluster, edit, restart, and delete an existing clusters. These APIs also allow you to view the list of clusters in a Qubole account, and add and remove nodes in some clusters. See Cluster API v1.3 for more information.

Command API

The Command APIs let you submit queries and commands, check the status of commands, retrieve results and logs, or cancel commands. See Command API for more information. The Qubole Data Service currently supports these command types on Google Cloud Platform:

• DbImport commands

• DbExport commands

• DBTap commands

• Hive commands

• Hadoop jobs

• Pig commands

• Presto commands

• Refresh Table command

• Shell commands

• Spark commands

• Workflow commands

Command Template API

Use a commmand template API to run a command more than one time and run the command with different parameter values. For more information, see Command Template API.

Custom Metastore API

These APIs allow you to connect to a custom metastore, edit and view the metastore details. For more information, see Custom Metastore API.

DbTap API

A DbTap identifies an external end point for import/export of data from QDS, such as a MySQL instance. The DbTap APIs let you create, view, edit or delete a DbTap. For more information, see DbTap API.

Folder API

This API is mainly used to create and manage a Notebook/Dashboard folders. For more information, see folder-api.

Groups API

These APIs allow you to create groups, add/delete users in a group, and assign/unassign roles to a group in a Qubole account. For more information, see groups-api.

Hive Metadata API

These APIs provide a set of read-only views that describe your Hive tables and the metadata. For more information, see Hive Metadata API.

Notebook API

These APIs allow you to create a notebook, clone, configure, run, import, and delete a notebook. For more information, see Notebook API.

Object Policy API

These APIs allow you to create object-level access policies for notebooks and clusters. For more information, see object-policy-api.

Reports API

These APIs let you view aggregated statistical and operational data for your commands. For more information, see Reports API.

Roles API

These APIs allow you to create and delete a role to perform a set of actions in a Qubole account. For more information, see Roles API.

Scheduler API

The Scheduler APIs let you schedule any command or workflow to run at regular intervals. For more information, see Scheduler API.

Sensor API

These APIs allow you to create file and partition sensors to monitor the file and Hive partition’s availability. For more information, see Sensor API.

Users API

These APIs let you view invite a user to a Qubole account and enable/disable users in a Qubole account. For more information, see Users API.