Attach a Cluster to the Package Management Environment

PUT /api/v1.2/package/<env ID>/attach_cluster

Use this API to attach a cluster to a package management environment. A cluster can have only one environment attached to it. You can attach a cluster to an environment only when it is down/inactive.

When you create a new Spark cluster, by default a package environment gets created and is attached to the cluster. This feature is not enabled by default. Create a ticket with Qubole Support to enable this feature on the QDS account.

In addition to attaching Spark clusters, you can attach environments with Python 3.5 to Airflow clusters that use Airflow version 1.8.2 only if the environment is detached from the cluster. For more information, see Configuring an Airflow Cluster.

A Conda virtual environment gets created for Python and R environments. In the Spark cluster, Python and R Conda environments are located in /usr/lib/envs/. The spark.pyspark.python configuration in /usr/lib/spark/conf/spark-defaults.conf points to the Python version installed in the Conda virtual environment for a Spark cluster.

In a Spark notebook associated with a cluster attached to the package management environment, configure these in its interpreter settings to point to the virtual environment:

  • Set zeppelin.R.cmd to cluster_env_default_r

  • Set zeppelin.pyspark.python to cluster_env_default_py

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.

  • Users who belong to a group associated with a role that allows updating an environment. See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold are mandatory. Others are optional and have default values.

Parameter

Description

cluster_id

It is the Spark cluster’s unique ID to which you want to attach the environment. One Spark cluster can only have one environment attached to it.

Request API Syntax

curl -X PUT -H "X-AUTH-TOKEN: <API-TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{"cluster_id": "<Cluster ID>"}` \
"https://api.qubole.com/api/v1.2/package/<env ID>/attach_cluster"

Note

The above syntax uses https://api.qubole.com as the endpoint. Qubole provides other endpoints to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.

Sample API Request

Here is attaching a cluster to the environment with 100 as its ID.

curl -X PUT -H "X-AUTH-TOKEN: <API-TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{"cluster_id": "125"}` \
"https://api.qubole.com/api/v1.2/package/100/attach_cluster"