Create a QDS Account using API version 2

POST /api/v2/accounts

Use this API to create a QDS account.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.

  • Users who belong to a group associated with a role that allows creating an account. See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter

Description

name

Name of the account. Provide a name to the account.

level

Denotes the type of account privilege and its possible value is free.

compute_type

Denotes the type of compute credentials for the account. Its value must be CUSTOMER_MANAGED.

storage_type

Denotes the type of storage credentials for the account. Its value must be CUSTOMER_MANAGED.

CacheQuotaSizeInGB

The default value of Amazon S3 cache size is 25 GB. It can be more than 25 GB and does not have a maximum limit. It is set to maintain a size-limited cache for files stored on the local device and once the cache size limit is reached, Qubole clears excess files from the local cache. Qubole’s caching filesystem buffers data locally and asynchronously uploads files to HDFS and S3. The S3 Cache Size is applicable to Hadoop and Spark clusters.

defloc

It is the default location on Amazon S3 for the data for storing logs, results, and cached data. It can be any location on S3 buckets. Change the Default Location as it is recommended and the purpose is highlighted in this warning.

Warning

To avoid adverse effects on the command latency and performance, specify a separate S3 bucket for each QDS account and avoid providing long prefixes and sharing the same S3 bucket for all accounts.

access_mode

Denotes the access type you want to use to create an account. The possible values are IAM_ROLES and IAM_KEYS (default). If the access mode is IAM_KEYS, the acc_key, secret, compute_access_key and compute_secret_key parameters are mandatory.

storage_access_key

The AWS Access Key used as a storage credential. It is mandatory when the access mode is IAM_KEYS.

storage_secret_key

The AWS Secret Key used as a storage credential. It is mandatory when the access mode is IAM_KEYS.

compute_access_key

The AWS Access Key used as a compute credential. It is mandatory when the access mode is IAM_KEYS.

compute_secret_key

The AWS Secret Key used as a compute credential. It is mandatory when the access mode is IAM_KEYS

aws_region

Denotes the AWS region. Valid values are, us-east-1, us-east-2, us-west-1, us-west-2, eu-west-1, eu-west-2, sa-east-1, ap-south-1, ap-southeast-1, ap-northeast-1, ap-northeast-2, and ca-central-1.

idle_cluster_timeout

The default cluster timeout is 2 hours. Optionally, you can configure it between 0 to 6 hours that is the value range is 0-6 hours. The unit of time supported is only hour. If the timeout is set at account level, it applies to all clusters within that account. However, you can override the timeout at cluster level. The timeout is effective on the completion of all queries on the cluster. Qubole terminates a cluster in an hour boundary. For example, when idle_cluster_timeout is 0, then if there is any node in the cluster near its hour boundary (that is it has been running for 50-60 minutes and is idle even after all queries are executed), Qubole terminates that cluster.

idle_cluster_timeout_in_secs

After enabling the aggressive downscaling feature on the QDS account, the Cluster Idle Timeout can be configured in seconds. Its minimum configurable value is 300 seconds and the default value would still remain 2 hours (that is 120 minutes or 7200 seconds).

Note

This feature is only available on a request. Contact the account team to enable this feature on the QDS account.

idle_session_timeout

If there is no activity on the UI, Qubole logs you out after a specified interval. The default value is 1440 minutes (24 hours). Use this option to configure the time interval in minutes if you do not want the default idle session timeout. The maximum value is 10800 minutes (1 week).

sub_account_creation

This parameter is set to use the account plan. By default, it is set to false. Possible values are true and false with the following implications:

  • Setting the value to true implies that the new account uses the parent account’s plan.

  • The default value set to false implies that the new account uses Qubole’s free trial-period plan.

Request API Syntax

curl -i -X POST -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{"account": {"name":"new_account", "defloc":"<s3location>", "access_mode": "<ACCESS_MODE>", "storage_access_key" : "<ACCESS_KEY>",
"storage_secret_key" :"<SECRET_KEY>", "compute_access_key" :"<COMPUTE_ACCESS_KEY>", "compute_secret_key":"<COMPUTE_SECRET_KEY>",
"compute_type":"CUSTOMER_MANAGED", "storage_type":"CUSTOMER_MANAGED", "aws_region": "us-east-1"} }' "https://api.qubole.com/api/v2/accounts"

Note

The above syntax uses https://api.qubole.com as the endpoint. Qubole provides other endpoints to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.

Sample Request using IAM Keys

Here is a sample request API to create a new QDS account using IAM keys.

curl -i -X POST -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
 -d '{
      "account":
      {
        "name": "Test",
        "level": "free",
        "compute_type":"CUSTOMER_MANAGED",
        "storage_type":"CUSTOMER_MANAGED",
        "CacheQuotaSizeInGB": 50,
        "defloc": "$DEFLOC",
        "access_mode" : "IAM_KEYS",
        "storage_access_key" : "$STORAGE_ACCESS_KEY",
        "storage_secret_key" : "$STORAGE_SECRET_KEY",
        "compute_access_key" : "$COMPUTE_ACCESS_KEY",
        "compute_secret_key": "$COMPUTE_SECRET_KEY",
        "idle_cluster_timeout": 3,
        "idle_session_timeout": 2880,
        "sub_account_creation": false,
        "aws_region": "us-east-1"
      }
      }' "https://api.qubole.com/api/v2/accounts/"

Sample Response using IAM Keys

{
    "name": "clone acc",
     "state": "success_create",
     "groups": [
         "system-admin",
         "everyone"
     ],
     "authentication_token": "$AUTH_TOKEN",
     "account_id": "$ACCOUNT_ID",
     "is_aws_keys_enabled": true,
     "status": "success"
}

Create an Account using IAM Roles

Perform the following steps to create an IAM role-based account:

  1. Use the following sample to create an account and get the external ID.

  2. Map the external ID to the Role ARN (in AWS). See Creating a Cross-account IAM Role.

  3. Update the account request with the Role ARN. See Edit a QDS Account.

Sample Request using IAM Roles

curl -i -X POST -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
 -d '{
      "account":
      {
        "name": "Test",
        "level": "free",
        "compute_type":"CUSTOMER_MANAGED",
        "storage_type":"CUSTOMER_MANAGED",
        "CacheQuotaSizeInGB": 50,
        "defloc": "$DEFLOC",
        "access_mode" : "IAM_ROLES",
        "idle_cluster_timeout": 3,
        "idle_session_timeout": 2880,
        "sub_account_creation": false,
        "aws_region": "us-east-1"
      }
      }' "https://api.qubole.com/api/v2/accounts/"

Sample Response using IAM Roles

{
    "name": "clone acc",
     "state": "success_create",
     "groups": [
         "system-admin",
         "everyone"
     ],
     "authentication_token": "$AUTH_TOKEN",
     "account_id": "$ACCOUNT_ID",
     "is_aws_keys_enabled": false,
     "storage_external_id": "$STORAGE_EXTERNAL_ID"
     "status": "success"
}