Managing Clusters

Clusters are pre-configured to run the jobs and queries you initiate from the Analyze page.

QDS starts clusters when they are needed and resizes them according to the query workload. A cluster lifespan depends on a session in a user account; the cluster runs as long as there is an active cluster session. A session stays alive for two hours of inactivity and can run for any amount of time as long as commands are running. Use the Sessions tab in Control Panel to create new sessions, terminate sessions, or extend sessions. For more information, see Managing Sessions.

Note

A cluster comes down after two hours of inactivity by default (as the default timeout is two hours). The timeout implies that no command is run on the cluster in the last two hours.

To see what clusters are active, and for information about each cluster, use the Clusters page. For more information on the overview of the Clusters UI page, see Using the Cluster User Interface.

Performing Cluster Operations

Cluster Operations describes how to start, terminate, create, modify, delete, and clone a cluster.

You can change the cluster configuration by clicking the Edit button against that specific cluster. Now choose one of the following sections, depending on your Cloud provider:

Modifying an AWS Cluster

You can create/modify a cluster configuration using the cluster user interface. To add a new cluster, click New on the Clusters page and to modify an existing cluster, click Edit that is against it.

Adding/Modifying the Cluster Configuration (AWS)

You can add/modify settings to a cluster through UI as described below:

  1. Type - Select the type of cluster. Once it is selected and other configurations are added, it cannot be changed or modified. This field is applicable to a new cluster that you create.
  2. Configuration. See Modifying Cluster Settings.
  3. Composition. See Modifying Cluster Composition.
  4. Advanced Configuration contains the below settings:

Modifying the Cluster Configuration (AWS)

When you add a cluster, you must select the type of the cluster and click Next to proceed to add the configuration.

For more information see Qubole Clusters.

Note

Click on the Help_Tooltip for help on a field or checkbox.

Setting Cluster as Default describes how to set a cluster as the default cluster.

QDS supports defining account-level default cluster tags. For more information, see Adding Account and User level Default Cluster Tags (AWS).

Modifying the Configuration (AWS)

In Configuration, you can modify:

  • Cluster Labels: A cluster can have one or more labels separated by a comma. You can set/label a cluster as default by selecting the default option.

  • Master Node Type: Set the master node type by selecting the preferred node type from the drop-down list.

  • Slave Node Type: Set the slave node type by selecting the preferred node type from the drop-down list. However, if you select the c3, c4, m3, m4, p2, r3, r4, and x1 slave node type, you get three additional Elastic Block Storage (EBS) configurations, that is EBS Volume Count, EBS Volume Type (Magnetic is the default EBS volume type), and EBS Volume Size (supported range is 100/500 GB - 16 TB). You can also enable EBS Upscaling on Spark and Hadoop 2 Clusters as described in Configuring EBS Upscaling in AWS Hadoop and Spark Clusters.

    Note

    You can launch the c4, i3, m4, p2, r4, and x1 instance types only in a VPC. The R4 instances are supported only on ap-south-1, eu-west-1, us-east-1, us-east-2, us-west-1, and us-west-2 regions.

    The EBS volume types that are supported with the EBS volume size are:

    • For standard (Magnetic) EBS volume type, the supported value range is 100 GB - 1 TB.
    • For ssd/gp2 (General Purpose SSD) EBS volume type, the supported value range is 100 GB - 16 GB.
    • For st1 (Throughput Optimized HDD) and sc1 (Cold HDD), the supported value range is 500 GB - 16 TB.

    Note

    For recommendations on using EBS volumes, see AWS EBS Volumes.

  • Use Multiple Slave Node Types - See Configuring Heterogeneous Slave Nodes in the Cluster Configuration UI for UI support in Hadoop 2 and Spark clusters.

  • Minimum Slave Nodes: Set the minimum number of slave nodes if you want to change it from the default 2.

  • Maximum Slave Nodes: Set the maximum number of slave nodes if you want to change it from the default 2.

  • Region - Denotes the region from which the nodes are launched. The default region is us-east-1. Select a region from the drop-down list to change it.

  • Availability Zone - By default, the AWS Availability Zone (AZ) is set to No Preference and Qubole lets AWS select an AZ while starting the cluster. Note that in any given condition, all slave nodes are in the same AZ. You can select an AZ from the drop-down list, in which instances are reserved to benefit from the AWS reserved instances. Click the refresh icon RefreshIcon2 to refresh the availability zones’ list.

    Note

    Qubole provides a feature to select an AWS AZ based on the optimal Spot price in a hybrid cluster (a cluster with On-Demand and Spot nodes). This feature is available for beta access. Create a ticket with Qubole Support to enable this feature for the account. For more information, see Understanding Cost Optimization in AWS EC2-Classic Clusters.

  • Node Bootstrap File: Set the node bootstrap file.

  • Disable Automatic Cluster Termination is off by default (that is, clusters are automatically terminated unless you check this box). See Shutting Down an Idle Cluster.

  • Idle Cluster Timeout - The default cluster timeout is 2 hours. Optionally, you can configure it between 0 to 6 hours; that is, the value range is 0-6 hours. The only unit of time supported is hours. If the timeout is set at the account level, it applies to all clusters within that account. However, you can override the timeout at the cluster level. The timeout is effective on the completion of all queries on the cluster. Qubole terminates a cluster on an hour boundary. For example, when idle_cluster_timeout is 0, then if there is any node in the cluster near its hour boundary (that is it has been running for 50-60 minutes and is idle even after all queries are executed), Qubole terminates that cluster. This setting is not applicable for Airflow clusters as they do not automatically terminate.

    Note

    The Idle Cluster Timeout can be configured in hours and minutes. This feature is not enabled on all accounts by default. Create a ticket with Qubole Support to enable this feature on the QDS account.

  • Node Bootstrap File: Set the node bootstrap file.

  • Other Settings:

    • Enable Ganglia Monitoring is enabled automatically if you use Datadog; otherwise it is disabled by default.

Cluster Basics and Configuring Clusters provide more information on the cluster and its configuration options.

How to Push Configuration to a Cluster provides more information on pushing certain configuration to a running cluster.

Click Next to update the cluster composition.

On each cluster configuration tab, you have a right pane for Summary from which you can:

  • Do a review and you can edit to change any configuration from the Summary pane.

  • After doing a review of the entire clusters’s configuration, click Create for a new cluster.

    Click Update for changing an existing cluster’s configuration. Click Cancel to restore the previous settings.

    Click Previous to go the previous tab.

Modifying Cluster Composition (AWS)

As part of the cluster composition, the settings are:

  • Master and Minimum Slave Nodes - By default the value is On-Demand nodes, which are stable and are not lost as the bid price is fixed. Select this option to avoid risk of losing nodes if Spot instances are unavailable or are terminated. The other two options are:

    • Spot Nodes

    • Spot Blocks - QDS supports configuring Spot Block instances on the new cluster UI page and also through cluster API calls. Spot Blocks are Spot instances that run continuously for a finite duration (1 to 6 hours). They are 30 to 45 percent cheaper than On-Demand instances based on the requested duration. They are stable than spot nodes as they are not susceptible to being taken away for the specified duration. However, these nodes certainly get terminated once the duration for which they are requested for is completed. For more details, see AWS spot blocks.

      QDS ensures that Spot blocks are acquired at a price lower than On-Demand nodes. It also ensures that autoscaled nodes are acquired for the remaining duration of the cluster. For example, if the duration of a Spot block cluster is 5 hours and there is a need to autoscale at the 2nd hour, Spot blocks are acquired for 3 hours.

      Once you select Spot block (beta), an additional text box is displayed to configure the duration. The default value is 2 hours. The supported value range is 1-6 hours.

  • Autoscaling Slave Nodes type. The default is Spot nodes. The other options are:

    • On-Demand nodes
    • Spot Block. You can only set it for autoscaling only if the master and minimum slave nodes are Spot Blocks.
  • Enable Fallback to on demand to avoid the risk of data loss as spot instance bid price is volatile

  • Set Maximum Bid Price in percentage if you want to change the default 100%

  • Set Request timeout if you want to change the default 10 minutes

  • Set Spot Nodes(%) if you want to change the default 50%

    Cluster Composition provides more description about all the options.

On each cluster configuration tab, you have a right pane for Summary from which you can:

  • Do a review and you can edit to change any configuration from the Summary pane.

  • After doing a review of the entire clusters’s configuration, click Create for a new cluster.

    Click Update for changing an existing cluster’s configuration. Click Cancel to restore the previous settings.

After updating the cluster composition, click Next to set the Advanced configuration that contains the options described in the following section.

Modifying Advanced Configuration

In Advanced Configuration, you can add/modify:

Note

The cluster options described are as organized in the new Clusters UI page. Since the new UI is not globally available yet, create a ticket with Qubole Support if you would like to enable it for the account.

Modifying EC2 Settings (AWS)

Amazon Elastic Cloud Compute (EC2) web service offers resizable compute capacity in the cloud. EC2 settings’ fields are visible only when the Compute Type is managed by you and not Qubole. The following figure shows an example of EC2 settings.

../../_images/EC2Settings.png

Note

If IAM Roles is set as the access mode type in account settings. Access and Secret Keys are not displayed in EC2 SETTINGS. Instead, the configured Role ARN and External ID text fields are displayed in EC2 SETTINGS (The two fields can be modified in Control Panel > Account Settings. Changing the two fields changes compute access and storage access for all clusters of the specific account).

EC2 SETTINGS contains the following options:

  • Same As Default Compute - Select this checkbox if you want the default compute settings for the cluster.

    Note

    Once you select Same as Default Compute, the Compute Access Key and Compute Secret Key text fields are not displayed.

  • Compute Access Key - Enter an access key (along with the secret key) to set AWS Credentials for launching the cluster.

  • Compute Secret Key - Enter a secret key (along with the access key) to set AWS Credentials for launching the cluster. Click Edit Secret Key to add a different key. An empty text field is displayed. Add a different secret key and click Hide Secret Key.

  • VPC - Set this configuration to launch the cluster within an Amazon Virtual Private Cloud (VPC). It is an optional configuration. If VPC is not specified, cluster nodes are placed in a Qubole created Security Group in EC2-Classic or in the default VPC for EC2-VPC accounts. After you select a VPC, the Bastion Host text field shows up and it is hidden when no VPC is selected. By default, None is selected. Click the refresh icon RefreshIcon1 to refresh the VPC list.

  • Subnet - When you select a VPC, a corresponding subnet gets selected. By default, the text field is blank as no VPC is selected. You can select a private subnet ID or a public subnet ID for the corresponding VPC.

    You can select multiple private subnet IDs or public subnet IDs (if configured) for the corresponding VPC in a hybrid cluster. (A hybrid cluster contains on-demand core/minimum-number slave nodes and spot autoscaled nodes.) Public and private subnets that you select must be from different AWS AZ. For more information on how Qubole helps in optimizing AWS AZ cost based on the Spot price, see Understanding Cost Optimization in Multiple-Subnet Clusters (AWS).

  • Bastion Node - You can see this text field after you select a VPC. Specify the Bastion host DNS in the text field. You must specify a Bastion host DNS for a private subnet and this text field is not used for public subnets.

  • Bastion Node Port - The default port is 22. You can specify a non-default port.

  • Bastion Node User - It has the default user specified. You can specify a non-default user if required.

  • Use Dedicated Instances - You can see this text field after you select a VPC. Enable it if you want the master and slave nodes to be dedicated instances. The dedicated instance would mean the instances launched do not share physical hardware with any other instances outside of the respective AWS account.

  • Elastic IP for Master Node - It is the Elastic IP address for attaching to the cluster master. For more information, see this documentation.

  • Custom EC2 Tags - You can set a custom EC2 tag if you want the instances of a cluster to get that tag on AWS. This tag is useful in billing across teams as you get the AWS cost per tag, which helps to calculate the AWS costs of different teams in a company.

    Add a tag in Custom Tag and enter a value for the tag in the corresponding text field as shown in the above figure. Tags and values must have alphanumeric characters and can contain only these special characters: + (plus-sign), . (full stop/period/dot), - (hyphen), @ (at-the-rate of symbol), and _ (an underscore). The tags, Qubole, and alias are reserved for use by Qubole (see Qubole Cluster EC2 Tags (AWS)). Tags beginning with aws- are reserved for use by Amazon. Click Add EC2 tags if you want to add more than one custom tag. A maximum of five custom tags can be added per cluster.

    Qubole now supports account level and user level default cluster tags, which is a feature enabled on request. For more information, see Adding Account and User level Default Cluster Tags (AWS).

EC2 Settings provides more information. See Manage Roles for information on how permissions to access features are granted and restricted.

Modifying Hadoop Cluster Settings (AWS)

Use the Edit Cluster page to make changes to a cluster, as follows.

In HADOOP CLUSTER SETTINGS, you can:

  • Specify the Default Fair Scheduler Pool if the pool is not submitted during job submission.
  • Override Hadoop Configuration Variables for the Slave Node Type specified in the Cluster Settings section. The settings shown in the Recommended Configuration field are used unless you override them.
  • Set Fair Scheduler Configuration values to override the default values.

Hadoop-specific Options provides more description about the options.

Note

Recommissioning can be enabled on Hadoop 1 and Hadoop 2 clusters as an Override Hadoop Configuration Variable. See Enable Recommissioning for more information.

See Enabling Container Packing in Hadoop 2 and Spark for more information on how to more effectively downscale in an Hadoop 2 cluster.

If cluster type is Presto, you must set Presto Settings as shown in the following figure.

../../_images/PrestoSettings.png

See Presto Cluster Configuration.

If the cluster type is Spark, Spark configuration is set in the Recommended Spark Configuration, which is in addition to Hadoop Cluster Settings as described in Configuring a Spark Cluster.

Modifying Security Settings (AWS)

In SECURITY SETTINGS:

  • Qubole Public Key cannot be changed. QDS uses this key to access the cluster and run commands on it.

It is now possible to enhance security of a cluster by authorizing Qubole to generate a unique SSH key every time a cluster is started. This feature is not enabled by default. Create a ticket with Qubole Support to enable this feature. Once this feature is enabled, Qubole starts using the unique SSH key to interact with the cluster. For clusters running in private subnets, enabling this feature generates a unique SSH key for the Qubole account. This SSH key must be authorized on the Bastion host.

Setting Customer Public SSH Key is optional.
  • Persistent security groups - This option overrides the account-level security group settings. By default, this option is not set but it inherits the account-level persistent security group, if any. Use this option if you want to give additional access permissions to cluster nodes.

    Select Enable encryption to encrypt ephemeral storage. It is an option to encrypt the data at rest on the node’s ephemeral (local) storage. This includes HDFS and any intermediate output generated by Hadoop. The block device encryption is setup before the node joins the cluster and can increase the bring up time of the cluster.

Modifying Cluster Monitoring Settings

In MONITORING:

Modifying Datadog Settings (AWS)

Qubole supports the Datadog cloud monitoring service on Hadoop 1, Hadoop 2 (Hive), Presto, and Spark clusters.

In MONITORING:

  • You can override the Datadog settings for a specific Hadoop 1, Hadoop 2 (Hive), Presto, and Spark cluster (if settings are configured in the Account Settings of a QDS account). Specify the API token and APP token in the corresponding text fields. When you configure Datadog settings, Ganglia monitoring is automatically enabled.

    Note

    Although the Ganglia monitoring is enabled, its link may not be visible in the cluster’s UI resources list.

On each cluster configuration tab, you have a right pane for Summary from which you can:

  • Do a review and you can edit to change any configuration from the Summary pane.

  • After doing a review of the entire clusters’s configuration, click Create for a new cluster.

    Click Update for changing an existing cluster’s configuration. Click Cancel to restore the previous settings.

    Click Previous to go the previous tab.

After adding/modifying the basic and advanced configuration of a cluster, click Create for a new cluster or click Update to modify the existing cluster’s configuration.

Modifying Cluster Settings for Azure

On the Configuration page, you can add or modify:

  • Cluster Labels: A cluster can have one or more labels separated by a commas. You can make a cluster the default cluster by including the label default.
  • Presto Version: For a Presto cluster, choose a supported version from the drop-down list.
  • Master Node Type: To change the master node type from the default (Standard_A5), select a different type from the drop-down list.
  • Slave Node Type: To change the slave node type from the default (Standard_A5), select a different type from the drop-down list.
  • Use Multiple Slave Node Types - (Hadoop 2 and Spark clusters) See Configuring Heterogeneous Slave Nodes in the Cluster Configuration UI.
  • Minimum Slave Nodes: Enter the minimum number of slave nodes if you want to change it (the default is 1).
  • Maximum Slave Nodes: Enter the maximum number of slave nodes if you want to change it (the default is 1).
  • Node Bootstrap File: You can append the name of a node bootstrap script to the default path.
  • Disable Automatic Cluster Termination: Check this box if you do not want QDS to terminate idle clusters automatically. Qubole recommends that you leave this box unchecked.

On the Advanced Configuration page, you can add or modify:

  • Same As Default Compute: Uncheck this to change your default Azure compute settings.

  • Location: Click on the drop-down list to choose the Azure geographical location.

  • Managed Disk Type: It denotes the type of disk storage. Set it as Standard or Premium depending on the disk type. Once you set the Managed Disk Type, you cannot set the Disk Storage Account.

    Note

    You can either set Managed Disk Type or Disk Storage Account Name and related configuration.

  • Disk Storage Account Name: Choose your Azure storage account.

  • Data Disk Count: Choose the number of reserved disk volumes to be mounted to each cluster node, if any. Reserved volumes provide additional storage reserved for HDFS and intermediate data such as Hadoop Mapper output.

  • Data Disk Size: Choose the size in gigabytes of each reserved volume.

  • Vnet Resource Group Name: Choose the resource group of your virtual network (VNet).

  • Vnet Name: Choose the virtual network.

  • Subnet Name: Choose the subnet.

  • Network Security Group: Optionally choose a network security group from the drop-down menu. A network security group contains a list of security rules that permit or restrict network access to resources connected to Vnets. If you specify a network security group here, QDS adds all of its rules to the default security group it creates for the cluster.

  • Custom Tags: You can create tags to be applied to the Azure virtual machines.

  • Override Hadoop Configuration Variables: Enter Hadoop variables here if you want to override the defaults (Recommended Configuration) that Qubole uses.

  • Override Presto Configuration: For a Presto cluster, enter Presto settings here if you want to override the defaults that Qubole uses.

  • Fair Scheduler Configuration: For a Hadoop or Spark cluster, enter Hadoop Fair Scheduler values if you want to override the defaults that Qubole uses.

  • Default Fair Scheduler Pool: Specify the default Fair Scheduler queue (used if no queue is specified when the job is submitted).

    Note

    In the Hadoop 2 implementation, pools are referred to as “queues”.

  • Override Spark Configuration: For a Spark cluster, enter Spark settings here if you want to override the defaults (Recommended Configuration) that Qubole uses.

  • Enable Ganglia Monitoring: Check this box to enable performance monitoring.

  • Customer SSH Public Key: The public key from an SSH public-private key pair, used to log in to QDS cluster nodes. To use SSH, you must configure a network security group with a rule that allows access to the cluster Subnet via an ACL (Access Control List) or other means.

Qubole Public Key cannot be changed. QDS uses this key to gain access to the cluster and run commands on it.

Note

You can improve the security of a cluster by authorizing Qubole to generate a unique SSH key every time a cluster is started. This is not enabled by default; create a ticket with Qubole Support to enable it. Once it is enabled, Qubole will use the unique SSH key to interact with the cluster. For clusters running in private subnets, QDS uses a unique SSH key for the Qubole account; the SSH key must be authorized on the Bastion host.

When you are satisfied with your changes, click Create or Update.

Modifying Cluster Settings for Oracle OCI

Under the Type tab, you can choose a Hadoop 2 or Spark Cluster.

Under the Configuration tab, you can modify:

  • Cluster Labels: A cluster can have one or more labels separated by a commas. You can make a cluster the default cluster by including the label “default”.
  • Master Node Type: To change the master node type from the default, select a different type from the drop-down list.
  • Slave Node Type: To change the slave node type from the default, select a different type from the drop-down list.
  • Use Multiple Slave Node Types - (Hadoop 2 and Spark clusters) See Configuring Heterogeneous Slave Nodes in the Cluster Configuration UI.
  • Minimum Slave Count: Enter the minimum number of slave nodes if you want to change it (the default is 2).
  • Maximum Slave Count: Enter the maximum number of slave nodes if you want to change it (the default is 2).
  • Region: To change the region, select a different one from the drop-down list.
  • Availability Zone: To change the Availability Zone, select a different one from the drop-down list.
  • Node Bootstrap File: You can append the name of a node bootstrap script to the default path.
  • Disable Automatic Cluster Termination: You can enable or disable automatic termination of an idle cluster. Qubole recommends leaving automatic termination enabled (that is, leave the box unchecked). Otherwise you may accidentally leave an idle cluster running and incur unnecessary and possibly large costs.

Under the Advanced Configuration tab, you can:

  • Choose Same as Default Compute (check the box); or
  • Modify the settings as described here.

You can also change the Compartment ID, VCN, and Subnet. These must meet the requirements described here.

In the HADOOP CLUSTER SETTINGS section, you can modify:

  • Hadoop Configuration Variables: Enter Hadoop variables here if you want to override the defaults that Qubole uses.

  • Fair Scheduler Configuration: Enter Hadoop Fair Scheduler values if you want to override the defaults that Qubole uses.

  • Default Fair Scheduler Pool: Specify the default Fair Scheduler pool (used if no pool is specified when the job is submitted).

    Note

    In the Hadoop 2 implementation, pools are referred to as “queues”.

In the MONITORING section, you can choose to enable Ganglia monitoring (check the box).

In the SECURITY SETTINGS section, you can modify:

  • Customer SSH Public Key: The public key from an SSH public-private key pair, used to log in to QDS cluster nodes.

Qubole Public Key cannot be changed. QDS uses this key to gain access to the cluster and run commands on it.

Note

You can improve the security of a cluster by authorizing Qubole to generate a unique SSH key every time a cluster is started. This is not enabled by default; create a ticket with Qubole Support to enable it. Once it is enabled, Qubole will use the unique SSH key to interact with the cluster. For clusters running in private subnets, QDS uses a unique SSH key for the Qubole account; the SSH key must be authorized on the Bastion host.

When you are satisfied with your changes, click Create or Update.

Modifying Cluster Settings for Oracle OCI Classic

Under the Type tab, you can choose a Hadoop 2 or Spark Cluster.

Under the Configuration tab, you can modify:

  • Cluster Labels: A cluster can have one or more labels separated by a commas. You can make a cluster the default cluster by including the label “default”.
  • Master Node Type: To change the master node type from the default, select a different type from the drop-down list.
  • Slave Node Type: To change the slave node type from the default, select a different type from the drop-down list.
  • Minimum Slave Count: Enter the minimum number of slave nodes if you want to change it (the default is 2).
  • Maximum Slave Count: Enter the maximum number of slave nodes if you want to change it (the default is 2).
  • Data Disk Count: Optionally specify the number of data disks to be mounted to each node (the default is none).
  • Data Disk Size: If you are adding data disks, specify the size of each disk, in GB (the default is 100).
  • Node Bootstrap File: You can append the name of a node bootstrap script to the default path.
  • Disable Automatic Cluster Termination: You can enable or disable automatic termination of an idle cluster. Qubole recommends leaving automatic termination enabled (that is, leave the box unchecked). Otherwise you may accidentally leave an idle cluster running and incur unnecessary and possibly large costs.

Under the Advanced Configuration tab, you can:

  • Choose Same as Default Compute (check the box); or
  • Modify the settings as described here.

You can also optionally specify a private subnet and ACL for this cluster.

In the HADOOP CLUSTER SETTINGS section, you can modify:

  • Hadoop Configuration Variables: Enter Hadoop variables here if you want to override the defaults that Qubole uses.
  • Fair Scheduler Configuration: Enter Hadoop Fair Scheduler values if you want to override the defaults that Qubole uses.
  • Default Fair Scheduler Queue: Specify the default Fair Scheduler queue (used if no queue is specified when the job is submitted).

In the MONITORING section, you can choose to:

  • Enable Ganglia monitoring (check the box).
  • Override your account-level Datadog settings, if any: provide the API and App tokens to be used by this particular cluster.

In the SECURITY SETTINGS section, you can modify:

  • Customer SSH Public Key: The public key from an SSH public-private key pair, used to log in to QDS cluster nodes.

When you are satisfied with your changes, click Save.