Configuring Cluster Overrides

Qubole has added a UI option known as Cluster Overrides in the cluster configuration to configure overrides for a few cluster configuration properties. This enhancement is part of Gradual Rollout.

You can configure the cluster overrides for these configuration properties:

  • Spot request timeout

  • Maximum price percentage of Spot, Coordinator, and Minimum nodes

  • Datadog custom tags

  • Spot allocation strategy of minimum and autoscaling nodes

You can configure it on the Advanced Configuration tab of the Clusters configuration UI while creating or updating a cluster.

Perform these steps to set cluster overrides:

  1. Navigate to the Clusters UI page. Select the cluster on which you need to configure cluster overrides. (You can also set it on a new cluster).

  2. Go to Advanced Configuration > CLUSTER OVERRIDES. It is as shown in the following figure.

    ../../_images/ClusterConfigOverride.png

  3. Pull the drop-down list to select the cluster override. It displays the list as shown in the following figure.

    ../../_images/ClusterConfigOverrideOptions.png

    Select the spot request timeout and maximum price percentage for the nodes as needed.

    The default value of the request timeout for spot nodes is auto. This lets Qubole to decide the timeout at runtime, to optimize spot fulfillment and minimize spot losses. This a the new default applied only on new clusters.

    Set the timeout in minutes. It can be any integer.

  4. After you select one cluster override and click Add another override, you can see the previously selected override as a disabled option as illustrated in this figure.

    ../../_images/ClusterConfigOverrideOptions2.png

    The default maximum price percentage is 100 %. You can change the percentage of maximum price percentage of spot nodes to any value as required.

  5. To set Datadog custom tags for Datadog metrics, click Add another override and select Datadog Custom Tags and add a value as illustrated in this figure.

    ../../_images/DatadogCustomTags.png

    You can add a comma-separated list of tags and values in the form: key1:value1,key2:value2.

  6. To change the spot allocation strategy, click Add another override and select Spot Allocation Strategy (Min Nodes) (or autoscaling nodes) as illustrated in this figure.

    ../../_images/SpotAllocationClusterOverride.png

    The spot allocation strategy determines how spot instances in the Spot fleet are fulfilled from spot instance pools.

    Qubole supports lowest price and capacity optimized spot allocation strategies. To know more, see this documentation. By default, lowest price (cost optimized) is the option that gets set. The other option is capacity optimized as illustrated in the following figure.

    ../../_images/SpotAllocationStrategyOptions.png

  7. After you configure all properties, the Add another override option gets disabled.

    Note

    In the near future, Qubole plans to add more fields as part of cluster overrides. When you are done with setting overrides, the Add another override option gets disabled.