Cost Optimization and Ranking of Subnets in Multi-subnet Clusters (AWS)

Qubole supports configuring multiple subnets for a cluster. Ensure that subnets are all public or all private subnets. The cost optimization is only valid for clusters with Spot nodes.

While starting a multi-subnet cluster, Qubole first eliminates subnets that do not have sufficient IP addresses available to launch Coordinator and minimum worker nodes. After the elimination, Qubole ranks remaining subnets based on the desirability of a subnet, which depends on the cluster composition. Here is how Qubole ranks the subnets for different cluster types:

  • If the cluster contains Spot/Spot-block nodes, Qubole ranks the subnets based on the current price of the worker instance type in different AZs corresponding to the specified subnets. For a heterogeneous cluster, Qubole considers only the primary worker instance type to compare the Spot price.

  • If the cluster is a pure On-Demand cluster, Qubole ranks the subnets based on the availability of IP addresses in different subnets.

After ranking the subnets, Qubole tries to launch cluster nodes in the AZ corresponding to the best-ranked subnet. If an AZ does not have sufficient instance availability to launch Coordinator or minimum worker nodes, then:

  • For homogeneous clusters, Qubole tries to launch them in the next best AZ and so on until all AZs corresponding to specified subnets are considered.

  • For heterogeneous clusters, Qubole tries to launch nodes with alternate instance types in the best AZ until all specified instance types are considered.

This algorithm is applied to Coordinator and worker nodes separately. Hence, it is possible for Coordinator and worker nodes (in some cases) to be in different AZs but all worker nodes are always in the same AZ. As there is very less data transfer (only control messages) between the Coordinator and worker nodes, the cross AZ network communication cost is negligible.

For more information on configuring EC2 settings through a REST API call, see Create a New Cluster; if you are using the QDS GUI, see Advanced configuration: Modifying EC2 Settings (AWS). See also Understanding the Cost Optimization in AWS EC2-Classic Clusters.