Understanding the Cost Optimization in AWS EC2-Classic Clusters

QDS allows you to select an AWS Availability Zone (AZ) on the basis of the Spot price history. This applies only to a hybrid cluster with no preferred AZ configured. (A hybrid cluster contains On-Demand core and minimum-number worker nodes and Spot autoscaled nodes.)

Note

This feature is available for beta access. Create a ticket with Qubole Support to enable this feature for the Qubole account.

After this feature is enabled for a Qubole account, while starting a multi-subnet cluster, Qubole selects an Availability Zone (AZ) based on the current spot-price of the worker instance type in all AZs corresponding to the specified subnets. AZs are tried in the ascending order of the spot-price. If an AZ does not have enough availability to launch Coordinator or worker nodes, then Qubole tries to launch them in the next best AZ and so on until all AZs of the region have been considered. Hence, it is possible for Coordinator and worker nodes in some cases to be in different AZs but all worker nodes are always in the same AZ. In such cases, there is an additional cost due to cross-AZ communication between Coordinator and worker nodes for control messages such as heart beats. However, this cost due to control messages is very low and should be more than offset by the EC2 cost savings due to this feature.

For more information on EC2 settings through a REST API call, see Create a New Cluster and on UI cluster configuration, see Advanced configuration: Modifying EC2 Settings (AWS), and also see Cost Optimization and Ranking of Subnets in Multi-subnet Clusters (AWS).