3. How can I use AWS Reserved Instances with Qubole?

Qubole uses Linux instances as cluster nodes. You can use reserved instances with Qubole by purchasing AWS Reserved Instances (RIs) from Amazon of type Linux/Unix to reduce your long running cluster costs. The instance type you reserve must be the same as the instance type you specify while configuring clusters. The RIs must be in VPC/non-VPC depending on whether the cluster is in VPC or non-VPC. When On-Demand instances are purchased in the AWS Availability Zone, the RIs are purchased first and after they get exhausted, regular On-Demand instances are purchased. A cluster can have any ratio of Spot and On-Demand nodes. As long as there are RIs available, AWS bills the On-Demand part of cluster nodes based on RI prices.

While configuring a cluster, choose the default On-Demand Instance or Spot instance autoscaling node purchase option to use AWS Reserved Instances on the cluster configuration UI. Set OnDemand as the slave_instance_type while configuring a cluster to use Reserved Instances. As long as there are OnDemand instances used in the cluster, these instances have RI prices.

For more information, see Cluster Composition Settings (AWS) and node_configuration.

Note

AWS supports configuring RIs purchased in any AZ of a specific AWS region only when you do not specify any particular AZ of that AWS region while purchasing the RIs. Otherwise, you purchase RIs per an AWS AZ and so you must configure that AWS AZ in the cluster’s EC2 settings.

Qubole supports Standard RIs and Convertible RIs. Convertible RIs can change instance families and this characteristic is useful in heterogeneous clusters as they can change instance families. For more information on heterogeneous clusters, see Using Heterogeneous Nodes in Hadoop 2/Presto/Spark Clusters.

Note

Go to the AWS console to verify if the cluster is using RIs.