Using AWS Spot Instances and Spot Blocks in QDS Clusters

Amazon Web Services (AWS) offers three major types of instances (computing hosts) that are suitable for use as Qubole cluster nodes:

• On-Demand instances

• Spot instances

• Spot Blocks

On-Demand instances are offered at a fixed price, whereas the price for Spot instances varies with market demand and availability, and may well vary over the lifetime of a Qubole cluster. Spot Blocks are Spot instances that run continuously for a finite duration (1 to 6 hours).

On-Demand versus Spot Instances versus Spot Blocks: Advantages and Disadvantages

On-Demand Instances

The advantages of On-Demand instances are stability, and speed of launching. An On-Demand instance comes up very quickly and is likely to remain available for the life of the cluster, helping to ensure that the cluster performs its work smoothly and reliably. The disadvantage is cost: a cluster composed entirely or mostly of On-Demand nodes may be many times more expensive than a similarly-sized cluster composed of Spot nodes.

Spot Instances

The obvious advantage of Spot instances is cost, and the potential disadvantage is instability. A Spot instance can cost only a small fraction of the price of an On-Demand instance, but may take a long time to become available (or may never become available), and may be reclaimed by AWS at any time if the supply simply runs out. This in turn can put running jobs in jeopardy. For a detailed discussion, see the Qubole blog post Riding the Spotted Elephant.

These are a few optimizations that Qubole uses while placing Spot instance requests:

• If there is a Spot loss in a specified time window (by default in the last 15 minutes), the corresponding instance family is classified as Unstable.

• Subsequently, when there are Spot provisioning requests:

  • Instance types from unstable instance families are removed from the list of worker node types.

  • If the remaining list of worker node types is not empty, then Qubole places asynchronous AWS Fleet Spot requests for this remaining list of instance types and waits for the configured Spot Request Timeout.

  • If the remaining list of worker node types is empty, then Qubole places a synchronous Fleet Spot API request for the original worker node types (that is without filtering for unstable instance families). Synchronous requests return instantly and the Spot Request Timeout is not applicable.

• If the required capacity is still not achieved, Qubole would fall back to On-Demand nodes if it is configured However, the additional On-Demand nodes launched as a result of fallback would be replaced with Spot nodes during rebalancing.

Spot Blocks

Spot Blocks are 30 to 45 percent cheaper on average than On-Demand instances running for the same amount of time. They are more stable than Spot nodes because AWS will not reclaim them during the configured period (up to six hours), though they will be reclaimed after that. For more information, see AWS spot blocks.

QDS ensures that Spot Block instances are acquired at a lower price than On-Demand nodes, and also that instances acquired during upscaling are acquired only for the remaining lifetime of the original Spot Block instances (the Coordinator and Minimum Nodes with which the cluster was launched); that is, they are acquired for the remainder of the Spot Block Duration you configured for the Coordinator and Minimum Nodes when you created or modified the cluster. For example, if the original Spot Block instances were acquired for five hours, and nodes need to be added after the cluster has been running for two hours, the new Spot Block instances are acquired for three hours, and are proportionately cheaper than the original instances.

Note

When the Spot Block Duration expires, AWS reclaims the instances, halting the cluster. This behavior overrides the normal QDS cluster-runtime controls such as Cluster Idle Timeout.

Cluster Composition Choices

You can choose to create a cluster in any of the following configurations:

• On-Demand nodes only

• Spot nodes only

• A mix of Spot and On-Demand nodes

• Spot blocks only. (See Configuring Spot Blocks).

• On-Demand nodes for Coordinator and minimum number of nodes and Spot blocks for autoscaling.

• On-Demand nodes for Coordinator and minimum number of nodes and a mix of Spot and Spot blocks for autoscaling.

You can find out about configuring each type of cluster here.

For most purposes, the third option is the best, because it provides a good balance between cost and stability. The fourth option of using Spot Blocks is also advantageous, as the cost is lower than for On-Demand instances and availability is fixed for a specific amount of time.

The remainder of this section focuses on the settings and mechanisms Qubole provides to help safeguard the overall functioning of a cluster that includes Spot nodes.

How You Configure Spot Instances into a QDS Cluster

The critical items when you configure Spot instances into a cluster are the Request Timeout, the Maximum Price, the Qubole Placement Policy option, the Fallback to on demand option, and the Spot Instances Percentage.

• The Request Timeout specifies how many minutes Qubole should keep trying to obtain Spot instances when launching the cluster or adding nodes.

• The Maximum Price is the maximum price you are willing to pay for Spot instances. If the price of the Spot node goes above this price, then Qubole does not procure the Spot node. Qubole recommends setting this value after carefully evaluating the On-Demand price that you are currently paying for EC2 Instances, post any discounts with AWS.

• The Qubole Placement Policy option, if selected, causes Qubole to make a best effort to store one replica of each HDFS block on a stable node (normally an On-Demand node, except in the case of a Spot-only cluster). Qubole recommends you select this option to prevent job failures that could occur if all replicas were lost as a result of AWS reclaiming many Spot instances at once.

• Fallback to on demand is discussed here.

• The Spot Instances Percentage:

  • In a mixed cluster this specifies the maximum percentage of autoscaling nodes that can be Spot instances. Autoscaling nodes are those that comprise the difference between the Minimum Worker Nodes and the Maximum Worker Nodes; Qubole adds and removes these nodes according to the cluster workload, as explained in detail here.

  • In a Spot-only cluster, this is always set to 100.

The AWS Availability Zone (AZ) is also important. You can select a specific AZ or you can allow Qubole to select it. To know more on how Qubole selects AZ, see selection of AZ.

Configuring a Spot-Only Cluster (Not Recommended)

You can configure a Spot-only cluster by choosing Spot Nodes when you create the cluster. This forces the Spot Instances Percentage to 100, meaning all the cluster nodes will be Spot instances, including the core nodes (the Coordinator Node and the nodes comprising the Minimum Worker Nodes).

Causes for Poor Performance in a Spot-only Cluster

100% Spot nodes configuration in a cluster can cause poor performance for these two main reasons:

• If fallback to On-Demand is disabled, this greatly slows down upscaling and can even completely stop upscaling. So, you are running the workload on a very small cluster for a long time.

• You can lose a majority of processing nodes at any time, potentially losing, and requiring to repeat the work done by them up to that time. (But it is even almost equally likely to face this problem if the cluster configuration contains 95% Spot nodes.)

To mitigate poor performance that may occur due to 100% Spot nodes, Qubole recommends you to:

• Use heterogeneous clusters as it is less likely all instance types would experience a Spot-price surge.

• Use a larger minimum-cluster size in the cluster

• Use a smaller Spot percentage in the cluster

• In the cluster settings, keep Fallback to On-Demand Nodes enabled.

Configuring a Mixed Cluster (On-Demand and Spot Nodes)

You configure a mixed cluster by doing all of the following:

• Setting the Autoscaling Node Purchasing Option to Spot Instance.

• Setting the Spot Instances Percentage to a number less than 100.

This configures a cluster in which the core nodes (the Coordinator Node and the nodes comprising the Minimum Worker Nodes) are On-Demand instances, and a percentage of the autoscaling nodes are Spot instances as specified by the Spot Instances Percentage.

For example, if the Minimum Worker Nodes is 2 and the and the Maximum Worker Nodes is 10, and you set the Spot Instances Percentage to 50, the resulting cluster will have, at any given time:

• A minimum of 3 nodes: the Coordinator Node plus the Minimum Worker Nodes, all of them On-Demand instances (the core nodes).

• (Usually) a maximum of 11 nodes, of which up to 4 (50% of the difference between 2 and 10) will be Spot instances, and the remainder On-Demand instances. (The cluster size can occasionally rise above the maximum for brief periods while the cluster is autoscaling.)

Fallback to On-Demand

In addition to the above settings, you should normally choose Fallback to on demand (check the box). This option, if selected, causes Qubole to launch On-Demand instances if it cannot obtain enough Spot/Spot block instances when adding nodes during autoscaling. This means that the cluster could possibly at times consist entirely of On-Demand nodes – if no Spot nodes are available:

• Because of a capacity limitation in AWS Spot Market for that instance type or

• Because the current Spot price is higher than the Maximum Price

But unless cost is all-important, this is a sensible option to choose because it allows the cluster to do its work even if Spot nodes are not available.

(If Spot nodes are unavailable, Qubole also falls back to On-Demand nodes for the Coordinator and minimum nodes.)

How QDS Manages Spot Nodes While the Cluster is Running

Qubole’s primary goal in managing cluster resources is productivity– making sure that the work you need to do gets done as efficiently and reliably as possible, and at the lowest cost that is consistent with that goal.

Qubole uses the following mechanisms to help ensure maximum productivity in running clusters that deploy Spot instances:

• The Fallback to on demand option described above.

• Spot Rebalancing. This works in conjunction with Fallback to on demand, ensuring that the cluster conforms to your original specification as much of the time as possible, by swapping out On-Demand nodes in favor of Spot nodes as soon as possible after Spot nodes are available in the spot market (capacity availability) and the current Spot price returns is equal to or lower than Maximum Price.

  For a more detailed discussion, see Spot Rebalancing.

• The Qubole Placement Policy described above.

• Intelligent Availability Zone (AZ) Selection. Unless you specify a particular AZ when you configure the cluster, Qubole can automatically select the AZ with the lowest Spot prices for the region and instance type you’ve specified. This capability is supported for non-VPC clusters and VPC clusters.

  On VPC clusters, it is enabled by default. But on non-VPC clusters, it is not enabled by default. You can create a ticket with Qubole Support to enable it for your account.

• Autoscaling. Autoscaling ensures that the cluster remains at just the right size for maximum productivity and efficiency, as explained in detail here.

Skipping Spot Instances with Spot Loss

Qubole skips Spot requests for instance families for which Spot node losses were seen at the cluster level within a specific time interval, which is configurable. The default time interval is last 15 minutes. This feature is part of Gradual Rollout.

Skipping spot requests can occur in these two scenarios:

• If the cluster configuration is heterogeneous, then QDS skips instance families for which Spot losses are seen when creating a Spot Fleet request. If all instances configured have seen spot loss, then QDS skips launching spot nodes and falls back to On-Demand nodes if this option is enabled. (The fallback to On-Demand nodes option is enabled by default.)

• If the cluster configuration is homogeneous and if the configured instance family has experienced Spot losses, then QDS skips launching spot nodes and falls back to On-Demand nodes if this option is enabled. (The fallback to On-Demand nodes option is enabled by default.)

Qubole recommends configuring instances of multiple families to maximize the Spot instances’ availability.

The feature tries to eliminate the issue where earlier, whenever there was a spot loss, Qubole had tried to allocate the same instance type, which did not succeed, that is, 10 minutes were spent in waste without spot nodes allocation or even when it succeeded, the instances were lost with a 33% probability. So, with this feature enabled, Qubole skips allocating instances within the same family for 15 minutes after the spot loss is seen.

For example, if a cluster’s configuration has m3.xlarge, m3.2xlarge, m4.xlarge, m4.2xlarge. When the cluster is running , if m3.* instance type experiences a spot loss, Qubole tries to only allocate m4.* instances for the next 15 minutes.

Configuring Spot Blocks

If you configure AWS Spot Blocks for the Coordinator node and minimum number of Worker nodes, you can configure:

• Spot blocks for autoscaling (the Coordinator nodes and minimum Worker nodes must be Spot blocks); or

• Spot nodes for autoscaling or

• On-Demand nodes for autoscaling. However, you cannot configure On-Demand nodes for autoscaling when the feature to configure Spot blocks for autoscaling even when the Coordinator and minimum worker nodes are On-Demand nodes, is enabled on a Hadoop 2 (Hive) cluster.

Qubole also allows you to configure Spot blocks for autoscaling even when On-Demand is set for the Coordinator node. You can have 50% of Spot nodes and 50% of Spot blocks (you can change the percentage as necessary), which is very cost effective as it replaces On-Demand nodes.

Note

The feature to set Spot blocks as autoscaling nodes even when the Coordinator node and minimum worker nodes are On-Demand nodes, is available for a beta access. Create a ticket with Qubole Support to enable it on the account. This feature to configure Spot blocks for autoscaling worker nodes even when the Coordinator and minimum worker nodes are On-Demand nodes, is supported on Hadoop 2 (Hive) clusters. It is also supported on Presto and Spark clusters.

cluster API calls describes how to configure Spot blocks-clusters through REST API and Cluster Composition Settings (AWS) describes how to configure the Spot blocks-clusters in the cluster UI.

How Qubole Manages Spot Blocks Set as Autoscaling Nodes when the Cluster is Running

This applies to the scenario where Coordinator node and minimum worker nodes are On-Demand nodes and autoscaling nodes are a combination of Spot blocks and spot nodes. As Spot blocks are provisioned for a specific duration that ranges from 1 to 6 hours, Qubole rotates Spot blocks that come with the same duration without affecting the cluster workload.

Spot Block Rotation

Spot-block nodes are fixed duration nodes and the expiry time is known when they are acquired. So, to run longer duration tasks which exceed the Spot-block duration, Qubole rotates a Spot-block node a few minutes (governed by the two parameters that are described below) before the expiry time of the node. Here, rotation means not scheduling any new tasks on the current Spot-block node but allowing existing tasks to finish and bringing up a new Spot-block node at the same time. Once running tasks on the first Spot-block node completes and there is no useful data present on that node, Qubole gets rid of the node even if its expiry is not reached.

The following illustration shows configuring the different ways of the Spot block rotation configuration.

The Spot block rotation configuration properties are:

• yarn.autoscaling.node_rotation.min_duration_to_leave_before_starting_gd: This is the minimum duration before which Spot-block nodes are rotated. It defaults to a minimum of (60, spot_block_duration/3) in minutes.

• yarn.autoscaling.node_rotation.gd_start_period: It is the time period during which a spot-block node is rotated randomly. This duration ensures that all spot-block nodes acquired during the same time do not go away at the same time but go away randomly within this interval. It defaults to 15 minutes.

If you want to change the default values related to a Spot block rotation and AM scheduling, you can override them in the cluster UI’s Advanced Configuration > HADOOP CLUSTER SETTINGS > Override Hadoop Configuration Variables.

By default, ApplicationMasters (AMs) are not scheduled on Spot-block nodes. But you can configure the value of yarn.scheduler.node_rotation.time_delta_to_stop_am_scheduling.minutes to schedule AMs on Spot-block nodes as well if needed. If this value is set to a positive integer, AMs are scheduled on Spot blocks. This parameter specifies the time after which AMs should not be scheduled on Spot blocks.

For example, if a Spot block was supposed to be removed gracefully at x minutes from the start of its duration (this time is determined when the Spot block registers with the cluster), then AMs are scheduled until x - y minutes (where y = yarn.scheduler.node_rotation.time_delta_to_stop_am_scheduling.minutes), after which Qubole still schedules task containers until x minutes but not the Application Masters(AMs). In case, you decide to use this parameter to schedule AMs on Spot-block nodes, you should set this value to a sufficiently large number to avoid loss of AMs on Spot-block nodes. Ideally, the value should be equal to the query runtime that you expect.

The AM Scheduling configuration property to stop scheduling tasks is:

• yarn.scheduler.node_rotation.time_delta_to_stop_am_scheduling.minutes: It governs the time duration for the AM to stop scheduling tasks during the Spot block rotation. It is disabled by default.