The new features and enhancements are:
- Qubole Supports m5ad and r5ad Instances
- Spot Blocks as Autoscaling Nodes
- Multi-instance HiveServer2 Configuration
- Presto Clusters Support Heterogeneous Nodes
Other enhancements and bug fixes are listed in:
Qubole Supports m5ad and r5ad Instances¶
ACM-4765: Qubole now supports m5ad and r5ad family instance types in US East (N. Virginia), US West (Oregon), US East (Ohio), and Asia Pacific (Singapore) AWS regions.
Spot Blocks as Autoscaling Nodes¶
ACM-4458: Qubole allows you to choose spot block nodes for configuring autoscaling nodes. Beta, Via Support As Spot blocks run for a finite duration, you can configure them depending on the length of the queries/jobs that are expected to be run on the cluster. Spot blocks provide cost effectiveness compared to On-Demand nodes and they are more reliable when compared to the Spot nodes.
The cluster UI allows you to choose multiple type of autoscaling worker nodes. Through the API, you can configure
it by setting
spot_block_settings. Cluster Restart Required
The Spot block configuration for autoscaling worker nodes even when the master and minimum worker nodes are On-Demand nodes, is only supported on Hadoop 2 (Hive) clusters.
For more information, see the documentation.
Multi-instance HiveServer2 Configuration¶
ACM-3227: You can configure multi-instances of HiveServer2 for a given Hadoop 2 (Hive) cluster in the Clusters UI. Beta, Via Support
You can also configure multi-instances of HiveServer2 through a cluster API call. Cluster Restart Required
ACM-4017: The Cluster Details which is part of the Cluster Governance and Administration feature now shows multi-instance HiveServer2 details.
Presto Clusters Support Heterogeneous Nodes¶
ACM-4218: Presto clusters now support heterogeneous nodes. Via Support
This allows you to pick any combination of instance types as worker nodes. You must carefully pick
instance types, which have similar CPU and memory capacity. Choosing instances types with significantly different CPU and
memory capacity may lead to degraded performance and increased query failures as the weakest configuration instance would
be the bottleneck during query execution. Qubole recommends you to first pick an instance family type (
then choose instance types of the same size, which are not more than one generation apart. For example,
c5.8xlarge) and so on.
Hadoop 2 (Hive) and Spark clusters already support heterogeneous nodes. For more information, see An Overview of Heterogeneous Nodes in Clusters.
- ACM-2524: Qubole allows you to configure the root volume of cluster instances using the cluster’s
node_configurationparameter through a cluster API. It plans to provide an UI option to configure the root volume in the near future. Cluster Restart Required
- ACM-3782: Qubole stores different daemons’ status and metrics of a particular cluster are now being stored. These metrics are available in the Cluster Details UI page.
- ACM-4039: Qubole now allows configuring a private subnet with no NAT configuration.
- ACM-4112: The Spot Request Timeout option in the Cluster Composition UI tab and
stable_spot_instance_settingof a cluster API now accepts 1 and 2 minutes as values.
- ACM-4221: Qubole supports Hive 3.1.1 (beta) to be configured on a Hadoop 2 (Hive) cluster. Beta, Via Support
- Qubole will rename existing Hadoop 2 (Hive) clusters as Hive clusters. Via Support
- ACM-4259: Qubole supports the
r5instance type in Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Seoul), Canada (Central), EU (London), US West (N. California), and EU (Frankfurt) AWS regions.
- ACM-4426: Qubole supports
m5dinstance types in Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Seoul), Canada (Central), EU (Ireland), EU (London), US West (N. California), and EU (Frankfurt) AWS regions.
- ACM-4058: There is a significant improvement in the loading time of the Clusters UI page.
- ACM-4659: Qubole now allows only On-Demand nodes to be configured as the master node for a 100% spot cluster. This enhancement is not available by default. Via Support
- SEC-3178: OpenJDK8 is now supported as the the default JDK on the cluster AMI and QDS Control Plane.
- ACM-4427: Fixed a bug which ensures that Qubole does not upscale a cluster beyond the maximum node limit even in case of an AWS error.
- ACM-4467: Instead of a fixed 1000 MB memory for HMS (Hive Metastore Server), the allocated memory for it will now be increased proportionally to the total memory of the cluster’s master node. Gradual Rollout
For a list of bug fixes between versions R55 and R56, see Changelog for api.qubole.com.