Cluster Management

The new features and enhancements are:

Other enhancements and bug fixes are listed in:

Qubole Supports m5ad and r5ad Instances

ACM-4765: Qubole now supports m5ad and r5ad family instance types in US East (N. Virginia), US West (Oregon), US East (Ohio), and Asia Pacific (Singapore) AWS regions.

Spot Blocks as Autoscaling Nodes

ACM-4458: Qubole allows you to choose spot block nodes for configuring autoscaling nodes. Beta, Via Support As Spot blocks run for a finite duration, you can configure them depending on the length of the queries/jobs that are expected to be run on the cluster. Spot blocks provide cost effectiveness compared to On-Demand nodes and they are more reliable when compared to the Spot nodes.

The cluster UI allows you to choose multiple type of autoscaling worker nodes. Through the API, you can configure it by setting use_spot_block_for_autoscale_stable_node to true in spot_block_settings. Cluster Restart Required

The Spot block configuration for autoscaling worker nodes even when the coordinator and minimum worker nodes are On-Demand nodes, is only supported on Hadoop 2 (Hive) clusters.

For more information, see the documentation.

Multi-instance HiveServer2 Configuration

  • ACM-3227: You can configure multi-instances of HiveServer2 for a given Hadoop 2 (Hive) cluster in the Clusters UI. Beta, Via Support

    You can also configure multi-instances of HiveServer2 through a cluster API call. Cluster Restart Required

  • ACM-4017: The Cluster Details which is part of the Cluster Governance and Administration feature now shows multi-instance HiveServer2 details.

For more information, see UI documentation and API documentation.

Presto Clusters Support Heterogeneous Nodes

ACM-4218: Presto clusters now support heterogeneous nodes. Via Support

This allows you to pick any combination of instance types as worker nodes. You must carefully pick instance types, which have similar CPU and memory capacity. Choosing instances types with significantly different CPU and memory capacity may lead to degraded performance and increased query failures as the weakest configuration instance would be the bottleneck during query execution. Qubole recommends you to first pick an instance family type (r/m/c) and then choose instance types of the same size, which are not more than one generation apart. For example, (r3.2xlarge, r4.2xlarge), (r4.4xlarge, r5.4xlarge, r5a.4xlarge), (c4.8xlarge, c5.8xlarge) and so on.

Hadoop 2 (Hive) and Spark clusters already support heterogeneous nodes. For more information, see An Overview of Heterogeneous Nodes in Clusters.

Enhancements

  • ACM-2524: Qubole allows you to configure the root volume of cluster instances using the cluster’s node_configuration parameter through a cluster API. It plans to provide an UI option to configure the root volume in the near future. Cluster Restart Required
  • ACM-3782: Qubole stores different daemons’ status and metrics of a particular cluster are now being stored. These metrics are available in the Cluster Details UI page.
  • ACM-4039: Qubole now allows configuring a private subnet with no NAT configuration. Feature to opt in | Cluster Restart Required
  • ACM-4112: The Spot Request Timeout option in the Cluster Composition UI tab and timeout_for_request under stable_spot_instance_setting of a cluster API now accepts 1 and 2 minutes as values.
  • ACM-4221: Qubole supports Hive 3.1.1 (beta) to be configured on a Hadoop 2 (Hive) cluster. Beta, Via Support
  • Qubole will rename existing Hadoop 2 (Hive) clusters as Hive clusters. Via Support
  • ACM-4259: Qubole supports the r5 instance type in Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Seoul), Canada (Central), EU (London), US West (N. California), and EU (Frankfurt) AWS regions.
  • ACM-4426: Qubole supports c5d and m5d instance types in Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Seoul), Canada (Central), EU (Ireland), EU (London), US West (N. California), and EU (Frankfurt) AWS regions.
  • ACM-4058: There is a significant improvement in the loading time of the Clusters UI page.
  • ACM-4659: Qubole now allows only On-Demand nodes to be configured as the coordinator node for a 100% spot cluster. This enhancement is not available by default. Via Support
  • SEC-3178: OpenJDK8 is now supported as the the default JDK on the cluster AMI and QDS Control Plane.

Bug Fixes

  • ACM-4427: Fixed a bug which ensures that Qubole does not upscale a cluster beyond the maximum node limit even in case of an AWS error.
  • ACM-4467: Instead of a fixed 1000 MB memory for HMS (Hive Metastore Server), the allocated memory for it will now be increased proportionally to the total memory of the cluster’s coordinator node. Gradual Rollout

For a list of bug fixes between versions R55 and R56, see Changelog for api.qubole.com.