Cluster Management

Multiple Subnets for All Cluster Compositions

ACM-1070 and ACM-2223: Earlier, multiple subnets in a VPC were supported only for hybrid clusters that is the clusters with this composition:

  • Minimum nodes: On-Demand nodes

  • Autoscaled nodes: Spot nodes

Note

This feature will be enabled for use in the next few days.

Now, Qubole supports multiple subnets for these other possible compositions: Cluster Restart Required

  • Pure On-Demand clusters

  • Pure Spot clusters

  • Spot block clusters

  • Heterogeneous clusters

Cluster Administration and Governance

ACM-2227: QDS has introduced a new cluster console providing granular visibility, enhanced governance, and ease of use with the following features Beta:

  • Customizable Cluster Overview Dashboard

  • Cluster Instances with the following metadata at each individual cluster instance

    • QCUH

    • Nodes

    • Logs

    • Autoscaling Charts at seconds level granularity (Node vs Time)

  • Usage: The UI provides QCUH consumption charts at the cluster level across node types (Spot vs ON-demand) at a monthly and daily level granularity.

  • Cluster Settings History and Auditing:

    • With this feature, admins can see the history of changes in the cluster configuration and see the configuration that is set at a given point in time.

    • In addition to this, users can also compare the differences between a set of (upto 4) cluster configuration changes and understand its lineage.

For more information, see the cluster governance description.

QDS Supports New AWS Instances

ACM-3640: Qubole now supports c5d, m5d, r5, r5d, and z1d instances. These new instances do not support Hadoop 1 clusters. The c5d.large instance does not support Presto and Spark clusters. Cluster Restart Required

Qubole does not support r5.large, r5d.large, and z1d.large instances.

Enhancements

  • ACM-3041: As Hadoop 1 is not supported now, you cannot see the Hadoop 1 cluster option when you try to create a new cluster. Contact the Qubole Support to extend the Hadoop 1’s availability until the next release if you are an existing Hadoop 1 user.

  • ACM-3628: The default Python version on clusters will be migrated to Python 2.7. As there are no further bug fixes and updates for the Python 2.6 series, it is highly recommended to migrate to Python 2.7. Via Support

  • ACM-3731: Qubole will use a single HVM image across all instance types going forward and stop publishing the PV image.

    Qubole does not support the PV custom image from now on. So, you do not have to bake PV custom images. The API which accepts custom images will continue accepting the PV image to maintain the backward compatibility but the PV image will not be used to bring up the clusters.

Bug Fixes

  • ACM-1336: To mitigate the S3 upload issues, the size of the user data file (UDF) sent to a cluster has been reduced.

  • ACM-3553: These are the issues fixed as part of this bug fix:

    • The issue where active and transitioning clusters were not displayed on the UI if there were more than 100 clusters in an account and the active clusters were not a part of the first 100 clusters is resolved.

    • The issue where searching for cluster IDs or labels was limited to the first 100 clusters in an account is resolved. Search and filters on the clusters list page now work across all clusters in the account.

    • The number of clusters displayed on the clusters page is now limited to 30 clusters. You can click the Load More button at the bottom of the list to see more clusters in the account. All active and transitioning clusters are however shown irrespective of the limit. Via Support

  • TOOLS-795: The Ruby enVironment Manager (RVM) version is frozen to 1.29.3 as opposed to always pulling the stable-version from the open source.

  • TOOLS-799: The errors that occurred while installing the Python packages have been resolved.