Cluster Management

New Features

  • ACM-2329: Qubole supports the x1e instances.
  • ACM-2551: A QDS account supports the SSE-KMS server-side encryption. This enables the logs and results to be stored in the encrypted format on the S3 location.
  • ACM-2634: QDS supports the AWS Canada Region.
  • ACM-3307: QDS Supports the AWS London Region.

Bug Fixes

  • ACM-1564: Commands that are submitted when the cluster is being terminated due to the inactivity, would not error out. Beta, Via Support
  • ACM-2344: The Remove Node API will correctly display the unsupported error for Spark clusters.
  • ACM-2504: Special characters in the metastore password caused the failure in the cluster start. It is resolved now.
  • ACM-2643: A cluster’s latest configuration will not be null.
  • ACM-3125: The ap-southeast-1c AWS region’s credentials will be verified with the test instance, t2.micro.
  • TOOLS-459: The monit version is upgraded to 5.25.2.
  • TOOLS-530: Qubole has replaced ntpd with chrony on its cluster AMIs.
  • TOOLS-564: ImportError: No module named geopandas that occurred while importing packages. This error is resolved now.

Enhancements

  • ACM-2113: The experimental obfuscation of IAM access/secret keys has been removed. Security conscious users must move to the IAM-Role based accounts instead of relying on the access/secret key obfuscation for security.
  • ACM-2358: A cluster’s latest configuration will not be null.
  • ACM-2750: The disk space issue due to expired Ganglia data that was not cleaned up is resolved.
  • ACM-3063: Qubole sends node addition metrics to Datadog to monitor the autoscaling of cluster nodes. Three consecutive node addition failures raise a service check failure (alert) in the Datadog monitoring service. Beta, Via Support
  • ACM-2806: QDS directly connects to the correct AWS region endpoint from the cluster nodes instead of trying to connect to the default AWS region endpoint and thereafter connect to the correct AWS endpoint.
  • ACM-3122: Qubole sends node removal metrics to Datadog to monitor downscaling/autoscaling events in a cluster. A single node removal failures raises an alert in Datadog. Beta, Via Support
  • ACM-3137: Qubole sends the free disk space in the coordinator node metric to Datadog to monitor health of the coordinator node. Beta, Via Support
  • ACM-3251: Qubole creates a dashboard from autoscaling metrics sent to the Datadog. Similarly, it creates a dashboard for free disk space in the coordinator node. Beta, Via Support
  • ACM-3267: Qubole now creates monitors to raise alerts in case of:
    • failure to add node
    • failure to remove node
    • less than desired free space in the coordinator node
  • ACM-3616: The latest Bastion image labelled qubole-bastion-hvm-amzn-linux is available on the Amazon community. If you find multiple images with the same name, choose the latest image as it supports all instance types including the newly supported instance types such as m5.
  • TOOLS-498: Pycryptodome 3.0, a replacement for pycrypto, is added in the cluster AMIs. Qubole plans to remove the deprecated pycrypto shortly.