What’s New
Important new features and improvements are as follows.
Note
A link in blue text next to a description in these Release Notes indicates the launch state, availability, and default state of the item (for example, Beta). The link provides more information. Unless otherwise stated, features are generally available, available as self-service (without intervention by Qubole support), and enabled by default.
The Account Level Concurrent Command Limit (shown under Account Settings in the QDS UI) has increased from 20 to 100. Gradual Rollout.
The Clusters page of the QDS UI displays a new cluster health tile card with metrics. Learn more.
QDS now supports Azure VMs of type E*as v4 (for example, E16as v4). This should be particularly useful in Presto clusters. See billing-qubole-index-Azure for billing information.
A new version of the Analyze page, previously released as New Analyze, is now called Workbench. Beta.
Cluster monitoring, includng daemon status, heap usage and coordinator node metrics, is available in Workbench and a REST API.
QDS now allows you to configure buffer capacity in Hadoop and Spark clusters. Learn more.
QDS Hadoop now allows more containers per node, improving memory management in YARN.
Hive 2.3 is generally available. Learn more.
Qubole JDBC Storage Handler is deprecated. Learn more.
QDS supports enterprise installations of Github and Gitlab. Via Support. Learn more.
The Environments UI is now available in the Control Panel by default for new users. Beta.
Ranger support for Spark is now available in Beta
QDS now supports Azure Active Directory (AD) for Single Sign-On (SSO).
QDS now supports Azure Data Lake Storage Gen 2 with users’ access to data controlled by their Azure Active Directory permissions.
Added a new scheduler to optimally schedule tasks based on locality of data cached with Rubix. See https://www.qubole.com/blog/presto-rubix-scheduler-improves-cache-reads/.
Added
call hive.default.clear_cache()procedure call to clear stale Hive metastore caches. Useful when metastore updates might have occurred from outside the Presto cluster.Made performance improvements in queries involving IN and NOT IN over a subquery. See https://prestosql.io/blog/2019/05/30/semijoin-precomputed-hasd.html.
Improved smart query retry to support INSERT OVERWRITE TABLE, CREATE TABLE AS and SELECT queries which failed without returning any data. Tracking of query retries has been improved in command logs with Query Tracker links for retries.
Qubole supports Apache Ranger integration with Spark on Spark 2.4.0 and later versions. Beta, Via Support.
Spark 2.4.3 is generally available.
For a list of bugs fixed between QDS versions R56 and R57, see Changelog for azure.qubole.com.