What is New¶
The new features and enhancements are listed in the section below.
The label (in blue) that is against the description indicates the launch state, availability, and the default state of the feature/enhancement. For more information, click the label.
Unless stated otherwise, features are generally available, available as self-service and enabled by default.
The metastore consistency check (MSCK) result is displayed only in Logs instead of the Results tab of the Analyze UI. Disabled
Qubole encrypts the Hive metastore passwords. Beta, Via Support
When Tez is the execution engine in Hive queries, QDS provides an account-level configuration to limit the number of AWS API calls. Beta
A Datadog dashboard for the Hive Metastore Server (HMS) is added for Hive, Spark, and Presto clusters. Beta
Presto 0.193 version is generally available. Detailed notes about the 0.193 release are available as a blog post at Introducing Presto 0.193 in QDS.
The Qubole Presto Server Bootstrap is an alternative to the Node Bootstrap Script to install external jars such as presto-udfs before the Presto Server is started. This prevents the additional restart of the Presto Server that happens when you install such jars through the Node Bootstrap Script which causes query failures during the node startup. It is only supported in Presto 0.180 and later versions. Disabled
Qubole Presto now supports providing full access to the S3 bucket owner for files written to the bucket by other users. Disabled
A new FastPath for queries executed by the QDS Drivers. The FastPath minimizes the QDS level time for Presto queries. It is only supported on Presto 0.180. With the Presto FastPath, the overall command latencies within QDS should come down to around 1.5 seconds when the cluster is active. Beta, Via Support
The file-based authentication feature has been enhanced to accept only the hashed passwords for each user. Earlier, the plain passwords were accepted and stored as-is on the cluster which posed a security threat. MD5, SHA1, Unix Crypt, and BCrypt hashed passwords are supported. Qubole recommends using MD5 and BCrypt as SHA1 and Unix Crypt are less secure. Learn more.
Support for Apache Spark 2.3.1, which is the latest version of Spark.
Qubole features and performance enhancements on Spark:
- Improvement in S3 listing performance Beta, Via Support, Disabled
- Disallow creation of Spark clusters with low memory instances Beta, Via Support, Disabled
- Integration of DDL commands with Snowflake through Spark
- Rolling and aggregation of Spark executor and driver logs into remote S3 Beta, Via Support, Disabled