What’s New

Important new features and improvements are as follows.

Note

Blue text next to a description in these Release Notes indicates the launch state, availability, and default state of the item. For more information, click the label. Unless otherwise stated, features are generally available, available as self-service (without intervention by Qubole support), and enabled by default.

Oracle-Specific Improvements

Better failure handling, retries. An OCI SDK upgrade now allows enhanced failure handling and retries.
New oci:// storage namespace. Starting with this release (R54), QDS uses the new oci:// storage namespace instead of oraclebmc:// by default. Clusters that were started prior to R54 used oraclebmc:// namespace, so those older clusters must be restarted to use the oci:// namespace.

Other Important Improvements

Hive

Hive 2.1 is now generally available. Cluster Restart Required
QDS now uses HAProxy on the cluster coordinator node to balance the load when there are multiple connections between the cluster and a QDS-managed Hive Metastore. Learn more. Via Support

Presto

Presto Notebooks are now generally available. Learn more.
The latest supported version is Presto 0.208. Learn more. Beta. Cluster Restart Required

Spark

Spark Dynamic Filtering improves JOIN performance. Via Support. Disabled
The Sparklens experimental open-source tool is available on http://sparklens.qubole.net. Learn more.
Proactive cleanup of shuffle block data allows faster downscaling of nodes. Learn more. Via Support. Disabled
Autoscaling is enabled by default for Qubole Spark clusters. The default value for the maximum number of autoscaling nodes has been increased from 2 to 10 for a new Spark cluster.
Large Spark SQL commands are now supported in the API and from the Analyze page of the QDS UI. Via Support. Disabled
Spark commands of sub-type scala, python, R, command line, and sql now support macros in a script file. Learn more. Via Support. Disabled

Deprecated Spark Versions as of R54: 1.5.1, 1.6.0, 1.6.1, 2.0.0, 2.1.0.

QDS continues to support Spark 1.6.2, and the latest maintenance versions of each minor version of Spark 2.x. See the Supported Versions page.

Notebooks

You can see the cluster status on the Notebooks page. Learn more. Beta. Via Support. Disabled

Administration

QDS has a new Service user type. Beta, Via Support, Disabled
Administrators can now allow Data Preview (for Hive tables) from the Manage Roles page of the QDS UO.

Learn more.

Data Analytics

QDS now allows you to set a maximum command concurrent limit percentage for all users of an account. Via Support, Disabled

Learn more.

Data Engineering

Airflow

QDS now allows you to monitor the health of Airflow clusters using integrated Monit, and turn certain services on and off. Cluster Restart Required

Learn more.

Security

R54 provides Apache Ranger integration for Hive workloads to help security administrators define fine-grained data-access policies for users and groups.
Security administrators can define and enforce RBAC policies across multiple QDS artifacts that contain data and metadata, such as commands, data stores connections, data previews, and results.

Learn more.