Important new features and improvements are as follows.
Blue text next to a description in these Release Notes indicates the launch state, availability, and default state of the item. For more information, click the label. Unless otherwise stated, features are generally available, available as self-service (without intervention by Qubole support), and enabled by default.
- Better failure handling, retries. An OCI SDK upgrade now allows enhanced failure handling and retries.
- New oci:// storage namespace. Starting with this release (R54), QDS uses the new
oci://storage namespace instead of
oraclebmc://by default. Clusters that were started prior to R54 used
oraclebmc://namespace, so those older clusters must be restarted to use the
Other Important Improvements¶
- Spark Dynamic Filtering improves JOIN performance. Via Support. Disabled
Sparklensexperimental open-source tool is available on http://sparklens.qubole.net. Learn more.
- Proactive cleanup of shuffle block data allows faster downscaling of nodes. Learn more. Via Support. Disabled
- Autoscaling is enabled by default for Qubole Spark clusters. The default value for the maximum number of autoscaling nodes has been increased from 2 to 10 for a new Spark cluster.
- Large Spark SQL commands are now supported in the API and from the Analyze page of the QDS UI. Via Support. Disabled
- Spark commands of sub-type
command line, and
sqlnow support macros in a script file. Learn more. Via Support. Disabled
Deprecated Spark Versions as of R54: 1.5.1, 1.6.0, 1.6.1, 2.0.0, 2.1.0.
QDS continues to support Spark 1.6.2, and the latest maintenance versions of each minor version of Spark 2.x. See the Supported Versions page.
- QDS has a new Service user type. Beta, Via Support, Disabled
- Administrators can now allow Data Preview (for Hive tables) from the Manage Roles page of the QDS UO.
- QDS now allows you to set a maximum command concurrent limit percentage for all users of an account. Via Support, Disabled
- QDS now allows you to monitor the health of Airflow clusters using integrated Monit, and turn certain services on and off. Cluster Restart Required
- R54 provides Apache Ranger integration for Hive workloads to help security administrators define fine-grained data-access policies for users and groups.
- Security administrators can define and enforce RBAC policies across multiple QDS artifacts that contain data and metadata, such as commands, data stores connections, data previews, and results.