What is New

The new features and enhancements are listed in the section below.

Note

The label (in blue) that is against the description indicates the launch state, availability, and the default state of the feature/enhancement. For more information, click the label.

Unless stated otherwise, features are generally available, available as self-service and enabled by default.

Cluster Management

• Qubole supports the x1e instances.

• QDS supports the AWS Canada and AWS London Regions.

  Learn more.

• QDS has added the cluster health monitoring enhancements that are available through the Datadog monitoring service. Learn more. Beta

Hive

• The metastore consistency check (MSCK) result is displayed only in Logs instead of the Results tab of the Analyze UI. Disabled

• Qubole encrypts the Hive metastore passwords. Beta, Via Support

• When Tez is the execution engine in Hive queries, QDS provides an account-level configuration to limit the number of AWS API calls. Beta

• A Datadog dashboard for the Hive Metastore Server (HMS) is added for Hive, Spark, and Presto clusters. Beta

  Learn more.

Presto

• Presto 0.193 version is generally available. Detailed notes about the 0.193 release are available as a blog post at Introducing Presto 0.193 in QDS.

• The Qubole Presto Server Bootstrap is an alternative to the Node Bootstrap Script to install external jars such as presto-udfs before the Presto Server is started. This prevents the additional restart of the Presto Server that happens when you install such jars through the Node Bootstrap Script which causes query failures during the node startup. It is only supported in Presto 0.180 and later versions. Disabled

  Learn more on the Qubole Presto Server Bootstrap.

• Qubole Presto now supports providing full access to the S3 bucket owner for files written to the bucket by other users. Disabled

• A new FastPath for queries executed by the QDS Drivers. The FastPath minimizes the QDS level time for Presto queries. It is only supported on Presto 0.180. With the Presto FastPath, the overall command latencies within QDS should come down to around 1.5 seconds when the cluster is active. Beta, Via Support

  Learn more.

• The file-based authentication feature has been enhanced to accept only the hashed passwords for each user. Earlier, the plain passwords were accepted and stored as-is on the cluster which posed a security threat. MD5, SHA1, Unix Crypt, and BCrypt hashed passwords are supported. Qubole recommends using MD5 and BCrypt as SHA1 and Unix Crypt are less secure. Learn more.

Spark

• Support for Apache Spark 2.3.1, which is the latest version of Spark.

• Qubole features and performance enhancements on Spark:

  • Improvement in S3 listing performance Beta, Via Support, Disabled

  • Disallow creation of Spark clusters with low memory instances Beta, Via Support, Disabled

  • Integration of DDL commands with Snowflake through Spark

  • Rolling and aggregation of Spark executor and driver logs into remote S3 Beta, Via Support, Disabled

  Learn more.

Data Analytics

• Partition columns are now marked distinctly on the Analyze page’s Tables tab.

• You can now view and edit Hive commands greater than 126KB (which were earlier uploaded to Amazon S3) from the UI. Beta, Via Support, Disabled

  Learn more.

Data Engineering

• For Workflow commands, you can now download subcommand logs as well as results from both the UI and the API. This allows you to get logs and results for subcommands that were run even if the command failed as a whole. Beta, Via Support, Disabled

  Learn more.