What’s New in QDS on Azure

This section calls out the most significant changes and improvements. For more information see:

Changes in Package Management

Package Management has been improved in the following ways: BetaImageP

  • The Package Management UI supports uninstalling packages.
  • The Control Panel of the QDS UI supports configuring account-level ACLs for package management.
  • ACLs can be configured at the object level for a Package Management environment through both the UI and the API.
  • You can delete an environment.

Read more here.

Changes in Cluster Management

These improvements are GA1.

  • You can configure a custom Azure role for the application that launches QDS clusters.
  • QDS clusters now support B Series, M Series, L Series, Ev3 Series, and Dv3 series instance types.
  • QDS clusters now support NCv2 and ND series GPU instances.
  • QDS is supported in the Azure Netherlands region.
  • QDS supports soft enforcement of cluster permissions at the object level. When you select one permission iIn the Manage Permissions dialog for a cluster, additional cluster permissions are automatically selected. You can disable those additional permissions if you decide to, but Qubole strongly recommends you accept them. See the documentation for more information.

Changes in Data Management

These improvements are GA.

  • QDS clusters support data-at-rest encryption (Azure Key Vault support).
  • QDS provides T-SQL support for queries against a data store. This allows PolyBase data transfers from Azure storage to an SQL Data Warehouse after Qubole ETL.
  • The problem of output write failures after partial file writes to object storage has been fixed.

Changes in Hive

These improvements are GA2.

  • Running Hive 2.1 on the Master node (Hive on Master) is still supported but is no longer the default.
  • QDS supports Hive metadata caching.

Changes in Notebooks

These enhancements are GA3.

  • Existing Spark interpreters have been made compact. Properties whose values are not explicitly overridden are removed from the interpreters and the values of those properties are set to the cluster defaults.
  • For help in debugging TTransport exceptions, a hyperlink to the FAQ that contains the solution has been added in the paragraph output.
  • The UI provides better scrolling while a notebook is being loaded.

Changes in Presto

  • QDS supports Presto 0.193 on Presto clusters OpenBeta.

    Improvements include autoscaling; user-defined functions (UDFs); a faster Rubix client; and the Kinesis connector.

Changes in Spark

  • QDS now supports Spark 2.2.1; this version is reflected as 2.2 latest (2.2.1) in the Spark cluster UI. All 2.2.0 clusters are automatically upgraded to 2.2.1 in accordance with Qubole’s Spark versioning policy.

Note

Spark 2.2.1 as the latest version will be rolled out in a patch after the R52 release.

See also New Beta Features (Support Ticket Required).

GDPR Readiness

As part of GDPR readiness, Qubole has updated its privacy policy. The updated privacy policy explains how Qubole will collect, use, disclose, and share personal data that belongs to QDS users.

As part of the Sign up process, when personal information such as name, email ID, company, title, location, and phone number is collected from a new user, he will be asked to give consent to Qubole’s Privacy Policy and Terms of Service, by selecting the associated checkbox before proceeding further in the Sign up process.

An existing user, during a fresh login, will be asked to provide consent to Qubole for processing his personal data by selecting the Privacy Policy and Terms of Service checkboxes. This will be required to successfully log into the QDS platform. The consent will be collected using a one-time pop-up which will not be shown to users during subsequent sign-ins. The same pop-up requesting for consent will be displayed when a user is invited to join QDS.