What is New

The new features and enhancements are listed in the section below.

Note

The label (in blue) that is against the description indicates the launch state, availability, and the default state of the feature/enhancement. For more information, click the label.

Unless stated otherwise, features are generally available, available as self-service and enabled by default.

Cluster Management

  • Qubole now supports c5d, m5d, r5, r5d, and z1d instances. Learn more.

  • QDS has introduced a new cluster console providing granular visibility, enhanced governance and ease of use with features such as an ability to track a specific cluster’s activity, view history of its configuration changes, cluster usage, and instances. Learn more. Beta

  • QDS supports configuring multiple subnets in a VPC when clusters are configured with all compositions including the heterogeneous clusters. Learn more. Cluster Restart Required

Hive

  • Hive 2.1 version is now generally available. Learn more. Cluster Restart Required

  • Qubole has added HA Proxy in the cluster coordinator node for balancing the load when there are multiple connections between the cluster and the Qubole-managed Hive Metastore. This removes a single point of failure and provides more stability. Learn more. Via Support

Hadoop 1

Hadoop 1 as-a-service is deprecated from this QDS version. Qubole will support Hadoop 1 on existing clusters until 31 December 2018. Creating a new Hadoop 1 cluster or cloning an existing Hadoop 1 cluster is not supported through the API and the UI. Learn more.

Presto

Spark

Deprecated Spark Versions

In this release, the following Spark versions are deprecated: 1.5.1, 1.6.0, 1.6.1, 2.0.0, and 2.1.0. Qubole will continue to support Spark 1.6.2 and latest maintenance versions of each minor version in Spark 2.x. See version support documentation.

Spark Structured Streaming

  • Qubole provides comprehensive support for Kinesis connector in Structured Streaming.

    • Support to read from Kinesis Source in micro-batch streaming and continuous streaming modes.

    • Support to ingest data into Kinesis.

    • Support for IAM roles in Kinesis Connector.

    Learn more

  • Support to write a structured streaming query to a Spark data source table. Learn more.

  • Streaming query progress graphs are now displayed in notebooks. Learn more. Via Support Disabled

  • Direct writes for checkpointing is supported. Learn more. Via Support Disabled

Notebooks

Administration

  • Administrators can now track usage on Qubole using the QCUH dashboard. Beta

  • Qubole has introduced a new Service user type. Beta, Via Support, Disabled

  • Administrators can now configure an additional resource, Data Preview, on HIVE tables while managing resources on the Manage Roles page.

Learn more.

Data Analytics

  • QDS has introduced a functionality where customers can request setting a maximum command concurrent limit percentage for all users of an account. Via Support, Disabled

Learn more.

Data Engineering

Explore

  • Qubole has enabled support for AWS Aurora-MySQL RDS as a database backend for Airflow contents.

Airflow

Learn more.

Security

  • Apache Ranger integration for Hive workloads to help security administrators in defining fine-grained data access policies across users and user groups.

  • Security administrators can define and enforce RBAC policies across multiple Qubole artifacts that contain data and metadata such as commands, data stores connections, data previews, and results.

Learn more.