What is New

The new features and enhancements are listed in the corresponding tabs below.

Note

The label (in blue) that is against the description indicates the launch state, availability, and the default state of the feature/enhancement. For more information, click the label.

Unless stated otherwise, features are generally available, available as self-service and enabled by default.

Administration

Cluster Management

Data Engineering

Hive

Notebooks and Dashboards

Presto

Spark

• QDS has introduced a new public API which you can use to reset your own authentication token. Account administrators can use this to reset the authentication token of all other users in the account (including system-administrators).
• Accounts are now sorted alphabetically in the Account drop-down list.
• The Account Settings tab now displays concurrent command limits at the account and user levels.
• QDS now displays cluster tags for deleted clusters.

Learn more.

• The Master/Worker terminology naming convention is now used in the QDS UI instead of the Master/Slave terminology that was used before.
• QDS now supports c5n, m5a, and r5a instances. Cluster Restart Required

Learn more about the complete list of changes.

• To monitor the running DAGs on Airflow, Prometheus is integrated for real-time monitoring and dashboard using Grafana (Cluster Restart Required). Via Support, Disabled

  Learn more

• You can now monitor the metrics with your own setup of Prometheus.

• Any changes to the DAG Explorer will now immediately reflect on the Airflow cluster after the changes are saved. Cluster Restart Required

• Qubole Airflow clusters now support the Airflow REST API.

• Apache Airflow v1.10.0 is now supported on QDS. While creating an Airflow cluster, you can run the new Airflow version using the Airflow Version drop-down on the cluster UI. Learn more

• The Sqoop version has been upgraded to 1.4.7 (Cluster Restart Required). Via Support, Disabled

Learn more about the complete list of changes.

• Hive 2.3 (beta) is available. You can configure it while creating a Hadoop 2 (Hive) cluster (Cluster Restart Required). Beta

  Learn more.

• Hadoop 2 (Hive) clusters now support Pig 0.17 version (Cluster Restart Required). Beta

  Learn more.

• Multi-column data from a Hive query result is displayed in the Results tab of the Analyze UI page. Via Support

  Learn more.

• A detailed log for a specific Hive query that is executed using HiveServer2 or Hive-on-coordinator is now available. Via Support

  Learn more.

Learn more about the complete list of changes.

• Dashboards can be exported as PNG, PDF, and HTML files. Via Support, Disabled.

  Learn more.

• Qubole has added a Presto Strict Mode for running expensive Presto queries (Cluster Restart Required). Beta, Disabled

  Learn more.

• Qubole has added the Apache Ranger integration with Presto for managing granular access control. Beta, Disabled

  Learn more.

Learn more about the complete list of changes.

• Amazon S3 Select is now integrated with Spark on Qubole, and works for CSV and JSON data sources and tables. This integration improves TPCDS query performance. Via Support.

  Learn more.

• Python UDF pushdown is optimized to improve join performance by pushing down UDF when the joined output is larger than individual tables. Learn more.

• Multitenant Qubole Job History Server that serves log and history of Spark jobs run on terminated cluster is now upgraded to Spark 2.3. Learn more.
• Spark on Qubole now supports Hive Admin commands to allow users to grant privileges such as (SELECT, UPDATE, INSERT and DELETE.) to other users or roles. Learn more.