What is New
The new features and enhancements are listed in the corresponding tabs below.
Note
The label (in blue) that is against the description indicates the launch state, availability, and the default state of the feature/enhancement. For more information, click the label.
Unless stated otherwise, features are generally available, available as self-service and enabled by default.
QDS has introduced a new public API which you can use to reset your own authentication token. Account administrators can use this to reset the authentication token of all other users in the account (including system-administrators).
Accounts are now sorted alphabetically in the Account drop-down list.
The Account Settings tab now displays concurrent command limits at the account and user levels.
QDS now displays cluster tags for deleted clusters.
The Master/Worker terminology naming convention is now used in the QDS UI instead of the Master/Slave terminology that was used before.
QDS now supports
c5n
,m5a
, andr5a
instances. Cluster Restart Required
To monitor the running DAGs on Airflow, Prometheus is integrated for real-time monitoring and dashboard using Grafana (Cluster Restart Required). Via Support, Disabled
You can now monitor the metrics with your own setup of Prometheus.
Any changes to the DAG Explorer will now immediately reflect on the Airflow cluster after the changes are saved. Cluster Restart Required
Qubole Airflow clusters now support the Airflow REST API.
Apache Airflow v1.10.0 is now supported on QDS. While creating an Airflow cluster, you can run the new Airflow version using the Airflow Version drop-down on the cluster UI. Learn more
The Sqoop version has been upgraded to 1.4.7 (Cluster Restart Required). Via Support, Disabled
Hive 2.3 (beta) is available. You can configure it while creating a Hadoop 2 (Hive) cluster (Cluster Restart Required). Beta
Hadoop 2 (Hive) clusters now support Pig 0.17 version (Cluster Restart Required). Beta
Multi-column data from a Hive query result is displayed in the Results tab of the Analyze UI page. Via Support
A detailed log for a specific Hive query that is executed using HiveServer2 or Hive-on-coordinator is now available. Via Support
Dashboards can be exported as PNG, PDF, and HTML files. Via Support, Disabled.
Qubole has added a Presto Strict Mode for running expensive Presto queries (Cluster Restart Required). Beta, Disabled
Qubole has added the Apache Ranger integration with Presto for managing granular access control. Beta, Disabled
Amazon S3 Select is now integrated with Spark on Qubole, and works for CSV and JSON data sources and tables. This integration improves TPCDS query performance. Via Support.
Python UDF pushdown is optimized to improve join performance by pushing down UDF when the joined output is larger than individual tables. Learn more.
Multitenant Qubole Job History Server that serves log and history of Spark jobs run on terminated cluster is now upgraded to Spark 2.3. Learn more.
Spark on Qubole now supports Hive Admin commands to allow users to grant privileges such as (SELECT, UPDATE, INSERT and DELETE.) to other users or roles. Learn more.