What is New¶
The new features and enhancements are listed in the corresponding tabs below.
Note
The label (in blue) that is against the description indicates the launch stage, availability, and the default state of the feature/enhancement. For more information, click the label.
Unless stated otherwise, features are generally available, available as self-service and enabled by default.
- A new parameter, update_all_clusters_compute_creds, has been added to the Edit Accounts API (v1.2 and v2 versions). When you set this parameter to true, Qubole pushes new settings to all clusters and vice versa.
- OpenJDK8 is now supported as the the default JDK on the cluster AMI and QDS Control Plane.
Qubole allows you to choose spot block nodes for configuring autoscaling nodes. Beta, Via Support
Learn more on the Spot block configuration here.
Qubole supports m5ad and r5ad cluster instances. Learn more.
Hadoop 2 (Hive) clusters support multi-instance HiveServer2 configuration. Beta, Via Support
Learn more on the multi-instance HiveServer2 here.
Presto clusters now support heterogeneous nodes. Via Support
Learn more on Presto clusters with heterogeneous nodes here.
OpenJDK8 is now supported as the the default JDK on the cluster AMI and QDS Control Plane.
- When your AWS S3 bucket is hosted in a region that is different from Qubole’s Control Plane, the contents now load correctly.
Qubole launches QuEST, a Streaming Platform-as-a-service to serve real-time reporting use cases. Beta, Via Support.
While viewing a Show Command API, it displays the email address of the user (Command Author) who submitted the command.
Airflow 1.10.2. is now available on QDS. It brings all the new features available in Apache Airflow 1.10.2 and some major bug fixes (introduced before 5th Feb 2019).
The user is now able to add and check multiple Hive Table dependencies (like S3 file paths) in the Scheduler.
A new API is introduced to get the Hive table partitions and locations.
Qubole now supports user-specific, secure connections to external data stores such as Redshift and Snowflake. User can share his or her connections as a template with other users and groups. Disabled, Via Support
The users can now create different notification channels (such as Email, Slack, PagerDuty, and Webhook) to receive notifications on success or failure of various Qubole products (Scheduler, Quest, and so on). Currently, it is available for the Scheduler and Quest.
For detecting network performance related issues on worker nodes, two metrics are sent to the Ganglia server.
Learn more here.
Qubole supports using AWS Glue as the primary metastore in Hive. Via Support
Learn more on the AWS Glue metastore here.
Qubole supports using the AWS Glue sync agent with QDS clusters to sync Hive Metastore with AWS Glue. Via Support
Learn more on the AWS Glue sync agent here.
Qubole supports multi-instance HiveServer2 (HS2) in an Hadoop 2 (Hive) cluster. Beta, Via Support
Learn more on multi-instance HiveServer2 here.
Hive 3.1.1 (beta) is the latest version that Qubole supports. Beta, Via Support
Learn more on Hive 3.1.1 (beta) here.
YARN ATS version 1.5 for Tez is supported only in Hive versions 2.1.1 and 2.3.1 (beta). Via Support
Learn more on the ATS v1.5 here.
Pig version 0.17 is now only supported with Hive version 2.1.1 and Tez version 0.8.4.
Learn more about the complete list of enhancements and bug fixes.
Qubole launches a new version of the Notebooks UI with usability enhancements in the paragraph editor. Beta, Via Support.
Qubole provides Logan, a new data-driven log analyzer that filters unnecessary Spark driver logs to enhance the debugging experience.
Learn more about the complete list of enhancements and bug fixes.
AWS Glue is supported as a primary Hive metastore. Qubole also supports syncing Hive Metastore with the AWS Glue catalog. Via Support
Learn more on the AWS Glue metastore here.
Qubole has introduced dynamic sizing of Presto clusters based on resource groups. Via Support
Learn more on the dynamic sizing of clusters here.
Qubole supports configuring the required number of worker nodes during the cluster autoscaling. Learn more.
Presto 0.208 version is generally available now. Presto version 0.157 is marked as deprecated in the Clusters UI. Presto 0.193 is the default Presto version. Learn more.
Qubole has added a cluster configuration property for controlling the downscaling velocity in Presto clusters. Controlling the Downscaling Velocity in Clusters.
Support for JOIN reordering and JOIN distribution type determination based on table size. Beta
Learn more on the JOIN reordering and the JOIN distribution type here.
Presto clusters now support heterogeneous nodes. Via Support
Learn more on Presto clusters with heterogeneous nodes here.
Ranger integration with Presto is generally available now. Learn more.
Learn more about the complete list of enhancements and bug fixes.
Redshift connector is supported with Spark on Qubole.
AWS Glue is supported as a metastore for Spark on Qubole. Via Support, Cluster Restart Required.
Executor based broadcast is introduced in which values to be broadcasted are not collected on driver.
Learn more about the complete list of enhancements and bug fixes.
RocksDB state store is implemented to optimize the performance of stateful structured streaming jobs.
Amazon S3-SQS data source is supported.
Learn more about the complete list of enhancements and bug fixes.