Changelog for eu-central.qubole.com¶
|Date and time of release||Version||Change type||Change|
|25th Nov, 2019 (05:59 AM PST)||57.0.79||Bug fix||ACM-5991: The issue in which a cluster containing spot nodes in its composition returned
|21st Nov, 2019 (03:11 PM PST)||57.0.77||Enhancement||
AIR-404: The users can now provide custom on_failure_callback for QuboleCheckOperator just like QuboleOperator. It allows users to define a custom callback if the QuboleCheckOperator fails (Cluster Restart Required).
AD-2805: You can now validate an account using AWS simulation. This helps in identifying any missing permissions.
|New feature||PRES-3070: Presto version 317 (beta) is now available on the Presto cluster.|
PRES-2990: Improved efficiency of dynamic partition pruning by preventing listing and creation of Hive splits from partitions, which are pruned at runtime.
ZEP-493: Bitbucket is integrated with notebooks. You can use Bitbucket to manage versions of your notebooks. Learn more.
PRES-3051: It fixes the
PRES-3113: Improved accounting of queued work for calculation of optimal size in autoscaling.
PRES-3131: Qubole has added
PRES-3163: It fixes a bug which could cause some additional nodes to be added to the cluster for a short duration during spot loss.
PRES-3187: It fixes a bug that caused the
|14th Nov, 2019 (08:03 AM PST)||57.0.64||Enhancement||ACM-5958: Qubole has done improvements in cluster UI page’s loading to resolve the slow-loading cluster UI page.|
ZEP-4081: Notebooks were failing due to jar conflicts with Spark. Thrift version used by Zeppelin is upgraded to 0.9.3 to fix this issue.
ZEP-4119: Python environment for the Notebooks was not getting configured properly. This issue is fixed now.
|6th Nov, 2019 (02:28 AM PST)||57.0.52||R57 Upgrade|
|27th Sep, 2019 (04:36 AM PST)||56.0.158||Enhancement||PRES-2791: Qubole has ported open-source changes that are related to improvements in S3 reads to Qubole Presto 0.208 version. For more information, see Faster S3 reads.|
PRES-2961: In an IAM-Role-based account, the issue where incorrect IAM-Role info had been sent to Presto has been resolved now.
PRES-2999: The NullPointerException when local memory limits are exceeded and a leak in operator peak memory computations in Presto version 0.208 queries have been resolved now.
PRES-3009: The issue where the Presto coordinator disk was filling up due to presence of RubiX logs in the autoscaling
log file has been resolved.
|Bug fix||ACM-5664: Fixed the caching issue of the cluster list in the Analyze (Workbench (beta) UI page.|
|Bug fix||ACM-5616: Status checking of Spot requests has been made more robust by doing retries for DescribeSpotInstanceRequests in case of the RequestResourceCountExceeded error.|
|Bug fix||QHIVE-4798: Fixed an issue which can lead to a memory leak in HiveServer2 JVM when a large number of concurrent applications are running on a given cluster.|
|Bug fix||ACM-5689: Fixed an issue where the default Hadoop cluster did not start during the AWS test drive signup.|
ACM-5266: Qubole supports
ACM-5413: Qubole supports
ACM-5558: During provisioning of nodes on a running heterogeneous cluster, Qubole tries providing instance types set in the heterogeneous configuration before falling back to On-Demand instances. For more information, see Additional Permissions.
ACM-5532: During the scrub run that is removing dead nodes for a cluster, there were nodes that were unreachable. This resulted in performance issues. To resolve this issue, Qubole terminates such nodes that cannot be connected. Via Support.
HADTWO-2098: In some cases, the scrub run did not remove dead nodes from a cluster. It happened as the scrub run connected to dead nodes as well through SSH to check on the node bootstrap completion. To resolve this issue, Qubole has improved the scrub run process, which does not connect to a dead node registered in the ResourceManager. This helps removing dead nodes that remained in the cluster and prevents such dead nodes from remaining in the cluster unlike before.
|22 Aug, 2019 (5:40 AM PST)||56.0.120||Enhancement||
JUPY-1: Qubole provides JupyterLab interface, which is the next generation user interface for Jupyter, to create and manage Jupyter notebooks. Jupyter notebooks are supported on Spark 2.2 and later versions. Beta, Via Support. Learn more.
AN-2168: New Analyze is now Workbench. Beta.
AN-1327: To make debugging easier, Qubole now displays the Cluster Instance ID under the Processing tab of the Status pane. This enables you to collect logs of the particular command by cluster instance. Beta.
AN-2210: You can now tag commands on the History tab in Workbench. You can later use these tags to filter out commands using the Tags field (in the history filter). Beta.
|Bug fix||AN-2219: Resource links are now clickable in the Logs pane in Workbench. Clicking the link redirects the user to the corresponding cluster dashboard.|
|06 Aug, 2019 (6:50 AM PST)||56.0.108||Enhancement||AIR-390: Now Airflow is supported on New Package Management. New Package Management brings in features like Python 3.7, a new version of Conda, and a lot of fixes on package installation such as support for no-arch packages, and so on (Cluster Restart Required, Disabled, Via Support).|
|21st Aug, 2019 (4:47) AM PST)||56.0.117||Enhancement||ZEP-3373: Users can now edit the notebooks even when the clusters are offline. Via Support|
|Bug fix||ACM-5344: Qubole supports configuring proxy setting of Internet proxy server and no_proxy settings for cluster nodes. To override them, contact Qubole Support. Via Support|
|9th August, 2019 (1:17 AM PST)||56.0.111||Bug fix||ACM-5171: Fixed the issue where a multi-instance HS2-enabled Hadoop (Hive) cluster, the multi-instance HS2 intermittently failed to start at the first attempt while waiting for the Hadoop (Hive) cluster.|
|06 Aug, 2019 (11:56 AM PST)||56.0.108||Enhancement||
ZEP-2717 and ZEP-3602: The Environments UI is now available in the Control Panel by default for the new users.
Limitation: The packages that are installed by default cannot be uninstalled in new version of Package Management.
|1st August, 2019 (12:42 AM PST)||56.0.102||Enhancement||AD-2476: To mitigate risk while rolling out a new feature, Qubole groups users, and accounts into different pods. When a change is rolled out to a pod, Qubole monitors the feature’s performance before rolling it out to subsequent pods. You can view the pod you belong to in the Account Details section of the Account Settings tab.|
|19th July, 2019 (7:56 AM PST)||56.0.92||Enhancement||PRES-2918: A new experimental configuration property called
PRES-2746: S3 buckets were inaccessible when Dual IAM Roles were configured on the Qubole account. To resolve this issue,
Presto has added
PRES-2797: Fixed the issue in the generated Presto Query Tracker when Presto version was changed on an active Presto cluster.
PRES-2856: Fixed the issue in which command results were displayed without column headers for Presto queries when Qubole drivers executed such queries in Presto FastPath.
PRES-2915: Fixed the issue in which a Presto cluster with idle cluster timeout configuration did not automatically terminate even when it was idle for a longer time.
|Enhancement||Hive version 2.3 is generally available.|
QHIVE-4385: To resolve FileNotFoundException while calculating FileSplits for the ORC file format, Qubole has added retries
in the Tez AM configuration. You can configure number of retries by using
|Enhancement||ACM-4221 and ACM-5016: Qubole supports Hive 3.1.1 (beta) on a Hive cluster. Starting cluster API v2.1, Hadoop 2 (Hive) clusters are renamed as Hive clusters. You can set Hive 3.1.1 (beta) version while creating/editing a cluster. Qubole supports creating Hive clusters only from cluster API v2.1 onwards. Via Support|
|13th June, 2019 (1:17 PM PST)||56.0.76||Enhancement||
AN-1639: The Status pane on the new Analyze page is now available for Quantum commands.
AN-1708: Permalinks for Hive tables on the new Analyze page now contain account IDs. When you navigate to a Hive table from a different account, a confirmation dialog box appears.
QUEST-321: Users can now use custom code in Python language for creating a streaming pipeline.
QUEST-332: Users can select AVRO as one of the input formats when creating a streaming pipeline in assisted mode with Kafka as source.
QUEST-341: Users can add additional configurations such as user defined metadata for data written to S3 sink.
SPAR-3514: The Test Run option in the QuEST UI, processes a limited number of records and maintains separate temporary checkpoint location to prevent any corruption to the runtime production checkpoint.
SPAR-3591: Users can pass
AN-1974: Column headers now appear on the Analyze page for large Hive queries (approximately 65KB and more).
QUEST-340: When the users performed a test run with Kinesis as source and IAM role enabled in the account, the test run failed. This issue is now fixed.
QUEST-324: If the checkpoint location is not unique, the QuEST UI displayed an inaccurate notification when running the streaming pipeline in assisted mode. Now, the accurate notification is displayed in this case.
|10th June, 2019 (1:50 AM PST)||56.0.66||Enhancement||
AD-1629: Qubole now supports IAM role-based account creation via API for AWS. Learn more.
AN-2100: All file downloads on the Explore page are now handled through AWS Signature Version 4.
QHIVE-4527: Qubole will deprecate Hive notebooks in the near future.
AD-2554: This bug fix resolves the intermittent provided token has expired error for AWS.
AD-2441: The Usage Status Dashboard has been revamped.
QTEZ-440: It is the fix for the issue in which a Tez DAG got hung when exceptions were uncaught during a DAG transition.
|4th June, 2019 (1:10 AM PST)||56.0.61||Enhancement||
JDBC-124: Qubole now supports concurrency of multiple statements in Presto FastPath.
PRES-2254: In a Presto notebook, you can now set
PRES-2600: These are the new enhancements in the Presto notebooks:
PRES-2515: The issue which caused ArrayIndexOutOfBoundsException for some queries when using Dynamic Filtering is resolved now.
PRES-2727: This fixes the issue where queries with a table containing sub column names of a
PRES-2775: This fixes a bug which could prevent an upscaled Presto cluster from downscaling if short running queries are regularly scheduled on the cluster.
|29th May, 2019 (1:29 AM PST)||56.0.59||Major Release R56|
|21st May, 2019 (7:53 AM PST)||55.0.65||Enhancement||AN-1382: Click the preview icon to preview a query. You can also dock the query preview while you work on another query.|
|15th May, 2019 (7:38 AM PST)||55.0.61||Enhancement||SQOOP-242: A new enhancement provides a better focus on a functional add-on for SQOOP rather than moving prematurely to serverless model. If Pixie (Qubole Cluster) is not available for the account (due to a configuration), the user must use his own cluster. If Pixie is present on the environment, it is a default option for all the new customers to select their own cluster.|
|14th May, 2019 (1:43 AM PST)||55.0.59||Enhancement||AD-2413: Added functionality to view selected data buckets from AWS. This can be enabled only if you have provided the S3:ListBucket permission on individual buckets. The functionality is currently available for role-based accounts only. Contact Qubole Support to enable this for your QDS account.|
|Bug fix||AN-1974: The issue of the Analyze page not displaying results for large Hive queries (approximately 65KB and more) was resolved by falling back to object storage to retrieve query metadata.|
|New feature||ACM-4765: Qubole now supports m5ad and r5ad family instance types in US East (N. Virginia), US West (Oregon), US East (Ohio), and Asia Pacific (Singapore) AWS regions.|
|22nd April, 2019 (9:00 AM PST)||55.0.48||Bug fix||AN-2056: Uploading objects or files through the Qubole UI is now supported for all AWS regions. KMS-enabled buckets are also supported.|
|03rd April, 2019 (1:40 AM PST)||55.0.37||Enhancement||
PRES-1350: Qubole supports configuring the required number of worker nodes during autoscaling. It is a cluster configuration
PRES-2397: Qubole supports escaping newline
PRES-2417: Presto clusters do not terminate while actively running Presto notebook paragraphs. The enhancement is not available by default. Create a ticket with Qubole Support to use it.
PRES-2474: The optimization to speed up queries on
PRES-1373: This is a fix for an issue where the retry operation was not working on the failed Presto queries. Retries work on the failed Presto queries now. As part of the fix, retries are configurable for Presto queries that run on the QDS platform.
Configuring retries will just do a blind retry of a Presto query. This may lead to data corruption for non-Insert Overwrite Directory (IOD) queries.
PRES-2306: Fixed a bug where the Usage page on the QDS UI did not show the Presto query’s bytes read statistics.
PRES-2493: Fixed the data loss issue in the Hive connector when writing bucketed sorted table in Qubole Presto 0.208 queries.
PRES-2567: Fixed a bug in the user-overridden IAM Role feature where stale credentials were being used in a long-running cluster.
PRES-2571: Fixed a concurrency related issue in autoscaling on Qubole Presto 0.193 and 0.208 versions due to which the cluster’s size permanently exceeds its maximum size.
QHIVE-3740: The Hive metastore API call has been replaced with a less expensive API call to resolve the issue of false alerts that were triggered on Datadog.
QHIVE-4040: It is a fix for an issue in
QHIVE-4053: It is a fix for an issue where the
QHIVE-4280: Hive jobs were failing with a Metaexception that included a timeout message. To resolve this issue, the SSH configuration parameters’ maximum value is increased on QDS servers.
QTEZ-335: It resolves an issue where the Tez Offline UI was inaccessible when
|Bug fix||TOOLS-1139: The previous docker version had security vulnerabilities, which could allow malicious containers to
gain root-level privileges on the host. To resolve security vulnerabilities of the previous docker version, the
latest patch version of
|13th March, 2019 (7:30 AM PST)||55.0.22||Enhancement||SPAR-3005: For the offline Spark clusters, only the event log files that are less than 400 MB are processed in the offline Spark History Server (SHS). This prevents high CPU utilization on the internal servers due to SHS.|
|Bug fix||SPAR-3371: Dynamic filtering now works for join keys of type Int, String, Long, and Short.|
|13th March, 2019 (7:30 AM PST)||55.0.22||Bug fix||
ZEP-3247: Zeppelin failed to start due to the
ZEP-3252: On a cluster restart, the latest dashboards are displayed after the scheduled runs.
ZEP-3230: Zeppelin server in Java 7 clusters was failing due to some ciphers that were disabled for security. Now, the Zeppelin server starts successfully as the required ciphers are added when starting the Zeppelin server.
ZEP-3243: A notebook with the ERROR status on a cluster failed to associate back to the cluster even after switching to another cluster. This issue is fixed.
ZEP-3234: Notebooks failed to load in case of higher latencies when fetching notebook permissions. This issue is fixed.
|11th March, 2019 (12:30 AM PST)||55.0.17||R55 Upgrade|
|7th January, 2019 (10:54 PM PST)||54.0.46||Bug fix||SPAR-3207: The Spark shuffle cleanup feature is not supported in Spark 2.2.1. However, the Spark shuffle cleanup feature continues to be supported in Spark 2.3.1 and later versions.|
|20th December, 2018 (2:30 PM PST)||54.0.30||Enhancement||
SPAR-3151: Qubole now supports the latest Apache Spark 2.4.0 version. It is displayed as 2.4 latest (2.4.0) in the Spark Version field of the Create New Cluster page on QDS UI.
|6th December, 2018 (4:10 PM PST)||54.0.38||Bug fix||EAM-1502: Fixed an issue in the Automatic Statistics Collection framework due to which the fresh statistics were not collected for a few tables.|
|29th November, 2018 (4:21 AM PST)||54.0.33||Enhancement||PRES-2380: Presto notebooks are GA and Presto 0.208 (beta) version is available for use.|
PRES-2338: Fixed a bug in Presto 0.180 and 0.193 to adhere to fallback to the On-Demand configuration in case of Spot loss.
PRES-2268: Fixed a bug in Presto 0.193 which caused SELECT queries with THE ROW type output to fail.
PRES-2269: Fixed a bug in Presto 0.193 due to which queries on Hive views from Presto were failing.
|21st November, 2018 (9:42 AM PST)||54.0.26||Bug fix||
AD-1752: Bug fix to restrict the Service user from logging in via Gmail or SAML.
AD-1763: Bug fix to enable updating the feature status in an internal table.
|20th November, 2018 (4:53 AM PST)||54.0.23||
For the list of changes until 20th November, 2018, see Changelog for api.qubole.com.