Changelog for in.qubole.com

Date and time of release Version Change type Change
19th July, 2019 (2:41 PM PST) 56.0.92 Enhancement PRES-2918: A new experimental configuration property called experimental.reserved-pool-enabled is added to Presto version 0.208 to allow disabling Reserved Pool, which is used to prevent deadlocks when memory is exhausted in the General Pool by promoting the biggest query to Reserved Pool. However, only one query gets promoted to Reserved Pool and queries in General Pool get into the blocked state whenever it becomes full. To avoid this scenario, you can set experimental.reserved-pool-enabled to false for disabling Reserved Pool. For more information, see Disabling Reserved Pool.
Bug fix

PRES-2746: S3 buckets were inaccessible when Dual IAM Roles were configured on the Qubole account. To resolve this issue, Presto has added hive.s3-secondary-role-arn and hive.s3-secondary-role-extid. You can add the ARN and External ID of the secondary IAM Role (which is inaccessible to Qubole) in Hive catalog properties. For more information, see catalog/hive.properties.

PRES-2797: Fixed the issue in the generated Presto Query Tracker when Presto version was changed on an active Presto cluster.

PRES-2856: Fixed the issue in which command results were displayed without column headers for Presto queries when Qubole drivers executed such queries in Presto FastPath.

PRES-2915: Fixed the issue in which a Presto cluster with idle cluster timeout configuration did not automatically terminate even when it was idle for a longer time.

    Enhancement Hive version 2.3 is generally available.
Bug fix

QHIVE-4385: To resolve FileNotFoundException while calculating FileSplits for the ORC file format, Qubole has added retries in the Tez AM configuration. You can configure number of retries by using hive.qubole.handle.s3.stale.listing.retries.split.generation, which defaults to 10. This configuration minimizes query failures due to inconsistency in S3 listing.

This is an enhancement over QHIVE-3675, which handled FileNotFoundException that occurred while processing a specific FileSplit. Via Support

    Enhancement ACM-4221 and ACM-5016: Qubole supports Hive 3.1.1 (beta) on a Hive cluster. Starting cluster API v2.1, Hadoop 2 (Hive) clusters are renamed as Hive clusters. You can set Hive 3.1.1 (beta) version while creating/editing a cluster. Qubole supports creating Hive clusters only from cluster API v2.1 onwards. Via Support
13th June, 2019 (3:04 PM PST) 56.0.76 Enhancement

AN-1639: The Status pane on the new Analyze page is now available for Quantum commands.

AN-1708: Permalinks for Hive tables on the new Analyze page now contain account IDs. When you navigate to a Hive table from a different account, a confirmation dialog box appears.

QUEST-321: Users can now use custom code in Python language for creating a streaming pipeline.

QUEST-332: Users can select AVRO as one of the input formats when creating a streaming pipeline in assisted mode with Kafka as source.

QUEST-341: Users can add additional configurations such as user defined metadata for data written to S3 sink.

SPAR-3514: The Test Run option in the QuEST UI, processes a limited number of records and maintains separate temporary checkpoint location to prevent any corruption to the runtime production checkpoint.

Bug fix

AN-1974: Column headers now appear on the Analyze page for large Hive queries (approximately 65KB and more).

QUEST-340: When the users performed a test run with Kinesis as source and IAM role enabled in the account, the test run failed. This issue is now fixed.

QUEST-324: If the checkpoint location is not unique, the QuEST UI displayed an inaccurate notification when running the streaming pipeline in assisted mode. Now, the accurate notification is displayed in this case.

10th June, 2019 (2:26 AM PST) 56.0.66 Enhancement

AD-1629: Qubole now supports IAM role-based account creation via API for AWS. Learn more.

AN-2100: All file downloads on the Explore page are now handled through AWS Signature Version 4.

QHIVE-4527: Qubole will deprecate Hive notebooks in the near future.

Bug fix

AD-2554: This bug fix resolves the intermittent provided token has expired error for AWS.

AD-2441: The Usage Status Dashboard has been revamped.

QTEZ-440: It is the fix for the issue in which a Tez DAG got hung when exceptions were uncaught during a DAG transition.

4th June, 2019 (2:25 AM PST) 56.0.61 Enhancement

JDBC-124: Qubole now supports concurrency of multiple statements in Presto FastPath.

PRES-2254: In a Presto notebook, you can now set zeppelin.presto.stacktrace as an interpreter property for displaying stacktrace for certain errors.

PRES-2600: These are the new enhancements in the Presto notebooks:

  • You can now set session properties in Presto notebooks in a paragraph and run it. When set, these session properties are applicable to paragraphs in the notebook’s current session.
  • In Presto notebooks, for improving debugging experience, the source field is set as notebook_<notebook-name>_<notebook-id> and in the dashboards, the source field is set as dashboard_<dashboard-name>_<dashboard-id>_<source-note-id>. A source field is directly searchable in the Presto UI. For example, in the Presto UI, you can search a notebook by its name or ID to quickly filter queries, which are run from that specific notebook while debugging an error.
Bug fix

PRES-2727: This fixes the issue where queries with a table containing sub column names of a struct column type starting with numbers failed. Sub-column names of a struct column type can now start with numbers.

PRES-2775: This fixes a bug which could prevent an upscaled Presto cluster from downscaling if short running queries are regularly scheduled on the cluster.

28th May, 2019 (2:49 AM PST) 56.0.58 Major Release R56
15th May, 2019 (7:38 AM PST) 55.0.61 Enhancement SQOOP-242: A new enhancement provides a better focus on a functional add-on for SQOOP rather than moving prematurely to serverless model. If Pixie (Qubole Cluster) is not available for the account (due to a configuration), the user must use his own cluster. If Pixie is present on the environment, it is a default option for all the new customers to select their own cluster.
22nd April, 2019 (8:56 AM PST) 55.0.48 Bug fix AN-2056: Uploading objects or files through the Qubole UI is now supported for all AWS regions. KMS-enabled buckets are also supported.
03rd April, 2019 (2:15 AM PST) 55.0.37 Enhancement

PRES-1350: Qubole supports configuring the required number of worker nodes during autoscaling. It is a cluster configuration override, query-manager.required-workers. You can set it to denote the number of worker nodes that must be in the cluster before a query is scheduled to be run on the cluster. This enhancement is only supported with Presto 0.193 and later versions. For more information, see Configuring the Required Number of Worker Nodes.

PRES-2397: Qubole supports escaping newline \n and carriage return (\r) characters in data for correctly parsing on the QDS UI. This enhancement is not available by default and it is only supported with Presto 0.193 and later versions. Create a ticket with the Qubole Support to enable it on the QDS account.

PRES-2417: Presto clusters do not terminate while actively running Presto notebook paragraphs. The enhancement is not available by default. Create a ticket with Qubole Support to use it.

PRES-2474: The optimization to speed up queries on system.jdbc.tables with filter on a single table name. This speeds up the extract operation in Business Intelligence tools such as DBeaver, which query system.jdbc.tables with filter on a single table name.

Bug fix

PRES-1373: This is a fix for an issue where the retry operation was not working on the failed Presto queries. Retries work on the failed Presto queries now. As part of the fix, retries are configurable for Presto queries that run on the QDS platform.

Caution

Configuring retries will just do a blind retry of a Presto query. This may lead to data corruption for non-Insert Overwrite Directory (IOD) queries.

PRES-2306: Fixed a bug where the Usage page on the QDS UI did not show the Presto query’s bytes read statistics.

PRES-2493: Fixed the data loss issue in the Hive connector when writing bucketed sorted table in Qubole Presto 0.208 queries.

PRES-2567: Fixed a bug in the user-overridden IAM Role feature where stale credentials were being used in a long-running cluster.

PRES-2571: Fixed a concurrency related issue in autoscaling on Qubole Presto 0.193 and 0.208 versions due to which the cluster’s size permanently exceeds its maximum size.

Bug fix

QHIVE-3740: The Hive metastore API call has been replaced with a less expensive API call to resolve the issue of false alerts that were triggered on Datadog.

QHIVE-4040: It is a fix for an issue in DelimitedJSONSerde that caused NullPointerException when hive.qubole.escape.crlf.in.result was enabled.

QHIVE-4053: It is a fix for an issue where the hive.qubole.drop.table.metastore.client.socket.timeout configuration was not honored when a Hive query containing the DROP Table statement was run.

QHIVE-4280: Hive jobs were failing with a Metaexception that included a timeout message. To resolve this issue, the SSH configuration parameters’ maximum value is increased on QDS servers.

QTEZ-335: It resolves an issue where the Tez Offline UI was inaccessible when yarn.timeline-service.store-class was set to RollingLevelDBTimelineStore in the cluster’s Hadoop overrides.

27th March, 2019 (2:56 AM PST) 55.0.32 Enhancement SPAR-3005:For the offline Spark clusters, only the event log files that are less than 400 MB are processed in the offline Spark History Server (SHS). This prevents high CPU utilization on the internal servers due to SHS.
Bug fix SPAR-3391: Some paragraphs ran successfully but with a warning message during the first run. However, those paragraphs ran successfully without the warning on a re-run. This issue is fixed.
TOOLS-1139: The previous docker version had security vulnerabilities, which could allow malicious containers to gain root-level privileges on the host. To resolve security vulnerabilities of the previous docker version, the latest patch version of docker-18.06.1ce is installed.
SPAR-3371: Dynamic filtering now works for join keys of type Int, String, Long, and Short.

ZEP-3247: Zeppelin failed to start due to the interpreter.json._COPYING_ file in defloc. As a result, the notebooks failed to load. With this fix, Zeppelin starts successfully and the interpreter.json._COPYING_ file is not seen on defloc.

ZEP-3252: On a cluster restart, the latest dashboards are displayed after the scheduled runs.

ZEP-3230: Zeppelin server in Java 7 clusters was failing due to some ciphers that were disabled for securi Now, the Zeppelin server starts successfully as the required ciphers are added when starting the Zeppelin server.

ZEP-3243: A notebook with the ERROR status on a cluster failed to associate back to the cluster even after switching to another cluster. This issue is fixed.

ZEP-3234: Notebooks failed to load in case of higher latencies when fetching notebook permissions. This issue is fixed.

12th March, 2019 (4:04 AM PST) 55.0.17 R55 Upgrade
08th January, 2019 (1:18 AM PST) 54.0.50 Bug fix SPAR-3207: The Spark shuffle cleanup feature is not supported in Spark 2.2.1. However, the Spark shuffle cleanup feature continues to be supported in Spark 2.3.1 and later versions.
20th December, 2018 (2:30 PM PST) 54.0.30 Enhancement

SPAR-3151: Qubole now supports the latest Apache Spark 2.4.0 version. It is displayed as 2.4 latest (2.4.0) in the Spark Version field of the Create New Cluster page on QDS UI.

Learn more

6th December, 2018 (1:35 PM PST) 54.0.38 Bug fix EAM-1502: Fixed an issue in the Automatic Statistics Collection framework due to which the fresh statistics were not collected for a few tables.
29th Nov 2018 at (5:26 AM PST) 54.0.33 Enhancement PRES-2380: Presto notebooks are GA and Presto 0.208 (beta) version is available for use.
Bug fix

PRES-2338: Fixed a bug in Presto 0.180 and 0.193 to adhere to fallback to the On-Demand configuration in case of Spot loss.

PRES-2268: Fixed a bug in Presto 0.193 which caused SELECT queries with THE ROW type output to fail.

PRES-2269: Fixed a bug in Presto 0.193 due to which queries on Hive views from Presto were failing.

21st November, 2018 (1:29 AM PST) 54.0.26

Major release

For the list of changes until 21st November, 2018, see Changelog for api.qubole.com.