QDS Components: Supported Versions and Cloud Platforms

Supported Versions

The following table shows the currently supported open source versions of QDS components and the Cloud platforms on which they run.

QDS Component Currently Supported Versions Supported on (Cloud Platforms)
Airflow 1.10.0, 1.10.2QDS AWS, Azure, Oracle OCI, GCP
Cascading All AWS, Azure, Oracle OCI
Hadoop 2 2.6.0 AWS, Azure, Oracle OCI, GCP
Hive 1.2, 2.1.1, 2.3, 3.1.1 (beta) AWS
1.2, 2.1.1, 2.3 Azure, Oracle OCI
2.1.1, 2.3 GCP
Java
  • Hadoop 1 supports Java 1.7
AWS
  • Hadoop 2 and Spark support Java 1.7 as the default version, but 1.8 can be enabled through the node bootstrap.
AWS, Azure, Oracle OCI
  • Presto supports only 1.8
AWS, Azure
  • GCP supports only 1.8
GCP
MapReduce 0.20.1 AWS
2.6.0 AWS, Azure, Oracle OCI, GCP
Pig 0.11, 0.15, 0.17 (beta) AWS
0.11 Azure, Oracle OCI
Presto 0.180 (deprecated), 0.193, 0.208 AWS
0.157, 0.180, 0.193, and 0.208 Azure
0.208 GCP
Python

2.6, 2.7, and 3.5

Airflow supports only 2.7 and later

AWS

See Can I use Python 2.7 for Hadoop tasks? for more information

2.7 and 3.5 Azure, Oracle OCI
3.6.x and 3.7 GCP
R 3.3.3 AWS, Oracle OCI
3.3.2 Azure
3.5.2 GCP
RubiX 0.2.11 AWS, Azure
Scala 2.10 for Spark versions older than 2.0.0 AWS
2.11 for Spark 2.0.0 and later AWS, Azure, Oracle OCI, GCP

Spark: see

Spark Version Support

1.6.2, 2.0.2, 2.1.1, 2.2.0, 2.2.1, 2.3.1, 2.4.0, 2.4.3 AWS
2.0.2, 2.2.1, 2.3.1 Azure, Oracle OCI
2.3.2, 2.4 GCP

Note

In the Spark Version drop-down list on the Clusters page of the QDS UI, latest means the latest open-source maintenance version. If you choose latest, Qubole Spark is automatically upgraded when a new maintenance version is released. For example, if 2.x latest currently points to Spark 2.x.y, then when 2.x.(y+1) is released, QDS clusters running 2.x latest automatically start using 2.x.(y+1) on a cluster restart.

Sqoop 1.4.7 AWS
1.4.6 AWS, Azure, Oracle OCI
Tez 0.7, 0.8.4, 0.9.1 AWS
0.7, 0.8.4 Azure, Oracle OCI
0.8.4 GCP
Zeppelin (notebooks) 0.6.2 AWS, Azure, Oracle OCI, GCP

Deprecated Versions

The following table shows the deprecated versions of the engines and the corresponding timelines.

Note

  • Deprecation Timeline: Deprecated version is marked as (deprecated) on the UI. Existing clusters with this version function normally, however, new features and bug fixes are not available. It is not recommended for production use, therefore, you should upgrade the clusters to the later versions of engines. For assistance, contact Qubole Support.
  • Removal Timeline: Version in the removal timeline is completely removed from the cluster image. Existing clusters with this version function when they are in running state. After the next restart, these clusters come up with the default version on the available version list.
Engines on QDS Deprecated Versions Deprecation Timeline Removal Timeline
Spark 1.3.1, 1.4.0, 1.4.1, 1.5.0 2017 Yet to be removed. Estimated release: R57
1.5.1, 1.6.0, 1.6.1, 2.0.0, 2.1.0 R54 (November 2018) Yet to be removed. Estimated release: R57
Hive 0.13 Not supported for new customers. For legacy customers, support ended in R55. Removed in R55
1.2 R58 (February 2020) Yet to be removed. Estimated release: R58
Presto 0.142 R52 (January 2018) Removed in R57
0.157 R57 (October 2019) Yet to be removed. Estimated release: R58
0.180 Yet to be deprecated but this version has a deprecated label in the cluster UI in R57. Yet to be removed. Estimated release: R59
Airflow 1.8.2 R57 (October 2019) Yet to be removed. Estimated release: R58

Spark Version Support

QDS supports two major versions of Spark, 1.x and 2.x. As of April 2017, Qubole began phasing out support for older versions. Those versions are marked deprecated in the drop-down list of available versions on the Clusters page of the QDS UI. You can still launch clusters running a version marked deprecated, but:

  • No new features or bug fixes will be applied to this version.
  • The version will no longer be eligible for Qubole Customer Support; tickets will not be addressed.