QDS Components: Supported Versions and Cloud Platforms

The following table shows the currently supported open source versions of QDS Components, and the Cloud platforms on which they run.

Important

Qubole is discontinuing support for HBase. You can no longer create HBase clusters.

QDS Component Currently Supported Versions Supported on (Cloud Platforms)
Airflow 1.7.0 AWS
Cascading All AWS, Azure, Oracle BMC
Hadoop 1 0.20.1 AWS
Hadoop 2 2.6.0 AWS, Azure, Oracle BMC
HBase 1.0 AWS (support being discontinued)
Hive 0.13.1, 1.2, 2.1.1 AWS
1.2 Azure, Oracle BMC
Java
  • Hadoop 1 supports Java 1.7
AWS
  • Hadoop 2 and Spark supports Java 1.7 as the default version, but 1.8 can be enabled through the node bootstrap.
AWS, Azure, Oracle BMC
  • Presto supports only 1.8
AWS, Azure
MapReduce 0.20.1 AWS
2.6.0 AWS, Azure, Oracle BMC
Pig 0.11, 0.15 AWS
Presto 0.142 (to be deprecated), 0.157 (stable), and 0.180 (unstable) AWS
0.142 Azure
Python

2.6, 2.7

Airflow supports only 2.7 and later

AWS

See Can I use Python 2.7 for Hadoop tasks? for more information

2.7 Azure, Oracle BMC
R 3.3.3 AWS, Oracle BMC
3.3.2 Azure
RubiX 0.2.11 AWS
Scala 2.10 for Spark versions older than 2.0.0 AWS
2.11 for Spark 2.0.0 and later AWS, Azure, Oracle BMC
Spark: see Spark Version Support 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 2.0.0, 2.0.2, 2.1.0, 2.1.1 AWS
2.1.0, 2.0.2 Azure, Oracle BMC

Note

In Spark versions, Spark 2.x-latest implies that the version always points to the latest open-source maintenance version of 2.x. Once a new maintenance version is released, Qubole Spark versions would be auto upgraded to that version. For example, if 2.x-latest currently points to 2.x.y, then when 2.x.(y+1) is released, 2.x-latest version automatically starts using 2.x.(y+1) on a cluster restart.

Sqoop 1.4.2 AWS
1.4.6 AWS, Azure, Oracle BMC
Tez 0.7 AWS
Zeppelin (notebooks) 1.0 AWS, Azure, Oracle BMC

Spark Version Support

QDS supports two major versions of Spark, 1.x and 2.x. As of April 2017, Qubole began phasing out support for older versions, as explained in the sections that follow.

Note

In Spark versions, Spark 2.x-latest implies that the version always points to the latest open-source maintenance version of 2.x. Once a new maintenance version is released, Qubole Spark versions would be auto upgraded to that version. For example, if 2.x-latest currently points to 2.x.y, then when 2.x.(y+1) is released, 2.x-latest version automatically starts using 2.x.(y+1) on a cluster restart.

Explanation of Terms

Qubole will phase out support of older Spark versions in two stages: deprecation and sunset. Each Spark version can be in one of three states in terms of QDS support:

  • Active: Fully supported; no current timeline for obsolescence.
  • Deprecated: You can still launch clusters running this version, but:
    • No new features or bug fixes will be applied to this version.
    • The version will no longer be eligible for Qubole Customer Support; tickets will not be addressed.
    • The version will sunset 90 days from the deprecation date.
  • Sunset: The version is no longer available in the QDS UI, and no longer eligible for Qubole Customer Support. You can no longer launch clusters running this version. Qubole will make a best effort not to break compatibility with versions in this state, but compatibility is not guaranteed.

Spark 1.x Deprecation

For Spark major version 1.x, QDS is deprecating and will eventually sunset versions older than 1.6.2, starting April 14, 2017:

Spark Versions Deprecation Date Sunset Date
1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1 April 14, 2017 July 31, 2017