QDS Components: Supported Versions and Cloud Platforms

The following table shows the currently supported open source versions of QDS Components, and the Cloud platforms on which they run.

Important

Qubole is discontinuing support for HBase. You can no longer create HBase clusters.

QDS Component Currently Supported Versions Supported on (Cloud Platforms)
Airflow 1.7.0 AWS
Cascading All AWS, Azure, Oracle
Hadoop 1 0.20.1 AWS
Hadoop 2 2.6.0 AWS, Azure, Oracle
HBase 1.0 AWS (support being discontinued)
Hive 0.13.1, 1.2 AWS
1.2 Azure, Oracle
Java
  • Hadoop 1 supports Java 1.7
AWS
  • Hadoop 2 supports Java 1.7 as the default version, but 1.8 can be enabled through the node bootstrap.
AWS, Azure, Oracle
  • Spark and Presto support only 1.8
AWS, Azure. Oracle for Spark only
MapReduce 0.20.1 AWS
2.6.0 AWS, Azure, Oracle
Pig 0.11, 0.15 AWS
Presto 0.142 (to be deprecated), 0.157 (stable), and 0.174 (unstable) AWS
0.142 Azure
Python

2.6, 2.7

Airflow supports only 2.7 and later

AWS

See Can I use Python 2.7 for Hadoop tasks? for more information

2.7 Azure, Oracle
R 3.3.3 AWS, Oracle
3.3.2 Azure
RubiX 0.2.10 AWS
Scala 2.1.0 for Spark versions older than 2.0.0 AWS
2.1.1 for Spark 2.0.0 and later AWS, Azure, Oracle
Spark: see Spark Version Support 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 2.0.0, 2.0.2, 2.1.0 AWS
2.1.0, 2.0.2 Azure, Oracle
Sqoop 1.4.2 AWS
1.4.6 AWS, Azure, Oracle
Tez 0.7 AWS
Zeppelin (notebooks) 1.0 AWS, Azure, Oracle

Spark Version Support

QDS supports two major versions of Spark, 1.x and 2.x. As of April 2017, Qubole began phasing out support for older versions, as explained in the sections that follow.

Explanation of Terms

Qubole will phase out support of older Spark versions in two stages: deprecation and sunset. Each Spark version can be in one of three states in terms of QDS support:

  • Active: Fully supported; no current timeline for obsolescence.
  • Deprecated: You can still launch clusters running this version, but:
    • No new features or bug fixes will be applied to this version.
    • The version will no longer be eligible for Qubole Customer Support; tickets will not be addressed.
    • The version will sunset 90 days from the deprecation date.
  • Sunset: The version is no longer available in the QDS UI, and no longer eligible for Qubole Customer Support. You can no longer launch clusters running this version. Qubole will make a best effort not to break compatibility with versions in this state, but compatibility is not guaranteed.

Spark 1.x Deprecation

For Spark major version 1.x, QDS is deprecating and will eventually sunset versions older than 1.6.2, starting April 14, 2017:

Spark Versions Deprecation Date Sunset Date
1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1 April 14, 2017 July 31, 2017