Hadoop 2

The new features and enhancements are:

Other enhancements and bug fixes are listed in:

Qubole Supports Configuring Buffer Capacity

HADTWO-2000: A Hadoop 2 (Hive) or Spark cluster can be configured to have some specified capacity as a buffer capacity. Set yarn.cluster_start.buffer_nodes.count to the number of nodes and pass it as a Hadoop override on the cluster to configure the buffer capacity. You can also let Qubole maintain the buffer capacity when you pass yarn.autoscaling.buffer_nodes.count.is_dynamic=true as a Hadoop override. Disabled | Cluster Restart Required

The cluster will always have this buffer (configured) capacity free throughout its lifetime except when the cluster size exceeds or reaches its configured maximum cluster size. The advantage of buffer capacity is that a new command need not have to wait for the cluster to upscale and it can immediately start running. For more information, see the documentation.

Qubole Supports Reserving Buffer Space in Application Masters on On-Demand Nodes

HADTWO-2149: Qubole supports reserving buffer space for ApplicationMaster (AM) on On-Demand nodes, which you can configure through Override Hadoop Configuration Variables under Hadoop Cluster Settings in the cluster UI’s Advanced Configuration. As the buffer space is reserved for AMs, no task containers will be run in this space. This feature is supported in Hadoop 2 (Hive) and Spark clusters. Disabled | Cluster Restart Required

AMs run for a longer period than containers. So, a spot loss can result in a job failure or multiple retries. You can avoid this by reserving buffer space that helps in scheduling AMs on On-Demand nodes providing more stability for running jobs. For more information, see the documentation.

Enhancements

  • HADTWO-1892: Qubole has enhanced Spot rebalancer to incorporate clusters with Spot blocks as well as Spot nodes when they are set as autoscaled nodes.

Bug Fixes

  • HADTWO-1942: The issue that cause Hadoop Job failure due to file permission checks in the job submission phase has been resolved. Qubole skips file permission checks in the job submission phase if staging directory of a MR job is S3 filesystem.

  • HADTWO-2101: The issue where users faced XML parsing errors while deleting objects using the s3a filesystem, has been resolved now. As part of the bug fix, configurable retries are added and the default number of retries is 1000.

For a list of bug fixes between versions R56 and R57, see Changelog for api.qubole.com.