Hadoop 2

Configuring Buffer Capacity

HADTWO-2000: A Hadoop 2 (Hive) or Spark cluster can be configured to have a specific buffer capacity. Set yarn.cluster_start.buffer_nodes.count to the number of nodes to be used for the buffer and pass it as a Hadoop override for the cluster. You can also let QDS maintain the buffer capacity by settting yarn.autoscaling.buffer_nodes.count.is_dynamic=true as a Hadoop override. Disabled | Cluster Restart Required

This buffer capacity will remain free for the lifetime of the cluster, except when the cluster reaches or exceeds its configured maximum number of nodes. The advantage of configuring buffer capacity is that a new command can run immediately without needing to wait for the cluster to upscale. For more information, see the documentation.

Bug Fixes

  • HADTWO-1942: Fixes a problem that caused Hadoop jobs to fail because of file permission checks when the job was submitted. QDS now skips file permission checks in the job submission phase if the staging directory of a MapReduce job is in an Oracle filesystem.
  • HADTWO-2094: The oci-hdfs-connector has been upgraded to version
  • HADTWO-2204: A NullPointerException can be caused by an attempt to update the scheduler node resource if the resource has already been removed as a result of an asynchronous event.