Hadoop 2

Configuring Buffer Capacity

HADTWO-2000: A Hadoop 2 (Hive) or Spark cluster can be configured to have a specific buffer capacity. Set yarn.cluster_start.buffer_nodes.count to the number of nodes to be used for the buffer and pass it as a Hadoop override for the cluster. You can also let QDS maintain the buffer capacity by settting yarn.autoscaling.buffer_nodes.count.is_dynamic=true as a Hadoop override. Disabled | Cluster Restart Required

This buffer capacity will remain free for the lifetime of the cluster, except when the cluster reaches or exceeds its configured maximum number of nodes. The advantage of configuring buffer capacity is that a new command can run immediately without needing to wait for the cluster to upscale. For more information, see the documentation.