Understanding the YARN and HDFS Metrics for Monitoring (AWS)
Hadoop 2 (Hive) and Spark clusters support the Datadog monitoring service.
You can configure the Datadog monitoring service at the cluster level as described in Advanced configuration: Modifying Cluster Monitoring Settings.
For more information on configuring the Datadog monitoring service at the account level in Control Panel > Account Settings, see Configuring your Access Settings using IAM Keys or Managing Roles.
Qubole also provides a default dashboard on Datadog and alerts to monitor Hadoop 2 (Hive) clusters. Default Dashboard for YARN and HDFS Metrics describes a sample default dashboard.
If you want to customize the threshold values or alerts about other metrics, you can set such alerts/values. For information on how to create alerts and configure email notifications, see the Datadog Alerts description.
This section describes:
YARN Metrics
This table describes the YARN metrics that are sent to Datadog. Log in to the Datadog account to see these metrics.
Metric |
Description |
|---|---|
yarn.QueueMetrics.AppsCompleted |
It denotes the number of completed applications. |
yarn.QueueMetrics.AppsPending |
It denotes the number of pending applications. |
yarn.QueueMetrics.AppsRunning |
It denotes the number of running applications. |
yarn.QueueMetrics.AppsFailed |
It denotes the number of failed applications. |
yarn.QueueMetrics.AppsKilled |
It denotes the number of killed applications. |
yarn.QueueMetrics.ReservedMB |
It denotes the size of the reserved memory. |
yarn.QueueMetrics.AvailableMB |
It denotes the size of the available memory in Mebibytes. |
yarn.QueueMetrics.AllocatedMB |
It denotes the size of the allocated memory in Mebibytes. |
yarn.QueueMetrics.ReservedVCores |
It denotes the number of reserved virtual cores. |
yarn.QueueMetrics.AvailableVCores |
It denotes the number of available virtual cores. |
yarn.QueueMetrics.AllocatedVCores |
It denotes the number of allocated virtual cores. |
yarn.NodeManagerMetrics.ContainersFailed |
It denotes the number of containers that have failed. |
yarn.NodeManagerMetrics.ContainersRunning |
It denotes the number of running containers. |
yarn.NodeManagerMetrics.ContainersKilled |
It denotes the number of containers that are killed. |
yarn.NodeManagerMetrics.ContainersCompleted |
It denotes the number of containers that are completed. |
yarn.QueueMetrics.AllocatedContainers |
It denotes the number of allocated containers. |
yarn.QueueMetrics.ReservedContainers |
It denotes the number of reserved containers. |
yarn.ClusterMetrics.NumActiveNMs |
It denotes the number of active NodeManagers. |
yarn.ClusterMetrics.NumDecommissionedNM |
It denotes the number of decommissioned NodeManagers. |
yarn.ClusterMetrics.NumDecommissioningNMs |
It denotes the number of decommissioning NodeManagers. |
yarn.ClusterMetrics.NumLostNMs |
It denotes the number of NodeManagers that are lost. |
yarn.ClusterMetrics.NumRebootedNMs |
It denotes the number of rebooted NodeManagers. |
yarn.ClusterMetrics.NumUnhealthyNMs |
It denotes the number of unhealthy NodeManagers. |
HDFS Metrics
Metric |
Description |
|---|---|
dfs.FSNamesystem.CapacityTotal |
It denotes the total disk capacity in bytes. |
dfs.FSNamesystem.CapacityUsed |
It denotes the disk usage in bytes. |
dfs.FSNamesystem.CapacityRemaining |
It denotes the remaining disk space left in bytes. |
dfs.FSNamesystem.CapacityUsedGB |
It denotes the disk usage in Gigabytes. |
dfs.FSNamesystem.CapacityTotalGB |
It denotes the total disk capacity in Gigabytes. |
dfs.FSNamesystem.TotalLoad |
It denotes the total load on the file system. |
dfs.FSNamesystem.BlocksTotal |
It denotes the total number of blocks. |
dfs.FSNamesystem.FilesTotal |
It denotes the total number of files. |
dfs.FSNamesystem.MissingBlocks |
It denotes the number of missing blocks. |
dfs.FSNamesystem.CorruptBlocks |
It denotes the number of corrupt blocks. |
dfs.FSNamesystem.PendingReplicationBlocks |
It denotes the number of blocks pending replication. |
dfs.FSNamesystem.UnderReplicatedBlocks |
It denotes the number of under replicated blocks. |
dfs.FSNamesystem.ScheduledReplicationBlocks |
It denotes the number of blocks scheduled for replication. |
dfs.FSNamesystem.PendingDeletionBlocks |
It denotes the number of pending deletion blocks. |
Default Dashboard for YARN and HDFS Metrics
QDS provides a default dashboard with these metrics:
Apps
Containers
DFS Used Capacity
Here is a sample default dashboard that contains YARN/HDFS metrics.
