Understanding the Spark Metrics for Monitoring (AWS)¶
Spark clusters support Datadog monitoring when the Datadog monitoring is enabled at the QDS account level. You can configure Datadog settings at the cluster level for a Spark cluster as described in Advanced configuration: Modifying Cluster Monitoring Settings.
The following table lists the different Spark metrics that are displayed in the Datadog account.
All metrics are aggregated from all the spark applications that are running on the cluster.
|JVM Heap Usage
|driver.jvm.heap.used, driver.jvm.heap.max, driver.jvm.heap.committed
|Driver jvm’s current memory usage of the heap that is used for object allocation. The heap consists of one or more memory pools. The used and committed size is the sum of those values of all heap memory pools.The used value is the amount of memory occupied by both live objects and garbage objects that have not been collected, if any.
|Executor jvm’s heap usage.
|Sum of corresponding metrics values from jvm.heap and jvm.non-heap.
|BlockManager Memory Status
|Sum of maximum memory available for all the block managers.
|Sum of memory used by caching RDDs + non RDD memory storage for all the block managers. memUsed_MB or maxMem_MB >
``spark.qubole.autoscaling.memorythreshold`` is used to upscale
|Sum of (maxMem - memUsed) across all the block managers.
|Sum of disk space used by RDDs + non RDD disk storage for all the block managers.
|Sum of all known executors running.
|Total executors including the following value: needed = (max (jobProgressExecutorsNeeded, memoryBasedExecutorsNeeded) + executors already running) , calculated based on dynamic allocation strategy.
|Number of executors that are requested to be killed.
|Sum of all the job Ids that were submitted for an application.
|Total number of jobs ids for which ActiveJob was created successfully after asserting that the job had valid rdds and partitions to work on and non zero tasks to run, and stages (ResultStage and its parent ShuffleMapStages dependencies) that were created successfully for the job id.
|Stages that should be resubmitted due to fetch failures.
|Stages that are running.
|Stages that are yet to run and waiting for the parent stages to complete.
|Stages that are successfully submitted.
|Total of completed stages that have been run.
|Total of all the stages waiting to be submitted.
|Sum of tasks across all the active stages.
|Sum of active tasks from latest attempt of all active stages.
|Sum of completed tasks from latest attempt of all active stages.
|Sum of pending tasks = ( totalTasks - (completedTasks + activeTasks) ) from latest attempt of all active stages.
|Sum of SQL tasks that are running.