Understanding the HiveServer2 Metrics

HiveServer2 is available on Hadoop2 (Hive) clusters. You can enable HS2 on a Hadoop 2 (Hive) cluster as described in Configuring a HiveServer2 Cluster.

QDS supports Datadog monitoring for HS2. You can configure the Datadog monitoring service at the cluster level as described in Advanced configuration: Modifying Cluster Monitoring Settings.

For more information on configuring the Datadog monitoring service at the account level in Control Panel > Account Settings, see Configuring your Access Settings using IAM Keys or Managing Roles.

Qubole also provides a default dashboard on Datadog and alerts to monitor HS2 metrics. If you want to customize the threshold values or alerts about other metrics, you can set such alerts/values. For information on how to create alerts and configure email notifications, see the Datadog Alerts description.

This section describes:

Metrics associated with the Query Lifecycle Execution

Metric

Description

hive.hs2.waiting_compile_ops.count

It denotes the number of queries waiting to be compiled on HS2.

hive.hs2.submitted_queries.count

It denotes the number of queries submitted on HS2.

hive.hs2.compiling_queries.count

It denotes the number of queries being compiled on HS2.

hive.hs2.executing_queries.count

It denotes the number of queries being executed on HS2.

hive.hs2.failed_queries.count

It denotes the number of queries failed on HS2.

hive.hs2.succeeded_queries.count

It denotes the number of queries succeeded on HS2.

Metrics associated with Active Queries

Metric

Description

active_calls_hive.hs2.executing_queries

It denotes the number of queries currently executed on HS2.

active_calls_hive.hs2.compiling_queries

It denotes the number of queries currently compiled on HS2.

active_calls_hive.hs2.submitted_queries

It denotes the number of queries currently submitted on HS2.

Metrics associated with Execution Engines

Metric

Description

hive.hs2.mapred_tasks.count

It denotes the number of queries that have run with MapReduce as the execution engine.

hive.hs2.tez_tasks.count

It denotes the number of queries that have run with Tez as the execution engine.

Metrics associated with HS2 Sessions

Metric

Description

hive.hs2.session.open_sessions

It denotes the number of open sessions on HS2.

hive.hs2.session.active_sessions

It denotes the number of active sessions on HS2.

hive.hs2.session.avg_open_session_time

It denotes the average session time of open sessions on HS2.

hive.hs2.session.avg_active_session_time

It denotes the average session time of active sessions on HS2.

Metrics associated with the Memory

Metric

Description

hive.hs2.memory.total_used

It denotes the total memory used by HS2.

hive.hs2.memory.heap_used

It denotes the total heap memory used by HS2.

hive.hs2.memory.pools_CMS-Perm-Gen_usage

It denotes the total Permanent Generation (PermGen) used by HS2.

Metrics associated with the Garbage Collection

Metric

Description

hive.hs2.gc.ConcurrentMarkSweep_count

It denotes the number of ConcurrentMarkSweep GC events.

hive.hs2.gc.ConcurrentMarkSweep_time

It denotes the time taken by ConcurrentMarkSweep GC events.

hive.hs2.gc.ParNew_count

It denotes the number of ParNew GC events.

hive.hs2.gc.ParNew_time

It denotes the time taken by ParNew GC events.

Default Dashboard for HS2 Metrics

The default dashboard that is available on Datadog contains these HS2 metrics:

  • HS2 GC Time

  • HS2 Memory Usage

  • Active Queries

  • HS2 Session Details

  • Query Counters

Here is a sample of the default dashboard.

../../../_images/HS2Dashboard.png