hive-hs2-memory-usage

This runbook shows how to respond to an alert that indicates a high percentage of memory usage on HiveServer2.

Alert Name: HS2 Memory Usage

Alert Message: “More than 90% memory in use on HS2”

Alert Condition: This alert is returned by the following query: avg(last_1m):avg:hive.hs2.memory.total_used{*} by {host} / avg:hive.hs2.memory.total_max{*} by {host} >= 0.9

Alert Explanation: The alert indicates that memory usage for HiveServer2 is above 90%.

Resolution:

• Check the dashbords “HS2 Memory Usage” and “HS2 GC Time.” Look for a hive.hs2.memory.pools_CMS-Perm-Gen_usage trend along with heap trends.
• Restart the HiveServer2 process in the interim if failures are not contained.