HMS Total Memory Used > 90%¶
This runbook shows steps to perform upon receiving an alert that the Hive Metastore Server’s memory usage is above 90%.
- Check the dashbord HMS Memory Usage. Specifically look for a trend in the metric
hive.hms.memory.pools_CMS-Perm-Gen_usage
, along with heap trends. - Restart the HMS process in the interim if failures are not contained.
Logs¶
- HMS logs are available on the coordinator node: “/media/ephemeral0/logs/hive1.2/hive_ms.log”
- Look for any evident errors(do basic grep and count of errors).
- Look at the dashboards defined above. (“title” is the name of the dashboard).
Restart of Process¶
sudo monit summary
to check the status of the process.sudo monit stop metastore1_2
to stop the process.sudo monit start metastore1_2
to start the process.