HMS Total Memory Used > 90%

This runbook shows steps to perform upon receiving an alert that the Hive Metastore Server’s memory usage is above 90%.

  • Check the dashbord HMS Memory Usage. Specifically look for a trend in the metric hive.hms.memory.pools_CMS-Perm-Gen_usage, along with heap trends.

  • Restart the HMS process in the interim if failures are not contained.

Logs

  • HMS logs are available on the coordinator node: “/media/ephemeral0/logs/hive1.2/hive_ms.log”

  • Look for any evident errors(do basic grep and count of errors).

  • Look at the dashboards defined above. (“title” is the name of the dashboard).

Restart of Process

  • sudo monit summary to check the status of the process.

  • sudo monit stop metastore1_2 to stop the process.

  • sudo monit start metastore1_2 to start the process.