Understanding the Hive Metastore Server

Qubole uses Hive Metastore Server (HMS) or the thrift metastore server to respond to Hive queries.

Starting and Stopping HMS

The startup script of HMS is stored in the following location: /usr/lib/hive/bin/thrift-metastore.

Use this command to start HMS: sudo /usr/lib/hive/bin/thrift-metastore server start.

Use this command to stop HMS: sudo /usr/lib/hive/bin/thrift-metastore server stop.

Default HMS Memory Set Based on the Coordinator Node Memory

Qubole allocates memory to HMS in proportion to the total memory of the cluster’s coordinator node. The default value differs according to the coordinator node’s memory. The following table maps the coordinator node memory with the default HMS heap memory.

Note

The default HMS heap memory below applies to Hadoop (Hive), Spark, and Presto clusters that are running Hive metastore version 2.3 or later.

Coordinator Node Memory	Default HMS Heap Memory
Between 8-16 GB	1 GB
Between 16-32 GB	1.5 GB
Between 32-64 GB	3 GB
64 GB	4 GB
Greater than 64 GB	6 GB

Configuring the HMS Memory

Create a ticket with Qubole Support to increase the metastore’s maximum heap memory at the cluster level. (If you get the metastore’s memory increased at the account-level through Qubole Support, then it applies to all clusters.)