Understanding the Hive Metastore Server

Qubole uses Hive Metastore Server (HMS) or the thrift metastore server to respond to Hive queries.

Starting and Stopping HMS

The startup script of HMS is stored in the following location: /usr/lib/hive/bin/thrift-metastore.

Use this command to start HMS: sudo /usr/lib/hive/bin/thrift-metastore server start.

Use this command to stop HMS: sudo /usr/lib/hive/bin/thrift-metastore server stop.

Default HMS Memory Set Based on the Master Node Memory

Qubole allocates memory to HMS in proportion to the total memory of the cluster’s master node. The default value differs according to the master node’s memory. The following table maps the master node memory with the default HMS heap memory.

Master Node Memory Default HMS Heap Memory
Between 8-16 GB 1 GB
Between 16-32 GB 1.5 GB
Between 32-64 GB 3 GB
64 GB 4 GB
Greater than 64 GB 6 GB

Configuring the HMS Memory

Create a ticket with Qubole Support to increase the metastore’s maximum heap memory at the cluster level. (If you get the metastore’s memory increased at the account-level through Qubole Support, then it applies to all clusters.)