View Cluster Health APIs

GET /api/v1.3/clusters/(string:id or label)/live_cluster_health

Use this API to view the latest health of a running cluster in a Qubole environment. It is supported with QDS version R57 onwards. It is supported in Cluster API v1.3, v2.0, and v2.1.

Required Role

Users belong to a group that has permission to read a cluster required to invoke this API.

Request API Syntax

curl -i -X GET -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json"
"https://api.qubole.com/api/v2/clusters/<cluster_id>/live_cluster_health

As the value of <Qubole Environment>, use the Qubole environment where you have the QDS account. For example, https://api.qubole.com is a Qubole environment.

Note

The above syntax uses cluster v2 and the response below are for the cluster API version 2.0.

Sample Response

Hive (as an additional cluster with HiveServer2 disabled and HiveServer2 enabled):

{
  "cluster_id": 5,
  "cluster_inst_id": 2,
     "metrics": {
        "captured_at": "2019-10-03T08:22:02Z",
        "engine": {
            "yarn": {
               "memory": "0",
               "containers": {
                   "pending": "0",
                   "failed": "0",
                   "killed": "0"
               }
           }
       },
       "daemons": {
           "hive_metastore": {
               "responsiveness_status": "UP",
               "liveliness_status": "UP",
               "heap": {
                   "usage_percent": "3.96",
                   "status": "green"
               }
           },
           "resourcemanager": "UP",
           "namenode": "UP"
       },
       "system": {
           "master": {
               "cpu_usage": "8.61",
               "disk_usage": "70.7",
               "spotloss_count": 0
           }
       }
   }
}

Hive (With HiveServer2 enabled on coordinator):

{
  "cluster_id": 1,
  "cluster_inst_id": 1,
  "metrics": {
      "captured_at": "2019-10-03T08:22:07Z",
      "engine": {
          "yarn": {
              "memory": "0",
              "containers": {
                  "pending": "0",
                  "failed": "0",
                  "killed": "0"
              }
          }
      },
      "daemons": {
          "hive_metastore": {
              "responsiveness_status": "UP",
              "liveliness_status": "UP",
              "heap": {
                  "usage_percent": "3.83",
                  "status": "green"
              }
          },
          "hs2_server": {
              "responsiveness_status": "UP",
              "liveliness_status": "UP",
              "heap": {
                  "usage_percent": "1.88",
                  "status": "green"
              }
          },
          "resourcemanager": "UP",
          "namenode": "UP"
      },
      "system": {
          "master": {
              "cpu_usage": "10.55",
              "disk_usage": "70.7",
              "spotloss_count": 0
          }
      }
  }}

Presto:

{
   "cluster_id": 213,
   "cluster_inst_id": 738,
   "metrics": {
       "captured_at": "2019-10-03T08:05:09Z",
        "engine": {
           "presto": {
               "status": "UP",
               "heap": {
                   "usage_percent": "0.48",
                   "status": "green"
               }
           }
       },
       "daemons": {
           "hive_metastore": {
               "responsiveness_status": "UP",
               "liveliness_status": "UP",
                 "heap": {
                     "usage_percent": "15.5",
                     "status": "green"
                }
             },
             "zeppelin": {
                 "status": "UP",
                 "heap": {
                     "usage_percent": "2.08",
                     "status": "green"
                 }
             }
         },
         "system": {
             "master": {
                 "cpu_usage": "20.41",
                 "disk_usage": "71.5",
                 "spotloss_count": 0
             }
         }
     }
   }

Spark:

{
    "cluster_id": 473,
    "cluster_inst_id": 737,
    "metrics": {
        "captured_at": "2019-10-03T08:00:15Z",
        "engine": {
            "yarn": {
                "memory": "0",
                "containers": {
                    "pending": "0",
                    "failed": "0",
                    "killed": "0"
                }
            },
            "spark": {}
        },
        "daemons": {
            "hive_metastore": {
                "responsiveness_status": "UP",
                "liveliness_status": "UP",
                "heap": {
                    "usage_percent": "10.07",
                    "status": "green"
                }
            },
            "zeppelin": {
                "status": "UP",
                "heap": {
                    "usage_percent": "3.68",
                    "status": "green"
                }
            },
            "resourcemanager": "UP",
            "namenode": "UP"
        },
        "system": {
            "master": {
                "cpu_usage": "0",
                "disk_usage": "73.0",
                "spotloss_count": 0
            }
        }
    }
}

Airflow:

{
   "cluster_id": 473,
   "cluster_inst_id": 737,
   "metrics": {
       "captured_at": "2019-10-03T08:00:15Z",
       "engine": {
           "yarn": {
               "memory": "0",
               "containers": {
                   "pending": "0",
                   "failed": "0",
                   "killed": "0"
               }
           },
           "spark": {}
       },
       "daemons": {
           "hive_metastore": {
               "responsiveness_status": "UP",
               "liveliness_status": "UP",
               "heap": {
                   "usage_percent": "10.07",
                   "status": "green"
               }
           },
           "zeppelin": {
               "status": "UP",
               "heap": {
                   "usage_percent": "3.68",
                   "status": "green"
               }
           },
           "resourcemanager": "UP",
           "namenode": "UP"
       },
       "system": {
           "master": {
               "cpu_usage": "0",
               "disk_usage": "73.0",
               "spotloss_count": 0
           }
       }
   }
}

Note

YARN-based metrics are only available when Ganglia is enabled on the cluster.

Cluster Health Services and Metrics Information:

Metrics/Service Available On Cluster Type
Binary Metrics (Services)
Hive Metastore All
Name Node Hive, Spark
Resource Manager Hive, Spark
HS2 Hive (HS2 enabled on coordinator)
Zeppelin Spark, Presto
Presto Presto
Bar Metrics (Float)
CPU Usage All
Coordinator Disk Usage All
Spot nodes lost count (Integer) All
Heap Information (All heap metrics are calculated from jstat command)
Hive Metastore Heap All
HS2 Heap Hive (HS2 enabled on coordinator)
Presto Heap Presto
Zeppelin Heap Presto, Spark