View Hadoop Jobs Spawned By a Command

GET /api/v1.2/commands/(int: command_id)/jobs

Use this API to retrieve the details of the Hadoop jobs spawned on the cluster by a command (command_id). This information is only available for commands, which have been completed.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.

  • Users who belong to a group associated with a role that allows viewing Hadoop jobs. See Managing Groups and Managing Roles for more information.

Parameters

Parameter

Description

exclude_job_stats

When you specify this parameter, the API does not fetch job stats for child jobs for a given Hadoop command. As a result, the API is much faster when this parameter is specified. In addition, you may use the API even when the cluster is down if this parameter is specified in a given API call.

Response

The response is an array of JSON objects with the following details.

Job Fields

Fields

Description

job_stats

The field displays various details of the job, namely its counters, start time, finish time, and so on. It displays values only when the cluster is active. Otherwise, the field shows NULL value as in job_stats:{} for inactive/terminated clusters.

url

The JobTracker URL for the job.

job_id

The job ID.

error_message

A message indicating an error when retrieving the job details. It is not present if there is no error.

http_error_code

The HTTP error code if any. It is present only if there was an HTTP error retrieving job details.

Note

To not see job statistics in the API response, see Example 2: Viewing Hadoop Jobs without Job Stats.

Examples

Example 1: Viewing Status and Counters of Hadoop Jobs

Goal: To view status and counters of Hadoop jobs spawned by command, 1234.

curl  -i -X GET -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
"https://api.qubole.com/api/v1.2/commands/1234/jobs"

Note

The above syntax uses https://api.qubole.com as the endpoint. Qubole provides other endpoints to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.

Response

[
{
    "job_stats": {
        "finished_at": "Mon Feb 16 07:54:56 UTC 2015",
        "user": "[email protected]",
        "pool_name": "[email protected]",
        "status": "SUCCEEDED",
        "job_name": "110163-ShellCommand",
        "reduce": {
            "num_tasks": "0",
            "pending_tasks": "0",
            "complete_tasks": "0",
            "complete_percent": "100.0",
            "killed_task_attemps": "0",
            "running_tasks": "0",
            "failed_task_attempts": "0",
            "killed_tasks": "0"
        },
        "counters": {
            "org.apache.hadoop.mapred.JobInProgress$Counter": [
                {
                    "name": "Total time spent by all maps",
                    "total_value": "4,545",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all reduces waiting after reserving slots (ms)",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all maps waiting after reserving slots (ms)",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Launched map tasks",
                    "total_value": "1",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all reduces",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                }
            ],
            "com.qubole.ShellLauncher$MapCounter": [
                {
                    "name": "HasActualJobStartedYet",
                    "total_value": "1",
                    "reduce_value": "0",
                    "map_value": "1"
                },
                {
                    "name": "ShellExitCode",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                }
            ],
            "org.apache.hadoop.mapred.Task$Counter": [
                {
                    "name": "Map input records",
                    "total_value": "1",
                    "reduce_value": "0",
                    "map_value": "1"
                },
                {
                    "name": "Total physical memory in bytes",
                    "total_value": "89,743,360",
                    "reduce_value": "0",
                    "map_value": "89,743,360"
                },
                {
                    "name": "Spilled Records",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "MAP_TASK_WALLCLOCK",
                    "total_value": "2,402",
                    "reduce_value": "0",
                    "map_value": "2,402"
                },
                {
                    "name": "Total cumulative CPU milliseconds",
                    "total_value": "530",
                    "reduce_value": "0",
                    "map_value": "530"
                },
                {
                    "name": "Map input bytes",
                    "total_value": "1",
                    "reduce_value": "0",
                    "map_value": "1"
                },
                {
                    "name": "Total virtual memory in bytes",
                    "total_value": "1,122,668,544",
                    "reduce_value": "0",
                    "map_value": "1,122,668,544"
                },
                {
                    "name": "Map output records",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                }
            ]
        },
        "started_at": "Mon Feb 16 07:54:50 UTC 2015",
        "map": {
            "num_tasks": "1",
            "pending_tasks": "0",
            "complete_tasks": "1",
            "complete_percent": "100.0",
            "killed_task_attemps": "0",
            "running_tasks": "0",
            "failed_task_attempts": "0",
            "killed_tasks": "0"
        }
    },
    "url": "https://api.qubole.net/qpal/handle_proxy?query=http%3A%2F%2Fec2-54-161-105-44.compute-1.amazonaws.com%3A50030%2Fjobdetails_json.jsp%3Fjobid%3Djob_11.201502160742_0001",
    "job_id": "job_11.201502160742_0001"
},
{
    "job_stats": {
        "finished_at": "Mon Feb 16 08:02:47 UTC 2015",
        "user": "[email protected]",
        "pool_name": "[email protected]",
        "status": "SUCCEEDED",
        "job_name": "TeraGen",
        "reduce": {
            "num_tasks": "0",
            "pending_tasks": "0",
            "complete_tasks": "0",
            "complete_percent": "100.0",
            "killed_task_attemps": "0",
            "running_tasks": "0",
            "failed_task_attempts": "0",
            "killed_tasks": "0"
        },
        "counters": {
            "org.apache.hadoop.mapred.JobInProgress$Counter": [
                {
                    "name": "Total time spent by all maps",
                    "total_value": "6,293,376",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all reduces waiting after reserving slots (ms)",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all maps waiting after reserving slots (ms)",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Launched map tasks",
                    "total_value": "46",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "Total time spent by all reduces",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                }
            ],
            "FileSystemCounters": [
                {
                    "name": "HDFS_FILES_CREATED",
                    "total_value": "40",
                    "reduce_value": "0",
                    "map_value": "40"
                },
                {
                    "name": "HDFS_BYTES_WRITTEN",
                    "total_value": "100,000,000,000",
                    "reduce_value": "0",
                    "map_value": "100,000,000,000"
                }
            ],
            "org.apache.hadoop.mapred.Task$Counter": [
                {
                    "name": "Map input records",
                    "total_value": "1,000,000,000",
                    "reduce_value": "0",
                    "map_value": "1,000,000,000"
                },
                {
                    "name": "Total physical memory in bytes",
                    "total_value": "5,167,636,480",
                    "reduce_value": "0",
                    "map_value": "5,167,636,480"
                },
                {
                    "name": "Spilled Records",
                    "total_value": "0",
                    "reduce_value": "0",
                    "map_value": "0"
                },
                {
                    "name": "MAP_TASK_WALLCLOCK",
                    "total_value": "6,066,445",
                    "reduce_value": "0",
                    "map_value": "6,066,445"
                },
                {
                    "name": "Total cumulative CPU milliseconds",
                    "total_value": "3,237,210",
                    "reduce_value": "0",
                    "map_value": "3,237,210"
                },
                {
                    "name": "Map input bytes",
                    "total_value": "1,000,000,000",
                    "reduce_value": "0",
                    "map_value": "1,000,000,000"
                },
                {
                    "name": "Total virtual memory in bytes",
                    "total_value": "66,841,980,928",
                    "reduce_value": "0",
                    "map_value": "66,841,980,928"
                },
                {
                    "name": "Map output records",
                    "total_value": "1,000,000,000",
                    "reduce_value": "0",
                    "map_value": "1,000,000,000"
                }
            ]
        },
        "started_at": "Mon Feb 16 07:55:11 UTC 2015",
        "map": {
            "num_tasks": "40",
            "pending_tasks": "0",
            "complete_tasks": "40",
            "complete_percent": "100.0",
            "killed_task_attemps": "6",
            "running_tasks": "0",
            "failed_task_attempts": "0",
            "killed_tasks": "0"
        }
    },
    "url": "https://api.qubole.net/qpal/handle_proxy?query=http%3A%2F%2Fec2-54-161-105-44.compute-1.amazonaws.com%3A50030%2Fjobdetails_json.jsp%3Fjobid%3Djob_11.201502160742_0003",
    "job_id": "job_11.201502160742_0003"
}
]

Example 2: Viewing Hadoop Jobs without Job Stats

curl  -i -X GET -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
"https://api.qubole.com/api/v1.2/commands/100/jobs?exclude_job_stats=true'"

Note

The above syntax uses https://api.qubole.com as the endpoint. Qubole provides other endpoints to access QDS that are described in Supported Qubole Endpoints on Different Cloud Providers.