Understanding the Qubole Scheduler Concepts

This section describes the concepts associated with the Qubole Scheduler.

Schedule Action

One occurrence of the schedule that runs at a particular time period is called a schedule action. A schedule can have many schedule action that can be daily, weekly, monthly, or hourly as it is configured. For more information, see List Schedule Actions.

Action

An action is run by the scheduler. An action can belong to any schedule in the account. For more information, see List All Actions.

Nominal Time

It is the time for which the Schedule Action was processed.

Next Materialized Time

It is the time when the next Schedule Action of the schedule is picked. The Next Materialized Time is calculated after the respective scheduler runs for the next time per the schedule that is after the first run of the respective job. But it is not determined at the time of schedule creation.

Note

While editing the Start Time, ensure that the Start Time is less than the Next Scheduled Time (Next Materialized Time) but more than the current time.

Created At

It is the time at which the Scheduler picked up the schedule.

The Created At time for any schedule action is greater than the Nominal Time. The Scheduler picks up a scheduled job after the Nominal Time is passed. For example, if the Nominal Time is 10:00 AM, the Scheduler picks the job at 10:01 AM. Thus, each schedule is run after the Nominal Time is passed. If a schedule action is skipped when Skip Missed Instance is enabled, that schedule action is never picked because the Created At time is greater than the Nominal Time. So, the Scheduler must run the last schedule action of the schedule.

Skip Missed Instances

When a new schedule is created, the scheduler runs schedule actions from start time to the current time. For example, if a daily schedule is created from Jan 1 2015 on May 1 2015, schedule actions are run for Jan 1 2015, Jan 2 2015, and so on. If you do not want the scheduler to run the schedule actios for months earlier to May, select the check box to skip them on the QDS UI or set no_catchup to true on a scheduler API call.

The main use of skipping a schedule action is if when you suspend a schedule and resume it later, in which case, there will be more than one schedule action and you might want to skip the earlier schedule actions.

Scheduled Job Rerun Limit

Qubole Scheduler has introduced a new limit for schedule reruns to be processed concurrently at a given point of time. Its default value is 20. When the concurrent reruns of a schedule exceeds the limit, you get this error message.

A maximum of 20 reruns are allowed for scheduled job in your account: #<account_number>.

When this limit is 20 (default value), if there are 20 reruns of a schedule to be processed and 2 of them are completed, then you can add 2 new reruns.

You can increase the rerun limit by creating a ticket with Qubole Support. The new value is applicable to all the jobs of a given QDS account.