Running Spark Notebooks in a Schedule¶
You can create a scheduler to run notebooks at periodic intervals without a manual intervention using Qubole Scheduler.
Perform the following steps to run a notebook by using the Qubole Scheduler:
Navigate to the Scheduler page, and click the +Create button in the left pane for creating a schedule.
- Press Ctrl + / to see the list of available keyboard shortcuts. For more information, see
For more information on using the user interface to schedule jobs, see Qubole Scheduler.
Through the Qubole Scheduler, you can schedule a notebook to run even when its associated cluster is down.
Create a schedule for running a notebook from the General Tab:
- Enter a name in the Schedule Name text field. This field is optional. If it is left blank, a system-generated ID is set as the schedule name.
- In the Tags text field, add one or a maximum of six tags to group commands together. Tags help in identifying commands. Each tag can contain a maximum of 20 characters. It is an optional field.
- In the command field, select Spark Command from the drop-down list. By default, Scala is selected as the programming language in the drop-down list that contains Command Line, Python, SQL, R and Notebook.
- Select Notebook from the list. A Notebook lists appears with the associated cluster.
- Pick the note that you want to run by scheduling. You must use the Notebooks UI to compose notebook paragraphs. See Run a Spark Notebook from the Analyze Query Composer for more information.
- If you want to set arguments and pass the values, enter the arguments in the Arguments field and pass the values in the Scheduler’s macros as illustrated here.
Follow the steps in Creating a New Schedule to get the steps after selecting the notebook from the list, which include setting parameters and notifications.