Interpreter Modes for Jupyter Notebooks

Jupyter Notebooks on QDS support two Interpreter modes: Scoped mode and Isolated mode.

Scoped mode

This the default mode. In this mode, there is one Spark application for each user. All the notebooks executed by a user share the same Spark application. To prevent namespace conflicts between notebooks, new interpreter group per kernel is created. Each interpreter group has its own REPL or spark-shell. The driver is created with the default 2 GB memory. When the user disconnects the kernel, the Spark application is not terminated immediately but is terminated after the 60 min idle timeout.

In this mode, you cannot modify the Spark configurations. The Spark application considers the default configuration set at the cluster level. All the notebooks in the Scoped mode run serially and cells within a notebook run serially.

Isolated mode

In this mode, there is one Spark application per notebook of a user. A new Spark application is started for a notebook. Multiple notebooks can run parallelly and cells within a notebook run serially.

The notebooks first cell should be %%configure cell magic.

For example:

%%configure -f
{ }

You can modify the Spark configuration in the Isolated mode. To modify the Spark configuration, you can use %%configure cell magic and pass the parameters mentioned in the Spark Configuration Parameters table.

For example:

%%configure -f
{ "driverMemory" : "2g" , "conf" : { "spark.jars.packages":"graphframes:graphframes:0.7.0-spark2.4-s_2.11" } }

When you use the %%configure magic to specify any spark configuration, a new Spark application in the Isolated mode is created with the specified configuration for your notebook.

If you want to change the default mode to isolated mode, contact Qubole Support.

Switching the Interpreter Modes for a Notebook

To switch from the Scoped mode to the Isolated mode for a notebook, use the %%configure magic. For more details, see Isolated mode.

To switch from the Isolated to Scoped mode, delete the cell that has %%configure magic from the notebook. Restart the kernel.