Interpreter Modes for Jupyter Notebooks

Jupyter Notebooks on QDS support two Interpreter modes: Scoped mode and Isolated mode.

Scoped mode

This the default mode. In this mode, there is one Spark application for each user. All the notebooks executed by a user share the same Spark application. To prevent namespace conflicts between notebooks, new interpreter group per kernel is created. Each interpreter group has its own REPL or spark-shell. The driver is created with the default 2 GB memory. When the user disconnects the kernel, the Spark application is not terminated immediately but is terminated after the 60 min idle timeout.

Isolated mode

In this mode, there is one Spark application per notebook of a user. A new Spark application is started for a notebook.

Switching the Interpreter Modes for a Notebook

To switch from the Scoped mode to the Isolated mode for a notebook, use the %%configure magic.

For example:

%%configure -f
{"driverMemory": "2g"}

When you use the %%configure magic to specify any spark configuration, a new Spark application in the Isolated mode is created with the specified configuration for your notebook.

To switch from the Isolated to Scoped mode, delete the cell that has %%configure magic from the notebook. Restart the kernel.

If you want to change the default mode to isolated mode, contact Qubole Support.