Notebooks and Dashboards

In this release, Qubole provides various new features, enhancements, and bug fixes for Notebooks and Dashboards.

Jupyter Notebooks (Beta)

New Features

  • JUPY-219: Users can view and edit their Jupyter notebooks while the clusters are down. The clusters have to be started only when the user has to execute the Jupyter notebooks. To execute a Jupyter notebook, the user must attach a cluster to the notebook before executing it. Via Support.
  • JUPY-199: Users can schedule Jupyter notebooks. Users can set custom parameters to their schedules, and view the schedules and their execution history from the JupyterLab interface. Via Support.
  • JUPY-197, JUPY-356 and JUPY-289 : Jupyter Notebooks are integrated with Version Control Systems: GitHub, GitLab, and BitBucket (only cloud). Users can use these version control systems to manage versions of the notebooks. Users can synchronize notebooks with the respective public and private repositories, and can view or compare notebook versions. Additionally, users can create pull requests. Via Support.
  • JUPY-195 and JUPY-334: Users can set Access Control for the Jupyter notebooks at both account level and object level. Users with system-admin role or roles with appropriate permissions can configure the Jupyter Notebook resource at the account level. Notebooks users can override the permissions by setting access control for the objects that they own. Via Support.
  • JUPY-251: Jupyter notebooks are accessible through the shareable links to the users with appropriate access control. Users can obtain the shareable link for Jupyter notebooks by performing the following actions:
    1. Navigate to the File Browser sidebar panel.
    2. Select the required notebook and right-click.
    3. From the menu, select Copy Shareable Link.
  • JUPY-193: Users can create and manage Jupyter notebooks by using the REST APIs.

Enhancements

  • JUPY-319: Users should specify the notebook/folder name when creating the notebook instead of creating a notebook/folder with the default Untitled* name.
  • JUPY-272: Users can use the QDS Object Storage Explorer on the left side bar of the JupyterLab interface to explore the cloud storage, such as S3, Blob Store, and ABFS and also perform actions such as upload or download file.
  • JUPY-271: Users can view the Table Explorer on the left side bar to explore Hive Metastore, schema, tables, and columns.
  • JUPY-234: Users can copy the example Jupyter notebooks from the right-click context menu of the example notebook and paste it in the File Browser sidebar panel by using the context menu.
  • JUPY-413: To prevent Livy session timeout for long running notebooks, users can configure the kernel and livy session idle timeout using the spark.qubole.idle.timeout setting. Users can set this configuration in the Override Spark Configuration field on the Advanced Configuration page of the Spark cluster. Possible values: an integer value in minutes or -1 for no timeout.

Bug Fixes

  • JUPY-332: Module not found errors occurred when trying to import code from the bootstrapped custom zip files in notebooks. This issue is fixed.
  • JUPY-308: Spark application startup used to fail when few third party jars were added to the Spark configuration. This issue is fixed.
  • JUPY-269: Jupyter notebooks were not working with clusters configured with default locations on ADLS and ABFS storage. This issue is fixed.
  • JUPY-214: The Spark applications that are stuck in the Accepted state when the session could not be started, are terminated when the timeout occurs.

Zeppelin Notebooks

New Features

  • ZEP-493: Bitbucket is integrated with notebooks. You can use Bitbucket to manage versions of your notebooks. Learn more.

Enhancements

  • ZEP-3915: The following enhancements are available with Zeppelin 0.8.0:

    • ZEP-2749: Pyspark and IPyspark interpreters are supported with IPython as the default shell. If you want to set Python shell as the default shell for Pyspark interpreter, then set zeppelin.pyspark.useIPython to false in the Interpreter settings. Via Support.
    • ZEP-4077: Notebooks now support z.run(noteId, paragraphId) and z.runNote(noteId) functions to run paragraphs or notebooks from within the notebook.
    • ZEP-3317: Users can run markdown (%md) paragraphs in edit mode even when cluster is down.
    • ZEP-1908: The geolocation graph type is available on the UI by default.

  • ZEP-4129: For non heterogeneous spark clusters with version 2.3.2 and later, spark driver memory is allocated based on the instance type of cluster to optimize the memory usage . As a result, for the notebooks attached to such clusters, user should not specify the spark.driver.memory property when overriding the Spark configuration of cluster.

  • ZEP-4169 and ZEP-134: Zeppelin application and all interpreters started by Zeppelin (including Spark and shell interpreters) can now be run as a yarn user. Earlier, Zeppelin applications and all the interpreters were run as root user which was a security concern for many enterprises. Feature to opt in.

Bug Fixes

  • ZEP-3298: In case of any failures, the scheduled notebooks with retry no.option specified in the Scheduler properties were not re-run. This issue is fixed.
  • ZEP-4193: Autocomplete now works for PySpark notebooks in Zeppelin 0.8.0 version.
  • ZEP-4194: Notebooks results were not displayed when clusters running Zeppelin 0.6.0 were upgraded to Zeppelin 0.8.0, or when clusters running Zeppelin 0.8.0 were downgraded to Zeppelin 0.6.0. This issue is fixed.
  • ZEP-3122: The stacked option for graphs/charts in Zeppelin notebooks failed to persist after a refresh. This issue is fixed.
  • ZEP-4198: The Notebooks home page was displayed when the cluster was started. This issue is fixed.
  • ZEP-3129: External web links referenced in markdown paragraph open in a separate tab.
  • ZEP-4195 and ZEP-4199: Notebooks content were not rendered correctly when switching notebooks between clusters. This issue is fixed.
  • ZEP-4181: The published at field in the Dashboard information on the Notebooks page displayed incorrect timestamp. This issue is fixed.
  • ZEP-4004 and ZEP-3562: With a large number of cardinality in multibar charts, the notebook becomes unresponsive. A maximum limit of 50 on cardinality is added to fix this issue. If you want to increase the limit, contact Qubole Support.

For a list of bug fixes between versions R57 and R58, see Changelog for api.qubole.com.