New Beta Features (Support Ticket Required)

The following Beta features are available on request from Qubole Support: create a ticket with Qubole Support to enable them BetaImage.

Notebooks

  • A new field Default Language has been added added for Spark notebooks, allowing you to choose the default language for the Spark interpreter while creating the notebook. This default language is retained when the notebook is detached from one cluster and is attached to another cluster and when the notebook is imported or exported.

  • Existing Spark interpreters have been made compact. Properties whose values are not explicitly overridden are removed from the interpreters and the values of those properties are set to the cluster defaults.

  • Qubole has improved the user experience with faster Zeppelin bring-up. This will also help in resolving the following intermittent issues reported by customers:

    • Loss of interpreters

    • The Zeppelin service not being up

Spark in Qubole

  • Qubole Spark supports the Hive 2.1 metastore for Spark 2.2.x.

  • Qubole Spark supports reading from Kinesis Source in Structured Streaming. You can now add Kinesis as a data source provider in streaming queries.

  • The default value of max-executors for a Spark application has been increased from 2 to 1000. If you want to use a different value, set the spark.dynamicAllocation.maxExecutors configuration explicitly at the Spark application level. If you want a different value for all Spark applications run on a cluster, set the value as a Spark override on that cluster. This takes effect only when the Spark application is run from the Analyze UI or through a REST API call. It does not apply to Spark notebooks.