Spark Structured Streaming¶
Qubole introduces Spark Structured Streaming in this release.
Comprehensive Support for Kinesis Connector in Structured Streaming¶
- SPAR-2189 and SPAR-2802: You can add Kinesis as a data source provider in your streaming queries for micro-batch streaming and continuous streaming modes.
- SPAR-2754: You can create a streaming application which can write output to a Kinesis stream by using
kinesisas a valid sink format.
- SPAR-2729: IAM roles are added in Kinesis Connector for structured streaming. You do not have to specify the credentials when writing streaming applications. Your account must be enabled for IAM roles and the permissions to read Kinesis stream to the attached IAM role must be granted.
Structured Streaming Query in a Spark Data Source Table¶
SPAR-3001: You can write a structured streaming query, which can append the data to a table, and you can read the updated table in real-time.
Streaming Query Progress Graphs in Notebooks¶
ZEP-2650: You can easily monitor the streaming query progress by viewing the streaming query progress graphs in notebooks. When you start a streaming query in a notebook paragraph, the monitoring graph is displayed in the same paragraph. Via Support Disabled