Spark Structured Streaming

Enhancements

  • SPAR-3755: The appendToTable API now supports mixed case schema and automatic data dataType casting similar to df.insertInto(table) by default. This feature is supported on Spark 2.4.3 and later versions.

Bug Fixes

  • SPAR-4079: The following issues with respect to Kinesis Connector for Structured Streaming are fixed:

    • Serialization/Deserialization of Kinesis Offset to avoid empty micro-batch.
    • Reprocessing of closed shard.
    • Incorrect metadata of batch-Id in Kinesis Offset.

  • INFRA-2751: Logs are now available on the UI for long-running COM commands (running for 40+ hrs).

For a list of bug fixes between versions R57 and R58, see Changelog for api.qubole.com.