Spark Structured Streaming¶
- SPAR-3755: The
appendToTableAPI now supports mixed case schema and automatic data dataType casting similar to
df.insertInto(table)by default. This feature is supported on Spark 2.4.3 and later versions.
SPAR-4079: The following issues with respect to Kinesis Connector for Structured Streaming are fixed:
- Serialization/Deserialization of Kinesis Offset to avoid empty micro-batch.
- Reprocessing of closed shard.
- Incorrect metadata of batch-Id in Kinesis Offset.
INFRA-2751: Logs are now available on the UI for long-running COM commands (running for 40+ hrs).
For a list of bug fixes between versions R57 and R58, see Changelog for api.qubole.com.