Spark Structured Streaming

In this release, Spark Structured Streaming has the following enhancement.

Enhancements

SPAR-4230: For the optimized performance, users must specify the --conf spark.sql.streaming.stateStore.rocksDb.localDir=<tmp-path> option when using RocksDB based state storage for structured streaming. This prevents creation of multiple temporary directories, which can result in high consumption of disk space.

Bug Fixes

There are no bug fixes in this version.

Feedback | Try Free Trial