Spark Structured Streaming

New Features

  • SPAR-3615: For Spark structured streaming, users can create a Snowflake datastore in the QDS UI, and use the corresponding catalog name (instead of passing username and password) on the QuEST UI or Notebooks UI . The Snowflake dependent jar is upgraded to support Snowflake structured streaming.

    Users can use the following API for Snowflake structured streaming:

    val query = df
    .option("sfDatabase", <DATABASE NAME> )
    .option("streaming_stage", <TEMP TABLE NAME>)
    .option("checkpointLocation", <CHECKPOINT LOCATION>)

    This is supported on Spark 2.3.3, 2.4.0, 2.4.3, and later versions.


  • SPAR-3747: Memory leak issues in RocksDB based state store are fixed, and the rocksdb configuration is better tuned to improve read performance. The previous rocksDB state-store provider class is deprecated. Users should use the following Spark configuration to enable RocksDB based state store:

    spark.sql.streaming.stateStore.providerClass = org.apache.spark.sql.execution.streaming.state.RocksDbStateStoreProvider

    This is supported on Spark 2.4.0 and later versions.

  • SPAR-3760: Users should specify the SQS queue region in the following format:

      val inputDf = spark
      .option("sqsUrl", queueUrl)
      .option("region", region)
      .option("fileFormat", "json")
      .option("sqsFetchIntervalSeconds", "2")
      .option("sqsLongPollingWaitTimeSeconds", "5")
      .option("maxFilesPerTrigger", "50")
      .option("ignoreFileDeletion", "true")
    This is supported on 2.4.0, 2.4.3, and later versions.
  • SPAR-3555: Users can write a structured streaming query, which can append the data to a Spark Datasource or Hive table, and users can query the table to view the latest data in real-time. Users can use the following API:

    .option("checkpointLocation", checkpointDir)

    This is supported on Spark 2.3.2, 2.4.0, 2.4.3, and later versions.

  • SPAR-3591: Users can pass x-amz-meta-metadata(key1=value1,key2=val2) while creating a new streaming job with s3 as the sink by setting the option fs.s3a.user.metadata as key1=val1,key2=val2. The streaming application creates new files with this metadata.

    This is supported on Spark 2.3.2 and later versions.

For a list of bug fixes between versions R56 and R57, see Changelog for