Supported Operators

You can use the predefined streaming operators when building the assisted streaming pipelines.

The predefined streaming operators are:

  • Select: You can select the set of columns that have to be propagated to the target.

  • Filter: You can select a source column and apply function to filter the required data.

  • Aggregation: You can select a source schema as a date column or time stamp column, specify the window length, and sliding interval options to generate aggregated data based on the specified filters.

  • Watermark: To manage late arrival of data, you can specify the streaming watermark in terms of any date or time column, and threshold in minutes.

The following animated GIF shows supported the operators.

../../../_images/out-of-box-operators.gif

For more information of the Structured Streaming operators, see Operations on streaming DataFrames/Datasets.