Package org.apache.flink.table.connector.source.abilities

Interface SupportsWatermarkPushDown

@PublicEvolving public interface SupportsWatermarkPushDown
Enables to push down a watermark strategy into a ScanTableSource.

The concept of watermarks defines when time operations based on an event time attribute will be triggered. A watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Thus, watermarks are a trade-off between latency and completeness.

Given the following SQL: 


 CREATE TABLE t (i INT, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND)  // `ts` becomes a time attribute

In the above example, generated watermarks are lagging 5 seconds behind the highest seen timestamp.

For correctness, it might be necessary to perform the watermark generation as early as possible in order to be close to the actual data generation within a source's data partition.

If the ScanTableSource.getScanRuntimeProvider(ScanTableSource.ScanContext) returns SourceProvider, watermarks will be automatically pushed into the runtime source operator by the framework. In this case, this interface does not need to be implemented.

If the ScanTableSource does not return a SourceProvider and this interface is not implemented, watermarks are generated in a subsequent operation after the source. In this case, it is recommended to implement this interface to perform the watermark generation within source's data partition.

This interface provides a WatermarkStrategy that needs to be applied to the runtime implementation. Most built-in Flink sources provide a way of setting the watermark generator.

See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<RowData> watermarkStrategy)
    Provides a WatermarkStrategy which defines how to generate Watermarks in the stream source.

  • Method Details

    • applyWatermark

      void applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<RowData> watermarkStrategy)
      Provides a WatermarkStrategy which defines how to generate Watermarks in the stream source.

      The WatermarkStrategy is a builder/factory for the actual runtime implementation consisting of TimestampAssigner (assigns the event-time timestamps to each record) and the WatermarkGenerator (generates the watermarks).

      Note: If necessary, the watermark strategy will contain required computed column expressions and consider metadata columns (if SupportsReadingMetadata is implemented).