Package org.apache.flink.streaming.api.operators.source

Class SourceOutputWithWatermarks<T>

java.lang.Object
org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks<T>
Type Parameters:
T - The type of emitted records.
All Implemented Interfaces:
org.apache.flink.api.common.eventtime.WatermarkOutput, org.apache.flink.api.connector.source.SourceOutput<T>
@Internal public class SourceOutputWithWatermarks<T> extends Object implements org.apache.flink.api.connector.source.SourceOutput<T>
Implementation of the SourceOutput. The records emitted to this output are pushed into a given PushingAsyncDataInput.DataOutput. The watermarks are pushed into the same output, or into a separate WatermarkOutput, if one is provided.

Periodic Watermarks

This output does not implement automatic periodic watermark emission. The method emitPeriodicWatermark() needs to be called periodically.

Note on Performance Considerations

The methods SourceOutput.collect(Object) and SourceOutput.collect(Object, long) are highly performance-critical (part of the hot loop). To make the code as JIT friendly as possible, we want to have only a single implementation of these two methods, across all classes. That way, the JIT compiler can de-virtualize (and inline) them better.

Currently, we have one implementation of these methods for the case where we don't need watermarks (see class NoOpTimestampsAndWatermarks) and one for the case where we do (this class). When the JVM is dedicated to a single job (or type of job) only one of these classes will be loaded. In mixed job setups, we still have a bimorphic method (rather than a poly/-/mega-morphic method).

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    SourceOutputWithWatermarks(PushingAsyncDataInput.DataOutput<T> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<T> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<T> watermarkGenerator)
    Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the (possibly different) WatermarkOutput.

  • Method Summary

    Modifier and Type
    Method
    Description
    final void
    collect(T record)
     
    final void
    collect(T record, long timestamp)
     
    static <E> SourceOutputWithWatermarks<E>
    createWithSeparateOutputs(PushingAsyncDataInput.DataOutput<E> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<E> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<E> watermarkGenerator)
    Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the different WatermarkOutputs.
    final void
    emitPeriodicWatermark()
     
    final void
    emitWatermark(org.apache.flink.api.common.eventtime.Watermark watermark)
     
    void
    markActive()
     
    final void
    markIdle()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • SourceOutputWithWatermarks

      protected SourceOutputWithWatermarks(PushingAsyncDataInput.DataOutput<T> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<T> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<T> watermarkGenerator)
      Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the (possibly different) WatermarkOutput.

  • Method Details

    • collect

      public final void collect(T record)
      Specified by:
      collect in interface org.apache.flink.api.connector.source.SourceOutput<T>

    • collect

      public final void collect(T record, long timestamp)
      Specified by:
      collect in interface org.apache.flink.api.connector.source.SourceOutput<T>

    • emitWatermark

      public final void emitWatermark(org.apache.flink.api.common.eventtime.Watermark watermark)
      Specified by:
      emitWatermark in interface org.apache.flink.api.common.eventtime.WatermarkOutput

    • markIdle

      public final void markIdle()
      Specified by:
      markIdle in interface org.apache.flink.api.common.eventtime.WatermarkOutput

    • markActive

      public void markActive()
      Specified by:
      markActive in interface org.apache.flink.api.common.eventtime.WatermarkOutput

    • emitPeriodicWatermark

      public final void emitPeriodicWatermark()

    • createWithSeparateOutputs

      public static <E> SourceOutputWithWatermarks<E> createWithSeparateOutputs(PushingAsyncDataInput.DataOutput<E> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<E> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<E> watermarkGenerator)
      Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the different WatermarkOutputs.