Package org.apache.flink.runtime.asyncprocessing.operators.windowing

Class AsyncWindowOperator<K,IN,ACC,OUT,W extends Window>

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateStreamOperator<OUT>
org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator<OUT,InternalAsyncWindowFunction<ACC,OUT,K,W>>
org.apache.flink.runtime.asyncprocessing.operators.windowing.AsyncWindowOperator<K,IN,ACC,OUT,W>
Type Parameters:
K - The type of key returned by the KeySelector.
IN - The type of the incoming elements.
OUT - The type of elements emitted by the InternalWindowFunction.
W - The type of Window that the WindowAssigner assigns.
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, Input<IN>, KeyContext, KeyContextHandler, OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.OutputTypeConfigurable<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, Triggerable<K,W>, UserFunctionProvider<InternalAsyncWindowFunction<ACC,OUT,K,W>>, YieldingOperator<OUT>, AsyncStateProcessing, AsyncStateProcessingOperator
Direct Known Subclasses:
AsyncEvictingWindowOperator
@Internal public class AsyncWindowOperator<K,IN,ACC,OUT,W extends Window> extends AbstractAsyncStateUdfStreamOperator<OUT,InternalAsyncWindowFunction<ACC,OUT,K,W>> implements OneInputStreamOperator<IN,OUT>, Triggerable<K,W>
An operator that implements the logic for windowing based on a WindowAssigner and AsyncTrigger. This is the async state access version of WindowOperator.

When an element arrives it gets assigned a key using a KeySelector and it gets assigned to zero or more windows using a WindowAssigner. Based on this, the element is put into panes. A pane is the bucket of elements that have the same key and same Window. An element can be in multiple panes if it was assigned to multiple windows by the WindowAssigner.

Each pane gets its own instance of the provided Trigger. This trigger determines when the contents of the pane should be processed to emit results. When a trigger fires, the given InternalWindowFunction is invoked to produce the results that are emitted for the pane to which the Trigger belongs.

See Also:

  • Field Details

    • windowAssigner

      protected final WindowAssigner<? super IN,W extends Window> windowAssigner

    • keySerializer

      protected final org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer
      For serializing the key in checkpoints.

    • windowSerializer

      protected final org.apache.flink.api.common.typeutils.TypeSerializer<W extends Window> windowSerializer
      For serializing the window in checkpoints.

    • allowedLateness

      protected final long allowedLateness
      The allowed lateness for elements. This is used for:
      • Deciding if an element should be dropped from a window due to lateness.
      • Clearing the state of a window if the system time passes the window.maxTimestamp + allowedLateness landmark.

    • lateDataOutputTag

      protected final org.apache.flink.util.OutputTag<IN> lateDataOutputTag
      OutputTag to use for late arriving events. Elements for which window.maxTimestamp + allowedLateness is smaller than the current watermark will be emitted to this.

    • numLateRecordsDropped

      protected transient org.apache.flink.metrics.Counter numLateRecordsDropped

    • timestampedCollector

      protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector
      This is given to the InternalWindowFunction for emitting elements with a given timestamp.

    • triggerContext

      protected transient AsyncWindowOperator<K,IN,ACC,OUT,W extends Window>.Context triggerContext

    • processContext

      protected transient AsyncWindowOperator<K,IN,ACC,OUT,W extends Window>.WindowContext processContext

    • windowDeclaredVariable

      protected transient DeclaredVariable<W extends Window> windowDeclaredVariable

    • windowAssignerContext

      protected transient WindowAssigner.WindowAssignerContext windowAssignerContext

    • internalTimerService

      protected transient InternalTimerService<W extends Window> internalTimerService

  • Constructor Details

    • AsyncWindowOperator

      public AsyncWindowOperator(WindowAssigner<? super IN,W> windowAssigner, org.apache.flink.api.common.typeutils.TypeSerializer<W> windowSerializer, org.apache.flink.api.java.functions.KeySelector<IN,K> keySelector, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, org.apache.flink.api.common.state.v2.StateDescriptor<?> windowStateDescriptor, InternalAsyncWindowFunction<ACC,OUT,K,W> windowFunction, AsyncTrigger<? super IN,? super W> trigger, long allowedLateness, org.apache.flink.util.OutputTag<IN> lateDataOutputTag)
      Creates a new WindowOperator based on the given policies and user functions.

  • Method Details

    • setup

      public void setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output)
      Overrides:
      setup in class AbstractAsyncStateUdfStreamOperator<OUT,InternalAsyncWindowFunction<ACC,OUT,K,W extends Window>>

    • open

      public void open() throws Exception
      Description copied from class: AbstractStreamOperator
      This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

      The default implementation does nothing.

      Specified by:
      open in interface StreamOperator<K>
      Overrides:
      open in class AbstractAsyncStateUdfStreamOperator<OUT,InternalAsyncWindowFunction<ACC,OUT,K,W extends Window>>
      Throws:
      Exception - An exception in this method causes the operator to fail.

    • close

      public void close() throws Exception
      Description copied from interface: StreamOperator
      This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

      This method is expected to make a thorough effort to release all resources that the operator has acquired.

      NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

      Specified by:
      close in interface StreamOperator<K>
      Overrides:
      close in class AbstractAsyncStateUdfStreamOperator<OUT,InternalAsyncWindowFunction<ACC,OUT,K,W extends Window>>
      Throws:
      Exception

    • processElement

      public void processElement(StreamRecord<IN> element) throws Exception
      Description copied from interface: Input
      Processes one element that arrived on this input of the MultipleInputStreamOperator. This method is guaranteed to not be called concurrently with other methods of the operator.
      Specified by:
      processElement in interface Input<K>
      Throws:
      Exception

    • onEventTime

      public void onEventTime(InternalTimer<K,W> timer) throws Exception
      Description copied from interface: Triggerable
      Invoked when an event-time timer fires.
      Specified by:
      onEventTime in interface Triggerable<K,IN>
      Throws:
      Exception

    • onProcessingTime

      public void onProcessingTime(InternalTimer<K,W> timer) throws Exception
      Description copied from interface: Triggerable
      Invoked when a processing-time timer fires.
      Specified by:
      onProcessingTime in interface Triggerable<K,IN>
      Throws:
      Exception

    • getOperatorAttributes

      public OperatorAttributes getOperatorAttributes()
      Description copied from interface: StreamOperator
      Called to get the OperatorAttributes of the operator. If there is no defined attribute, a default OperatorAttributes is built.
      Specified by:
      getOperatorAttributes in interface StreamOperator<K>
      Returns:
      OperatorAttributes of the operator.

    • sideOutput

      protected void sideOutput(StreamRecord<IN> element)
      Write skipped late arriving element to SideOutput.
      Parameters:
      element - skipped late arriving element to side output

    • isWindowLate

      protected boolean isWindowLate(W window)
      Returns true if the watermark is after the end timestamp plus the allowed lateness of the given window.

    • isElementLate

      protected boolean isElementLate(StreamRecord<IN> element)
      Decide if a record is currently late, based on current watermark and allowed lateness.
      Parameters:
      element - The element to check
      Returns:
      The element for which should be considered when sideoutputs

    • registerCleanupTimer

      protected void registerCleanupTimer(W window)
      Registers a timer to cleanup the content of the window.
      Parameters:
      window - the window whose state to discard

    • deleteCleanupTimer

      protected void deleteCleanupTimer(W window)
      Deletes the cleanup timer set for the contents of the provided window.
      Parameters:
      window - the window whose state to discard

    • isCleanupTime

      protected final boolean isCleanupTime(W window, long time)
      Returns true if the given time is the cleanup time for the given window.

    • getTrigger

      @VisibleForTesting public AsyncTrigger<? super IN,? super W> getTrigger()

    • getKeySelector

      @VisibleForTesting public org.apache.flink.api.java.functions.KeySelector<IN,K> getKeySelector()

    • getWindowAssigner

      @VisibleForTesting public WindowAssigner<? super IN,W> getWindowAssigner()

    • getStateDescriptor

      @VisibleForTesting public org.apache.flink.api.common.state.v2.StateDescriptor<?> getStateDescriptor()

    • getProcessContext

      @VisibleForTesting public AsyncWindowOperator<K,IN,ACC,OUT,W>.WindowContext getProcessContext()

    • getWindowAssignerContext

      @VisibleForTesting public WindowAssigner.WindowAssignerContext getWindowAssignerContext()