Package org.apache.flink.table.runtime.operators.bundle

Class AbstractMapBundleOperator<K,V,IN,OUT>

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator<K,V,IN,OUT>
Type Parameters:
K - The type of the key in the bundle map
V - The type of the value in the bundle map
IN - Input type for the operator.
OUT - Output type for the operator.
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>, BundleTriggerCallback
Direct Known Subclasses:
KeyedMapBundleOperator, MapBundleOperator
public abstract class AbstractMapBundleOperator<K,V,IN,OUT> extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, BundleTriggerCallback
The AbstractMapBundleOperator simply used a java Map to store the input elements in key-value form. The map key is typically the same with the state key, so we can do some optimizations before accessing states, like pre aggregate values for each key. And we will only need to access state every key we have, but not every element we processed.

NOTES: if all elements we processed have different keys, such operator will only increase memory footprint, and will not have any performance improvement.

See Also:

  • Field Summary

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    void
    finish()
     
    void
    finishBundle()
    This method is invoked to finish current bundle and start a new one when the trigger was fired.
    protected abstract K
    getKey(IN input)
    Get the key for current processing element, which will be used as the map bundle's key.
    void
    open()
     
    void
    prepareSnapshotPreBarrier(long checkpointId)
     
    void
    processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
     
    void
    processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted, notifyCheckpointComplete

    Methods inherited from interface org.apache.flink.streaming.api.operators.Input

    processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

    setKeyContextElement

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

  • Method Details

    • open

      public void open() throws Exception
      Specified by:
      open in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
      Overrides:
      open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • processElement

      public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element) throws Exception
      Specified by:
      processElement in interface org.apache.flink.streaming.api.operators.Input<K>
      Throws:
      Exception

    • getKey

      protected abstract K getKey(IN input) throws Exception
      Get the key for current processing element, which will be used as the map bundle's key.
      Throws:
      Exception

    • finishBundle

      public void finishBundle() throws Exception
      Description copied from interface: BundleTriggerCallback
      This method is invoked to finish current bundle and start a new one when the trigger was fired.
      Specified by:
      finishBundle in interface BundleTriggerCallback
      Throws:
      Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

    • processWatermark

      public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark) throws Exception
      Specified by:
      processWatermark in interface org.apache.flink.streaming.api.operators.Input<K>
      Overrides:
      processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • prepareSnapshotPreBarrier

      public void prepareSnapshotPreBarrier(long checkpointId) throws Exception
      Specified by:
      prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
      Overrides:
      prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • finish

      public void finish() throws Exception
      Specified by:
      finish in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
      Overrides:
      finish in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface org.apache.flink.streaming.api.operators.StreamOperator<K>
      Overrides:
      close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception