Package org.apache.flink.connector.file.table.stream.compact

Class CompactCoordinator

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
org.apache.flink.connector.file.table.stream.compact.CompactCoordinator
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorInput>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperator<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<CompactMessages.CoordinatorOutput>
@Internal public class CompactCoordinator extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,CompactMessages.CoordinatorOutput>
This is the single (non-parallel) monitoring task which coordinate input files to compaction units. - Receives in-flight input files inside checkpoint. - Receives all upstream end input messages after the checkpoint completes successfully, starts coordination.

CompactMessages.CompactionUnit and CompactMessages.EndCompaction must be sent to the downstream in an orderly manner, while CompactMessages.EndCompaction is broadcast emitting, so unit and endCompaction use the broadcast emitting mechanism together. Since unit is broadcast, we want it to be processed by a single task, so we carry the ID in the unit and let the downstream task select its own unit.

NOTE: The coordination is a stable algorithm, which can ensure that the downstream can perform compaction at any time without worrying about fail over.

STATE: This operator stores input files in state, after the checkpoint completes successfully, input files are taken out from the state for coordination.

See Also:

  • Field Summary

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    CompactCoordinator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory, long targetFileSize)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
     
    void
    processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorInput> element)
     
    void
    snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, close, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted, notifyCheckpointComplete

    Methods inherited from interface org.apache.flink.streaming.api.operators.Input

    processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

    setKeyContextElement

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

  • Constructor Details

    • CompactCoordinator

      public CompactCoordinator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory, long targetFileSize)

  • Method Details

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
      Throws:
      Exception

    • processElement

      public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorInput> element) throws Exception
      Specified by:
      processElement in interface org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorInput>
      Throws:
      Exception

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
      Throws:
      Exception