Package org.apache.flink.connector.file.sink.compactor.operator

Class CompactorOperator

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
org.apache.flink.connector.file.sink.compactor.operator.CompactorOperator
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<CompactorRequest>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
@Internal public class CompactorOperator extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.api.common.state.CheckpointListener
An operator that perform compaction for the FileSink.

Requests received from the CompactCoordinator will firstly be held in memory, and snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all requests received before can be submitted. The results can be emitted at the next prepareSnapshotPreBarrier(long) invoking after the compaction is finished, to ensure that committers can receive only one CommittableSummary and the corresponding number of Committable for a single checkpoint.

See Also:

  • Field Summary

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    CompactorOperator(org.apache.flink.streaming.api.operators.StreamOperatorParameters<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> parameters, FileCompactStrategy strategy, org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer, FileCompactor fileCompactor, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,String> bucketWriter)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    void
    endInput()
     
    CompletableFuture<?>
    getAllTasksFuture()
     
    void
    initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
     
    void
    notifyCheckpointComplete(long checkpointId)
     
    void
    open()
     
    void
    prepareSnapshotPreBarrier(long checkpointId)
     
    void
    processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element)
     
    void
    snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted

    Methods inherited from interface org.apache.flink.streaming.api.operators.Input

    processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

    setKeyContextElement

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

  • Constructor Details

    • CompactorOperator

      public CompactorOperator(org.apache.flink.streaming.api.operators.StreamOperatorParameters<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> parameters, FileCompactStrategy strategy, org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer, FileCompactor fileCompactor, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,String> bucketWriter)

  • Method Details

    • open

      public void open() throws Exception
      Specified by:
      open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Overrides:
      open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • processElement

      public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element) throws Exception
      Specified by:
      processElement in interface org.apache.flink.streaming.api.operators.Input<CompactorRequest>
      Throws:
      Exception

    • endInput

      public void endInput() throws Exception
      Specified by:
      endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
      Throws:
      Exception

    • notifyCheckpointComplete

      public void notifyCheckpointComplete(long checkpointId) throws Exception
      Specified by:
      notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
      Overrides:
      notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • prepareSnapshotPreBarrier

      public void prepareSnapshotPreBarrier(long checkpointId) throws Exception
      Specified by:
      prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Overrides:
      prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Overrides:
      close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
      Throws:
      Exception

    • getAllTasksFuture

      @VisibleForTesting public CompletableFuture<?> getAllTasksFuture()