Package org.apache.flink.connector.file.table.stream

Class AbstractStreamingWriter<IN,OUT>

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
org.apache.flink.connector.file.table.stream.AbstractStreamingWriter<IN,OUT>
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>
Direct Known Subclasses:
CompactFileWriter, StreamingFileWriter
public abstract class AbstractStreamingWriter<IN,OUT> extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.BoundedOneInput
Operator for file system sink. It is a operator version of StreamingFileSink. It can send file and bucket information to downstream.
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,String>
    buckets
     
    protected long
    currentWatermark
     

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    AbstractStreamingWriter(long bucketCheckInterval, org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,?>> bucketsBuilder)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    protected void
    commitUpToCheckpoint(long checkpointId)
    Commit up to this checkpoint id.
    void
    endInput()
     
    void
    initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
     
    void
    notifyCheckpointComplete(long checkpointId)
     
    protected abstract void
    onPartFileOpened(String partition, org.apache.flink.core.fs.Path newPath)
    Notifies a new file has been opened.
    protected abstract void
    partitionCreated(String partition)
    Notifies a partition created.
    protected abstract void
    partitionInactive(String partition)
    Notifies a partition become inactive.
    void
    processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
     
    void
    processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
     
    void
    snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted

    Methods inherited from interface org.apache.flink.streaming.api.operators.Input

    processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

    setKeyContextElement

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

  • Field Details

    • buckets

      protected transient org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,String> buckets

    • currentWatermark

      protected transient long currentWatermark

  • Constructor Details

    • AbstractStreamingWriter

      public AbstractStreamingWriter(long bucketCheckInterval, org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,?>> bucketsBuilder)

  • Method Details

    • partitionCreated

      protected abstract void partitionCreated(String partition)
      Notifies a partition created.

    • partitionInactive

      protected abstract void partitionInactive(String partition)
      Notifies a partition become inactive. A partition becomes inactive after all the records received so far have been committed.

    • onPartFileOpened

      protected abstract void onPartFileOpened(String partition, org.apache.flink.core.fs.Path newPath)
      Notifies a new file has been opened.

      Note that this does not mean that the file has been created in the file system. It is only created logically and the actual file will be generated after it is committed.

    • commitUpToCheckpoint

      protected void commitUpToCheckpoint(long checkpointId) throws Exception
      Commit up to this checkpoint id.
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • processWatermark

      public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark) throws Exception
      Specified by:
      processWatermark in interface org.apache.flink.streaming.api.operators.Input<IN>
      Overrides:
      processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • processElement

      public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element) throws Exception
      Specified by:
      processElement in interface org.apache.flink.streaming.api.operators.Input<IN>
      Throws:
      Exception

    • notifyCheckpointComplete

      public void notifyCheckpointComplete(long checkpointId) throws Exception
      Specified by:
      notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
      Overrides:
      notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • endInput

      public void endInput() throws Exception
      Specified by:
      endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>
      Overrides:
      close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception