Package org.apache.flink.connector.base.source.reader

Class SourceReaderBase<E,T,SplitT extends org.apache.flink.api.connector.source.SourceSplit,SplitStateT>

java.lang.Object
org.apache.flink.connector.base.source.reader.SourceReaderBase<E,T,SplitT,SplitStateT>
Type Parameters:
E - The rich element type that contains information for split state update or timestamp extraction.
T - The final element type to emit.
SplitT - the immutable split type.
SplitStateT - the mutable type of split state.
All Implemented Interfaces:
AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<T,SplitT>
Direct Known Subclasses:
SingleThreadMultiplexSourceReaderBase
@PublicEvolving public abstract class SourceReaderBase<E,T,SplitT extends org.apache.flink.api.connector.source.SourceSplit,SplitStateT> extends Object implements org.apache.flink.api.connector.source.SourceReader<T,SplitT>
An abstract implementation of SourceReader which provides some synchronization between the mail box main thread and the SourceReader internal threads. This class allows user to just provide a SplitReader and snapshot the split state.

This implementation provides the following metrics out of the box:

  • OperatorIOMetricGroup.getNumRecordsInCounter()

  • Field Details

    • recordEmitter

      protected final RecordEmitter<E,T,SplitStateT> recordEmitter
      The record emitter to handle the records read by the SplitReaders.

    • splitFetcherManager

      protected final SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit> splitFetcherManager
      The split fetcher manager to run split fetchers.

    • options

      protected final SourceReaderOptions options
      The configuration for the reader.

    • config

      protected final org.apache.flink.configuration.Configuration config
      The raw configurations that may be used by subclasses.

    • context

      protected org.apache.flink.api.connector.source.SourceReaderContext context
      The context of this source reader.

    • eofRecordEvaluator

      @Nullable protected final RecordEvaluator<T> eofRecordEvaluator

  • Constructor Details

  • Method Details

    • start

      public void start()
      Specified by:
      start in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • pollNext

      public org.apache.flink.core.io.InputStatus pollNext(org.apache.flink.api.connector.source.ReaderOutput<T> output) throws Exception
      Specified by:
      pollNext in interface org.apache.flink.api.connector.source.SourceReader<E,T>
      Throws:
      Exception

    • isAvailable

      public CompletableFuture<Void> isAvailable()
      Specified by:
      isAvailable in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • snapshotState

      public List<SplitT> snapshotState(long checkpointId)
      Specified by:
      snapshotState in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • addSplits

      public void addSplits(List<SplitT> splits)
      Specified by:
      addSplits in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • notifyNoMoreSplits

      public void notifyNoMoreSplits()
      Specified by:
      notifyNoMoreSplits in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • handleSourceEvents

      public void handleSourceEvents(org.apache.flink.api.connector.source.SourceEvent sourceEvent)
      Specified by:
      handleSourceEvents in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • pauseOrResumeSplits

      public void pauseOrResumeSplits(Collection<String> splitsToPause, Collection<String> splitsToResume)
      Specified by:
      pauseOrResumeSplits in interface org.apache.flink.api.connector.source.SourceReader<E,T>

    • close

      public void close() throws Exception
      Specified by:
      close in interface AutoCloseable
      Throws:
      Exception

    • getNumberOfCurrentlyAssignedSplits

      public int getNumberOfCurrentlyAssignedSplits()
      Gets the number of splits the reads has currently assigned.

      These are the splits that have been added via addSplits(List) and have not yet been finished by returning them from the SplitReader.fetch() as part of RecordsWithSplitIds.finishedSplits().

    • onSplitFinished

      protected abstract void onSplitFinished(Map<String,SplitStateT> finishedSplitIds)
      Handles the finished splits to clean the state if needed.

    • initializedState

      protected abstract SplitStateT initializedState(SplitT split)
      When new splits are added to the reader. The initialize the state of the new splits.
      Parameters:
      split - a newly added split.

    • toSplitType

      protected abstract SplitT toSplitType(String splitId, SplitStateT splitState)
      Convert a mutable SplitStateT to immutable SplitT.
      Parameters:
      splitState - splitState.
      Returns:
      an immutable Split state.