Package org.apache.flink.connector.base.source.reader.fetcher

Class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>
Direct Known Subclasses:
SingleThreadFetcherManager
@PublicEvolving public abstract class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit> extends Object
A class responsible for starting the SplitFetcher and manage the life cycles of them. This class works with the SourceReaderBase.

The split fetcher manager could be used to support different threading models by implementing the addSplits(List) method differently. For example, a single thread split fetcher manager would only start a single fetcher and assign all the splits to it. A one-thread-per-split fetcher may spawn a new thread every time a new split is assigned.

  • Field Details

    • fetchers

      protected final Map<Integer,SplitFetcher<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>> fetchers
      A map keeping track of all the split fetchers.

  • Constructor Details

    • SplitFetcherManager

      public SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration)
      Create a split fetcher manager.
      Parameters:
      splitReaderFactory - a supplier that could be used to create split readers.
      configuration - the configuration of this fetcher manager.

    • SplitFetcherManager

      public SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration, Consumer<Collection<String>> splitFinishedHook)
      Create a split fetcher manager.
      Parameters:
      splitReaderFactory - a supplier that could be used to create split readers.
      configuration - the configuration of this fetcher manager.
      splitFinishedHook - Hook for handling finished splits in split fetchers.

  • Method Details

    • addSplits

      public abstract void addSplits(List<SplitT> splitsToAdd)

    • removeSplits

      public abstract void removeSplits(List<SplitT> splitsToRemove)

    • pauseOrResumeSplits

      public void pauseOrResumeSplits(Collection<String> splitIdsToPause, Collection<String> splitIdsToResume)

    • startFetcher

      protected void startFetcher(SplitFetcher<E,SplitT> fetcher)

    • createSplitFetcher

      protected SplitFetcher<E,SplitT> createSplitFetcher()
      Synchronize method to ensure no fetcher is created after the split fetcher manager has closed.
      Returns:
      the created split fetcher.
      Throws:
      IllegalStateException - if the split fetcher manager has closed.

    • maybeShutdownFinishedFetchers

      public boolean maybeShutdownFinishedFetchers()
      Check and shutdown the fetchers that have completed their work.
      Returns:
      true if all the fetchers have completed the work, false otherwise.

    • getQueue

      @Internal public FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> getQueue()
      Return the queue contains data produced by split fetchers.This method is Internal and only used in SourceReaderBase.

    • close

      public void close(long timeoutMs) throws Exception
      Close the split fetcher manager.
      Parameters:
      timeoutMs - the max time in milliseconds to wait.
      Throws:
      Exception - when failed to close the split fetcher manager.

    • checkErrors

      public void checkErrors()

    • getNumAliveFetchers

      @VisibleForTesting public int getNumAliveFetchers()