Package org.apache.flink.connector.file.src

Class AbstractFileSource<T,SplitT extends FileSourceSplit>

java.lang.Object

org.apache.flink.connector.file.src.AbstractFileSource<T,SplitT>

Type Parameters:

T - The type of the events/records produced by this source.

SplitT - The subclass type of the FileSourceSplit used by the source implementation.

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<T,SplitT,PendingSplitsCheckpoint<SplitT>>, org.apache.flink.api.connector.source.SourceReaderFactory<T,SplitT>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

Direct Known Subclasses:

FileSource

@PublicEvolving
public abstract class AbstractFileSource<T,SplitT extends FileSourceSplit>
extends Object
implements org.apache.flink.api.connector.source.Source<T,SplitT,PendingSplitsCheckpoint<SplitT>>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

The base class for File Sources. The main implementation to use is the FileSource, which also has the majority of the documentation.

To read new formats, one commonly does NOT need to extend this class, but should implement a new Format Reader (like StreamFormat, BulkFormat and use it with the FileSource.

The only reason to extend this class is when a source needs a different type of split, meaning an extension of the FileSourceSplit to carry additional information.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	AbstractFileSource.AbstractFileSourceBuilder<T,SplitT extends FileSourceSplit,SELF extends AbstractFileSource.AbstractFileSourceBuilder<T,SplitT,SELF>>	The generic base builder.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	AbstractFileSource(org.apache.flink.core.fs.Path[] inputPaths, FileEnumerator.Provider fileEnumerator, FileSplitAssigner.Provider splitAssigner, BulkFormat<T,SplitT> readerFormat, ContinuousEnumerationSettings continuousEnumerationSettings)

Method Summary

Modifier and Type	Method	Description
org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> enumContext)
org.apache.flink.api.connector.source.SourceReader<T,SplitT>	createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)
FileSplitAssigner.Provider	getAssignerFactory()
org.apache.flink.api.connector.source.Boundedness	getBoundedness()
ContinuousEnumerationSettings	getContinuousEnumerationSettings()
org.apache.flink.core.io.SimpleVersionedSerializer<PendingSplitsCheckpoint<SplitT>>	getEnumeratorCheckpointSerializer()
protected FileEnumerator.Provider	getEnumeratorFactory()
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
abstract org.apache.flink.core.io.SimpleVersionedSerializer<SplitT>	getSplitSerializer()
org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>>	restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> enumContext, PendingSplitsCheckpoint<SplitT> checkpoint)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.Source

declareWatermarks

Constructor Details

AbstractFileSource

protected AbstractFileSource(org.apache.flink.core.fs.Path[] inputPaths,
 FileEnumerator.Provider fileEnumerator,
 FileSplitAssigner.Provider splitAssigner,
 BulkFormat<T,SplitT> readerFormat,
 @Nullable
 ContinuousEnumerationSettings continuousEnumerationSettings)

Method Details

getEnumeratorFactory

protected FileEnumerator.Provider getEnumeratorFactory()

getAssignerFactory

public FileSplitAssigner.Provider getAssignerFactory()

getContinuousEnumerationSettings

@Nullable
public ContinuousEnumerationSettings getContinuousEnumerationSettings()

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

Specified by:

getBoundedness in interface org.apache.flink.api.connector.source.Source<T,SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

createReader

public org.apache.flink.api.connector.source.SourceReader<T,SplitT> createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)

Specified by:

createReader in interface org.apache.flink.api.connector.source.SourceReaderFactory<T,SplitT extends FileSourceSplit>

createEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> enumContext)

Specified by:

createEnumerator in interface org.apache.flink.api.connector.source.Source<T,SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

restoreEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> enumContext,
 PendingSplitsCheckpoint<SplitT> checkpoint)

Specified by:

restoreEnumerator in interface org.apache.flink.api.connector.source.Source<T,SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

getSplitSerializer

public abstract org.apache.flink.core.io.SimpleVersionedSerializer<SplitT> getSplitSerializer()

Specified by:

getSplitSerializer in interface org.apache.flink.api.connector.source.Source<T,SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

getEnumeratorCheckpointSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<PendingSplitsCheckpoint<SplitT>> getEnumeratorCheckpointSerializer()

Specified by:

getEnumeratorCheckpointSerializer in interface org.apache.flink.api.connector.source.Source<T,SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>