Package org.apache.flink.formats.avro

Class AvroInputFormat<E>

java.lang.Object
org.apache.flink.api.common.io.RichInputFormat<OT,org.apache.flink.core.fs.FileInputSplit>
org.apache.flink.api.common.io.FileInputFormat<E>
org.apache.flink.formats.avro.AvroInputFormat<E>
Type Parameters:
E - the type of the result Avro record. If you specify GenericRecord then the result will be returned as a GenericRecord, so you do not have to know the schema ahead of time.
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>, org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>
public class AvroInputFormat<E> extends org.apache.flink.api.common.io.FileInputFormat<E> implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>
Provides a FileInputFormat for Avro records.
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

    org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

  • Field Summary

    Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

    currentSplit, enumerateNestedFiles, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

  • Constructor Summary

    Constructors
    Constructor
    Description
    AvroInputFormat(org.apache.flink.core.fs.Path filePath, Class<E> type)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.flink.api.java.tuple.Tuple2<Long,Long>
    getCurrentState()
     
    org.apache.flink.api.common.typeinfo.TypeInformation<E>
    getProducedType()
     
    long
    getRecordsReadFromBlock()
     
    E
    nextRecord(E reuseValue)
     
    void
    open(org.apache.flink.core.fs.FileInputSplit split)
     
    boolean
    reachedEnd()
     
    void
    reopen(org.apache.flink.core.fs.FileInputSplit split, org.apache.flink.api.java.tuple.Tuple2<Long,Long> state)
     
    void
    setReuseAvroValue(boolean reuseAvroValue)
    Sets the flag whether to reuse the Avro value instance for all records.
    void
    setUnsplittable(boolean unsplittable)
    If set, the InputFormat will only read entire files.

    Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

    acceptFile, close, configure, createInputSplits, decorateInputStream, extractFileExtension, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, getSupportedCompressionFormats, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

    Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

    closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • AvroInputFormat

      public AvroInputFormat(org.apache.flink.core.fs.Path filePath, Class<E> type)

  • Method Details

    • setReuseAvroValue

      public void setReuseAvroValue(boolean reuseAvroValue)
      Sets the flag whether to reuse the Avro value instance for all records. By default, the input format reuses the Avro value.
      Parameters:
      reuseAvroValue - True, if the input format should reuse the Avro value instance, false otherwise.

    • setUnsplittable

      public void setUnsplittable(boolean unsplittable)
      If set, the InputFormat will only read entire files.

    • getProducedType

      public org.apache.flink.api.common.typeinfo.TypeInformation<E> getProducedType()
      Specified by:
      getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<E>

    • open

      public void open(org.apache.flink.core.fs.FileInputSplit split) throws IOException
      Specified by:
      open in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>
      Overrides:
      open in class org.apache.flink.api.common.io.FileInputFormat<E>
      Throws:
      IOException

    • reachedEnd

      public boolean reachedEnd() throws IOException
      Specified by:
      reachedEnd in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>
      Throws:
      IOException

    • getRecordsReadFromBlock

      public long getRecordsReadFromBlock()

    • nextRecord

      public E nextRecord(E reuseValue) throws IOException
      Specified by:
      nextRecord in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>
      Throws:
      IOException

    • getCurrentState

      public org.apache.flink.api.java.tuple.Tuple2<Long,Long> getCurrentState() throws IOException
      Specified by:
      getCurrentState in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>
      Throws:
      IOException

    • reopen

      public void reopen(org.apache.flink.core.fs.FileInputSplit split, org.apache.flink.api.java.tuple.Tuple2<Long,Long> state) throws IOException
      Specified by:
      reopen in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>
      Throws:
      IOException