Package org.apache.flink.runtime.fs.hdfs

Class HadoopRecoverableWriter

java.lang.Object
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter
All Implemented Interfaces:
org.apache.flink.core.fs.RecoverableWriter
@Internal public class HadoopRecoverableWriter extends Object implements org.apache.flink.core.fs.RecoverableWriter
An implementation of the RecoverableWriter for Hadoop's file system abstraction.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

    org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected final org.apache.hadoop.fs.FileSystem
    fs
    The Hadoop file system on which the writer operates.

  • Constructor Summary

    Constructors
    Constructor
    Description
    HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs)
    Creates a new Recoverable writer.
    HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs, boolean noLocalWrite)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected void
    checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs)
     
    boolean
    cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
     
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable>
    getCommitRecoverableSerializer()
     
    protected org.apache.flink.core.fs.RecoverableFsDataOutputStream
    getRecoverableFsDataOutputStream(org.apache.hadoop.fs.Path targetFile, org.apache.hadoop.fs.Path tempFile)
     
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable>
    getResumeRecoverableSerializer()
     
    org.apache.flink.core.fs.RecoverableFsDataOutputStream
    open(org.apache.flink.core.fs.Path filePath)
     
    org.apache.flink.core.fs.RecoverableFsDataOutputStream
    recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable)
     
    org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer
    recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable)
     
    boolean
    requiresCleanupOfRecoverableState()
     
    boolean
    supportsResume()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • fs

      protected final org.apache.hadoop.fs.FileSystem fs
      The Hadoop file system on which the writer operates.

  • Constructor Details

    • HadoopRecoverableWriter

      public HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs)
      Creates a new Recoverable writer.
      Parameters:
      fs - The Hadoop file system on which the writer operates.

    • HadoopRecoverableWriter

      public HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs, boolean noLocalWrite)

  • Method Details

    • checkSupportedFSSchemes

      protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs)

    • open

      public org.apache.flink.core.fs.RecoverableFsDataOutputStream open(org.apache.flink.core.fs.Path filePath) throws IOException
      Specified by:
      open in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • getRecoverableFsDataOutputStream

      protected org.apache.flink.core.fs.RecoverableFsDataOutputStream getRecoverableFsDataOutputStream(org.apache.hadoop.fs.Path targetFile, org.apache.hadoop.fs.Path tempFile) throws IOException
      Throws:
      IOException

    • recover

      public org.apache.flink.core.fs.RecoverableFsDataOutputStream recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable) throws IOException
      Specified by:
      recover in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • requiresCleanupOfRecoverableState

      public boolean requiresCleanupOfRecoverableState()
      Specified by:
      requiresCleanupOfRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

    • cleanupRecoverableState

      public boolean cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable) throws IOException
      Specified by:
      cleanupRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • recoverForCommit

      public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable) throws IOException
      Specified by:
      recoverForCommit in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • getCommitRecoverableSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()
      Specified by:
      getCommitRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

    • getResumeRecoverableSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()
      Specified by:
      getResumeRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

    • supportsResume

      public boolean supportsResume()
      Specified by:
      supportsResume in interface org.apache.flink.core.fs.RecoverableWriter