Package org.apache.flink.fs.osshadoop.writer

Class OSSRecoverableWriter

java.lang.Object
org.apache.flink.fs.osshadoop.writer.OSSRecoverableWriter
All Implemented Interfaces:
org.apache.flink.core.fs.RecoverableWriter
public class OSSRecoverableWriter extends Object implements org.apache.flink.core.fs.RecoverableWriter
An implementation of the RecoverableWriter against OSS.

This implementation makes heavy use of MultiPart Uploads in OSS to persist intermediate data as soon as possible.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

    org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable

  • Constructor Summary

    Constructors
    Constructor
    Description
    OSSRecoverableWriter(OSSAccessor ossAccessor, long ossUploadPartSize, int streamConcurrentUploads, Executor executor, org.apache.flink.util.function.FunctionWithException<File,org.apache.flink.core.fs.RefCountedFileWithStream,IOException> cachedFileCreator)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
     
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable>
    getCommitRecoverableSerializer()
     
    org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable>
    getResumeRecoverableSerializer()
     
    org.apache.flink.core.fs.RecoverableFsDataOutputStream
    open(org.apache.flink.core.fs.Path path)
     
    OSSRecoverableFsDataOutputStream
    recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
     
    org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer
    recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable resumable)
     
    boolean
    requiresCleanupOfRecoverableState()
     
    boolean
    supportsResume()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • OSSRecoverableWriter

      public OSSRecoverableWriter(OSSAccessor ossAccessor, long ossUploadPartSize, int streamConcurrentUploads, Executor executor, org.apache.flink.util.function.FunctionWithException<File,org.apache.flink.core.fs.RefCountedFileWithStream,IOException> cachedFileCreator)

  • Method Details

    • open

      public org.apache.flink.core.fs.RecoverableFsDataOutputStream open(org.apache.flink.core.fs.Path path) throws IOException
      Specified by:
      open in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • recover

      public OSSRecoverableFsDataOutputStream recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable) throws IOException
      Specified by:
      recover in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • requiresCleanupOfRecoverableState

      public boolean requiresCleanupOfRecoverableState()
      Specified by:
      requiresCleanupOfRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

    • cleanupRecoverableState

      public boolean cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable) throws IOException
      Specified by:
      cleanupRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • recoverForCommit

      public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable resumable) throws IOException
      Specified by:
      recoverForCommit in interface org.apache.flink.core.fs.RecoverableWriter
      Throws:
      IOException

    • getCommitRecoverableSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()
      Specified by:
      getCommitRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

    • getResumeRecoverableSerializer

      public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()
      Specified by:
      getResumeRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

    • supportsResume

      public boolean supportsResume()
      Specified by:
      supportsResume in interface org.apache.flink.core.fs.RecoverableWriter