Package org.apache.flink.runtime.fs.hdfs

Class HadoopFileSystem

java.lang.Object
org.apache.flink.core.fs.FileSystem
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem
All Implemented Interfaces:
org.apache.flink.core.fs.IFileSystem
public class HadoopFileSystem extends org.apache.flink.core.fs.FileSystem
A FileSystem that wraps an Hadoop File System.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

    org.apache.flink.core.fs.FileSystem.FSKey, org.apache.flink.core.fs.FileSystem.WriteMode

  • Constructor Summary

    Constructors
    Constructor
    Description
    HadoopFileSystem(org.apache.hadoop.fs.FileSystem hadoopFileSystem)
    Wraps the given Hadoop File System object as a Flink File System object.

  • Method Summary

    Modifier and Type
    Method
    Description
    HadoopDataOutputStream
    create(org.apache.flink.core.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
     
    HadoopDataOutputStream
    create(org.apache.flink.core.fs.Path f, org.apache.flink.core.fs.FileSystem.WriteMode overwrite)
     
    org.apache.flink.core.fs.RecoverableWriter
    createRecoverableWriter()
     
    org.apache.flink.core.fs.RecoverableWriter
    createRecoverableWriter(Map<String,String> conf)
     
    boolean
    delete(org.apache.flink.core.fs.Path f, boolean recursive)
     
    boolean
    exists(org.apache.flink.core.fs.Path f)
     
    long
    getDefaultBlockSize()
     
    org.apache.flink.core.fs.BlockLocation[]
    getFileBlockLocations(org.apache.flink.core.fs.FileStatus file, long start, long len)
     
    org.apache.flink.core.fs.FileStatus
    getFileStatus(org.apache.flink.core.fs.Path f)
     
    org.apache.hadoop.fs.FileSystem
    getHadoopFileSystem()
    Gets the underlying Hadoop FileSystem.
    org.apache.flink.core.fs.Path
    getHomeDirectory()
     
    URI
    getUri()
     
    org.apache.flink.core.fs.Path
    getWorkingDirectory()
     
    boolean
    isDistributedFS()
     
    org.apache.flink.core.fs.FileStatus[]
    listStatus(org.apache.flink.core.fs.Path f)
     
    boolean
    mkdirs(org.apache.flink.core.fs.Path f)
     
    HadoopDataInputStream
    open(org.apache.flink.core.fs.Path f)
     
    HadoopDataInputStream
    open(org.apache.flink.core.fs.Path f, int bufferSize)
     
    boolean
    rename(org.apache.flink.core.fs.Path src, org.apache.flink.core.fs.Path dst)
     
    static org.apache.hadoop.fs.Path
    toHadoopPath(org.apache.flink.core.fs.Path path)
     

    Methods inherited from class org.apache.flink.core.fs.FileSystem

    create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initialize, initOutPathDistFS, initOutPathLocalFS

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.core.fs.IFileSystem

    canCopyPaths

  • Constructor Details

    • HadoopFileSystem

      public HadoopFileSystem(org.apache.hadoop.fs.FileSystem hadoopFileSystem)
      Wraps the given Hadoop File System object as a Flink File System object. The given Hadoop file system object is expected to be initialized already.
      Parameters:
      hadoopFileSystem - The Hadoop FileSystem that will be used under the hood.

  • Method Details

    • getHadoopFileSystem

      public org.apache.hadoop.fs.FileSystem getHadoopFileSystem()
      Gets the underlying Hadoop FileSystem.
      Returns:
      The underlying Hadoop FileSystem.

    • getWorkingDirectory

      public org.apache.flink.core.fs.Path getWorkingDirectory()
      Specified by:
      getWorkingDirectory in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      getWorkingDirectory in class org.apache.flink.core.fs.FileSystem

    • getHomeDirectory

      public org.apache.flink.core.fs.Path getHomeDirectory()
      Specified by:
      getHomeDirectory in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      getHomeDirectory in class org.apache.flink.core.fs.FileSystem

    • getUri

      public URI getUri()
      Specified by:
      getUri in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      getUri in class org.apache.flink.core.fs.FileSystem

    • getFileStatus

      public org.apache.flink.core.fs.FileStatus getFileStatus(org.apache.flink.core.fs.Path f) throws IOException
      Specified by:
      getFileStatus in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      getFileStatus in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • getFileBlockLocations

      public org.apache.flink.core.fs.BlockLocation[] getFileBlockLocations(org.apache.flink.core.fs.FileStatus file, long start, long len) throws IOException
      Specified by:
      getFileBlockLocations in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      getFileBlockLocations in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • open

      public HadoopDataInputStream open(org.apache.flink.core.fs.Path f, int bufferSize) throws IOException
      Specified by:
      open in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      open in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • open

      public HadoopDataInputStream open(org.apache.flink.core.fs.Path f) throws IOException
      Specified by:
      open in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      open in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • create

      public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException
      Overrides:
      create in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • create

      public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f, org.apache.flink.core.fs.FileSystem.WriteMode overwrite) throws IOException
      Specified by:
      create in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      create in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • delete

      public boolean delete(org.apache.flink.core.fs.Path f, boolean recursive) throws IOException
      Specified by:
      delete in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      delete in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • exists

      public boolean exists(org.apache.flink.core.fs.Path f) throws IOException
      Specified by:
      exists in interface org.apache.flink.core.fs.IFileSystem
      Overrides:
      exists in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • listStatus

      public org.apache.flink.core.fs.FileStatus[] listStatus(org.apache.flink.core.fs.Path f) throws IOException
      Specified by:
      listStatus in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      listStatus in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • mkdirs

      public boolean mkdirs(org.apache.flink.core.fs.Path f) throws IOException
      Specified by:
      mkdirs in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      mkdirs in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • rename

      public boolean rename(org.apache.flink.core.fs.Path src, org.apache.flink.core.fs.Path dst) throws IOException
      Specified by:
      rename in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      rename in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • getDefaultBlockSize

      public long getDefaultBlockSize()
      Overrides:
      getDefaultBlockSize in class org.apache.flink.core.fs.FileSystem

    • isDistributedFS

      public boolean isDistributedFS()
      Specified by:
      isDistributedFS in interface org.apache.flink.core.fs.IFileSystem
      Specified by:
      isDistributedFS in class org.apache.flink.core.fs.FileSystem

    • createRecoverableWriter

      public org.apache.flink.core.fs.RecoverableWriter createRecoverableWriter() throws IOException
      Specified by:
      createRecoverableWriter in interface org.apache.flink.core.fs.IFileSystem
      Overrides:
      createRecoverableWriter in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • createRecoverableWriter

      public org.apache.flink.core.fs.RecoverableWriter createRecoverableWriter(Map<String,String> conf) throws IOException
      Specified by:
      createRecoverableWriter in interface org.apache.flink.core.fs.IFileSystem
      Overrides:
      createRecoverableWriter in class org.apache.flink.core.fs.FileSystem
      Throws:
      IOException

    • toHadoopPath

      public static org.apache.hadoop.fs.Path toHadoopPath(org.apache.flink.core.fs.Path path)