Package com.mapr.fs

Class MFS

java.lang.Object
org.apache.hadoop.fs.AbstractFileSystem
com.mapr.fs.MFS
All Implemented Interfaces:
org.apache.hadoop.fs.PathCapabilities
public class MFS extends org.apache.hadoop.fs.AbstractFileSystem
This class is an implementation of AbstractFileSystem, as a wrapper class around MapRFileSystem. It allows the new Yarn APIs to run on top of MapRFS.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final org.apache.commons.logging.Log
    LOG
     

    Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

    statistics

  • Constructor Summary

    Constructors
    Constructor
    Description
    MFS()
    A no-args constructor that is required for reflection.
    MFS(URI uri, org.apache.hadoop.conf.Configuration conf)
    Calls super constructor, with parameter authorityNeeded=false and initializes maprfs with uri and conf.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    checkPath(org.apache.hadoop.fs.Path path)
    Takes the string component of path from the first '/' and appends it to the path of the URI before calling super.checkPath(Path).
    org.apache.hadoop.fs.FSDataOutputStream
    createInternal(org.apache.hadoop.fs.Path p, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
    Creates internal file with indicated permission, bufferSize, replication, blockSize etc.
    void
    createSymlink(org.apache.hadoop.fs.Path target, org.apache.hadoop.fs.Path link, boolean createParent)
    The specification of this method matches that of FileContext#createSymlink(Path, Path, boolean);
    boolean
    delete(org.apache.hadoop.fs.Path f, boolean recursive)
     
    org.apache.hadoop.fs.BlockLocation[]
    getFileBlockLocations(org.apache.hadoop.fs.Path p, long start, long len)
     
    org.apache.hadoop.fs.FileChecksum
    getFileChecksum(org.apache.hadoop.fs.Path path)
     
    org.apache.hadoop.fs.FileStatus
    getFileStatus(org.apache.hadoop.fs.Path p)
     
    org.apache.hadoop.fs.FsStatus
    getFsStatus()
     
    org.apache.hadoop.fs.FsServerDefaults
    getServerDefaults()
    Returns default settings for server.
    int
    getUriDefaultPort()
     
    org.apache.hadoop.fs.FileStatus[]
    listStatus(org.apache.hadoop.fs.Path p)
     
    void
    mkdir(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)
    Creates a directory at the specified path with specified permission.
    org.apache.hadoop.fs.FSDataInputStream
    open(org.apache.hadoop.fs.Path p, int bufferSize)
     
    void
    renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
     
    void
    setOwner(org.apache.hadoop.fs.Path p, String user, String group)
     
    void
    setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
     
    boolean
    setReplication(org.apache.hadoop.fs.Path p, short replication)
     
    void
    setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
     
    void
    setVerifyChecksum(boolean verifyChecksum)
     
    boolean
    supportsSymlinks()
    Returns true if the file system supports symlinks, false otherwise.

    Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

    access, checkScheme, clearStatistics, create, createFileSystem, createMultipartUploader, createSnapshot, deleteSnapshot, equals, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getEnclosingRoot, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getStatistics, getStatistics, getStoragePolicy, getUri, getUriPath, getXAttr, getXAttrs, getXAttrs, hashCode, hasPathCapability, isValidName, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, listXAttrs, makeQualified, methodNotSupported, modifyAclEntries, msync, open, openFileWithOptions, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameInternal, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setStoragePolicy, setXAttr, setXAttr, truncate, unsetStoragePolicy

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • LOG

      public static final org.apache.commons.logging.Log LOG

  • Constructor Details

  • Method Details

    • createInternal

      public org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path p, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent) throws org.apache.hadoop.security.AccessControlException, org.apache.hadoop.fs.FileAlreadyExistsException, FileNotFoundException, org.apache.hadoop.fs.ParentNotDirectoryException, org.apache.hadoop.fs.UnsupportedFileSystemException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Creates internal file with indicated permission, bufferSize, replication, blockSize etc. Ignores checksumOpt and createParent variables.
      Specified by:
      createInternal in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      org.apache.hadoop.fs.FileAlreadyExistsException
      FileNotFoundException
      org.apache.hadoop.fs.ParentNotDirectoryException
      org.apache.hadoop.fs.UnsupportedFileSystemException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException
      See Also:
      • TODO determine user of ChecksumOpt (Spencer)

    • checkPath

      public void checkPath(org.apache.hadoop.fs.Path path)
      Takes the string component of path from the first '/' and appends it to the path of the URI before calling super.checkPath(Path). This works around an issue where the path contains no host information causing jobs to fail.
      Overrides:
      checkPath in class org.apache.hadoop.fs.AbstractFileSystem
      Parameters:
      path - The path to be checked TODO use only super.checkPath() instead of this modified version that manually inserts the URI (Spencer)

    • delete

      public boolean delete(org.apache.hadoop.fs.Path f, boolean recursive) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      delete in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • getFileBlockLocations

      public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path p, long start, long len) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      getFileBlockLocations in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • getFileChecksum

      public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path path) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      getFileChecksum in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • getFileStatus

      public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path p) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      getFileStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • getFsStatus

      public org.apache.hadoop.fs.FsStatus getFsStatus() throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, IOException
      Specified by:
      getFsStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      IOException

    • getServerDefaults

      public org.apache.hadoop.fs.FsServerDefaults getServerDefaults() throws IOException
      Returns default settings for server.
      Specified by:
      getServerDefaults in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException

    • getUriDefaultPort

      public int getUriDefaultPort()
      Specified by:
      getUriDefaultPort in class org.apache.hadoop.fs.AbstractFileSystem
      Returns:
      the current port stored in URI of maprfs.

    • listStatus

      public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path p) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      listStatus in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • mkdir

      public void mkdir(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent) throws org.apache.hadoop.security.AccessControlException, org.apache.hadoop.fs.FileAlreadyExistsException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Creates a directory at the specified path with specified permission. Currently ignores the createParent option, maprfs will always create parents.
      Specified by:
      mkdir in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      org.apache.hadoop.fs.FileAlreadyExistsException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • open

      public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path p, int bufferSize) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      open in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • supportsSymlinks

      public boolean supportsSymlinks()
      Returns true if the file system supports symlinks, false otherwise.
      Overrides:
      supportsSymlinks in class org.apache.hadoop.fs.AbstractFileSystem
      Returns:
      true if filesystem supports symlinks

    • createSymlink

      public void createSymlink(org.apache.hadoop.fs.Path target, org.apache.hadoop.fs.Path link, boolean createParent) throws IOException, org.apache.hadoop.fs.UnresolvedLinkException
      The specification of this method matches that of FileContext#createSymlink(Path, Path, boolean);
      Overrides:
      createSymlink in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      IOException
      org.apache.hadoop.fs.UnresolvedLinkException

    • renameInternal

      public void renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws org.apache.hadoop.security.AccessControlException, org.apache.hadoop.fs.FileAlreadyExistsException, FileNotFoundException, org.apache.hadoop.fs.ParentNotDirectoryException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      renameInternal in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      org.apache.hadoop.fs.FileAlreadyExistsException
      FileNotFoundException
      org.apache.hadoop.fs.ParentNotDirectoryException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • setOwner

      public void setOwner(org.apache.hadoop.fs.Path p, String user, String group) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      setOwner in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • setPermission

      public void setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      setPermission in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • setReplication

      public boolean setReplication(org.apache.hadoop.fs.Path p, short replication) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      setReplication in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • setTimes

      public void setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime) throws org.apache.hadoop.security.AccessControlException, FileNotFoundException, org.apache.hadoop.fs.UnresolvedLinkException, IOException
      Specified by:
      setTimes in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      FileNotFoundException
      org.apache.hadoop.fs.UnresolvedLinkException
      IOException

    • setVerifyChecksum

      public void setVerifyChecksum(boolean verifyChecksum) throws org.apache.hadoop.security.AccessControlException, IOException
      Specified by:
      setVerifyChecksum in class org.apache.hadoop.fs.AbstractFileSystem
      Throws:
      org.apache.hadoop.security.AccessControlException
      IOException