Package com.mapr.fs

Class MFS

  • All Implemented Interfaces:
    org.apache.hadoop.fs.PathCapabilities
    public class MFS
extends org.apache.hadoop.fs.AbstractFileSystem
    This class is an implementation of AbstractFileSystem, as a wrapper class around MapRFileSystem. It allows the new Yarn APIs to run on top of MapRFS.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static org.apache.commons.logging.Log LOG  

      • Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

        statistics

    • Constructor Summary

      Constructors 
      Constructor Description
      MFS()
      A no-args constructor that is required for reflection.
      MFS​(java.net.URI uri, org.apache.hadoop.conf.Configuration conf)
      Calls super constructor, with parameter authorityNeeded=false and initializes maprfs with uri and conf.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void checkPath​(org.apache.hadoop.fs.Path path)
      Takes the string component of path from the first '/' and appends it to the path of the URI before calling super.checkPath(Path).
      org.apache.hadoop.fs.FSDataOutputStream createInternal​(org.apache.hadoop.fs.Path p, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
      Creates internal file with indicated permission, bufferSize, replication, blockSize etc.
      void createSymlink​(org.apache.hadoop.fs.Path target, org.apache.hadoop.fs.Path link, boolean createParent)
      The specification of this method matches that of FileContext#createSymlink(Path, Path, boolean);
      boolean delete​(org.apache.hadoop.fs.Path f, boolean recursive)  
      org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations​(org.apache.hadoop.fs.Path p, long start, long len)  
      org.apache.hadoop.fs.FileChecksum getFileChecksum​(org.apache.hadoop.fs.Path path)  
      org.apache.hadoop.fs.FileStatus getFileStatus​(org.apache.hadoop.fs.Path p)  
      org.apache.hadoop.fs.FsStatus getFsStatus()  
      org.apache.hadoop.fs.FsServerDefaults getServerDefaults()
      Returns default settings for server.
      int getUriDefaultPort()  
      org.apache.hadoop.fs.FileStatus[] listStatus​(org.apache.hadoop.fs.Path p)  
      void mkdir​(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)
      Creates a directory at the specified path with specified permission.
      org.apache.hadoop.fs.FSDataInputStream open​(org.apache.hadoop.fs.Path p, int bufferSize)  
      void renameInternal​(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)  
      void setOwner​(org.apache.hadoop.fs.Path p, java.lang.String user, java.lang.String group)  
      void setPermission​(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)  
      boolean setReplication​(org.apache.hadoop.fs.Path p, short replication)  
      void setTimes​(org.apache.hadoop.fs.Path p, long mtime, long atime)  
      void setVerifyChecksum​(boolean verifyChecksum)  
      boolean supportsSymlinks()
      Returns true if the file system supports symlinks, false otherwise.

      • Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

        access, checkScheme, clearStatistics, create, createFileSystem, createMultipartUploader, createSnapshot, deleteSnapshot, equals, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getEnclosingRoot, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getStatistics, getStatistics, getStoragePolicy, getUri, getUriPath, getXAttr, getXAttrs, getXAttrs, hashCode, hasPathCapability, isValidName, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, listXAttrs, makeQualified, methodNotSupported, modifyAclEntries, msync, open, openFileWithOptions, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameInternal, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setStoragePolicy, setXAttr, setXAttr, truncate, unsetStoragePolicy

      • Methods inherited from class java.lang.Object

        clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • LOG

        public static final org.apache.commons.logging.Log LOG

    • Constructor Detail

      • MFS

        public MFS()
    throws java.net.URISyntaxException,
           java.io.IOException
        A no-args constructor that is required for reflection.
        Throws:
        java.net.URISyntaxException
        java.io.IOException

      • MFS

        public MFS​(java.net.URI uri,
           org.apache.hadoop.conf.Configuration conf)
    throws java.net.URISyntaxException,
           java.io.IOException
        Calls super constructor, with parameter authorityNeeded=false and initializes maprfs with uri and conf.
        Parameters:
        uri - of the filesystem
        conf - The current configuration
        Throws:
        java.net.URISyntaxException
        java.io.IOException

    • Method Detail

      • createInternal

        public org.apache.hadoop.fs.FSDataOutputStream createInternal​(org.apache.hadoop.fs.Path p,
                                                              java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
                                                              org.apache.hadoop.fs.permission.FsPermission absolutePermission,
                                                              int bufferSize,
                                                              short replication,
                                                              long blockSize,
                                                              org.apache.hadoop.util.Progressable progress,
                                                              org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
                                                              boolean createParent)
                                                       throws org.apache.hadoop.security.AccessControlException,
                                                              org.apache.hadoop.fs.FileAlreadyExistsException,
                                                              java.io.FileNotFoundException,
                                                              org.apache.hadoop.fs.ParentNotDirectoryException,
                                                              org.apache.hadoop.fs.UnsupportedFileSystemException,
                                                              org.apache.hadoop.fs.UnresolvedLinkException,
                                                              java.io.IOException
        Creates internal file with indicated permission, bufferSize, replication, blockSize etc. Ignores checksumOpt and createParent variables.
        Specified by:
        createInternal in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        org.apache.hadoop.fs.FileAlreadyExistsException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.ParentNotDirectoryException
        org.apache.hadoop.fs.UnsupportedFileSystemException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException
        See Also:
        TODO determine user of ChecksumOpt (Spencer)

      • checkPath

        public void checkPath​(org.apache.hadoop.fs.Path path)
        Takes the string component of path from the first '/' and appends it to the path of the URI before calling super.checkPath(Path). This works around an issue where the path contains no host information causing jobs to fail.
        Overrides:
        checkPath in class org.apache.hadoop.fs.AbstractFileSystem
        Parameters:
        path - The path to be checked TODO use only super.checkPath() instead of this modified version that manually inserts the URI (Spencer)

      • delete

        public boolean delete​(org.apache.hadoop.fs.Path f,
                      boolean recursive)
               throws org.apache.hadoop.security.AccessControlException,
                      java.io.FileNotFoundException,
                      org.apache.hadoop.fs.UnresolvedLinkException,
                      java.io.IOException
        Specified by:
        delete in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • getFileBlockLocations

        public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations​(org.apache.hadoop.fs.Path p,
                                                                  long start,
                                                                  long len)
                                                           throws org.apache.hadoop.security.AccessControlException,
                                                                  java.io.FileNotFoundException,
                                                                  org.apache.hadoop.fs.UnresolvedLinkException,
                                                                  java.io.IOException
        Specified by:
        getFileBlockLocations in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • getFileChecksum

        public org.apache.hadoop.fs.FileChecksum getFileChecksum​(org.apache.hadoop.fs.Path path)
                                                  throws org.apache.hadoop.security.AccessControlException,
                                                         java.io.FileNotFoundException,
                                                         org.apache.hadoop.fs.UnresolvedLinkException,
                                                         java.io.IOException
        Specified by:
        getFileChecksum in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • getFileStatus

        public org.apache.hadoop.fs.FileStatus getFileStatus​(org.apache.hadoop.fs.Path p)
                                              throws org.apache.hadoop.security.AccessControlException,
                                                     java.io.FileNotFoundException,
                                                     org.apache.hadoop.fs.UnresolvedLinkException,
                                                     java.io.IOException
        Specified by:
        getFileStatus in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • getFsStatus

        public org.apache.hadoop.fs.FsStatus getFsStatus()
                                          throws org.apache.hadoop.security.AccessControlException,
                                                 java.io.FileNotFoundException,
                                                 java.io.IOException
        Specified by:
        getFsStatus in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        java.io.IOException

      • getServerDefaults

        public org.apache.hadoop.fs.FsServerDefaults getServerDefaults()
                                                        throws java.io.IOException
        Returns default settings for server.
        Specified by:
        getServerDefaults in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        java.io.IOException

      • getUriDefaultPort

        public int getUriDefaultPort()
        Specified by:
        getUriDefaultPort in class org.apache.hadoop.fs.AbstractFileSystem
        Returns:
        the current port stored in URI of maprfs.

      • listStatus

        public org.apache.hadoop.fs.FileStatus[] listStatus​(org.apache.hadoop.fs.Path p)
                                             throws org.apache.hadoop.security.AccessControlException,
                                                    java.io.FileNotFoundException,
                                                    org.apache.hadoop.fs.UnresolvedLinkException,
                                                    java.io.IOException
        Specified by:
        listStatus in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • mkdir

        public void mkdir​(org.apache.hadoop.fs.Path p,
                  org.apache.hadoop.fs.permission.FsPermission permission,
                  boolean createParent)
           throws org.apache.hadoop.security.AccessControlException,
                  org.apache.hadoop.fs.FileAlreadyExistsException,
                  java.io.FileNotFoundException,
                  org.apache.hadoop.fs.UnresolvedLinkException,
                  java.io.IOException
        Creates a directory at the specified path with specified permission. Currently ignores the createParent option, maprfs will always create parents.
        Specified by:
        mkdir in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        org.apache.hadoop.fs.FileAlreadyExistsException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • open

        public org.apache.hadoop.fs.FSDataInputStream open​(org.apache.hadoop.fs.Path p,
                                                   int bufferSize)
                                            throws org.apache.hadoop.security.AccessControlException,
                                                   java.io.FileNotFoundException,
                                                   org.apache.hadoop.fs.UnresolvedLinkException,
                                                   java.io.IOException
        Specified by:
        open in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • supportsSymlinks

        public boolean supportsSymlinks()
        Returns true if the file system supports symlinks, false otherwise.
        Overrides:
        supportsSymlinks in class org.apache.hadoop.fs.AbstractFileSystem
        Returns:
        true if filesystem supports symlinks

      • createSymlink

        public void createSymlink​(org.apache.hadoop.fs.Path target,
                          org.apache.hadoop.fs.Path link,
                          boolean createParent)
                   throws java.io.IOException,
                          org.apache.hadoop.fs.UnresolvedLinkException
        The specification of this method matches that of FileContext#createSymlink(Path, Path, boolean);
        Overrides:
        createSymlink in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        java.io.IOException
        org.apache.hadoop.fs.UnresolvedLinkException

      • renameInternal

        public void renameInternal​(org.apache.hadoop.fs.Path src,
                           org.apache.hadoop.fs.Path dst)
                    throws org.apache.hadoop.security.AccessControlException,
                           org.apache.hadoop.fs.FileAlreadyExistsException,
                           java.io.FileNotFoundException,
                           org.apache.hadoop.fs.ParentNotDirectoryException,
                           org.apache.hadoop.fs.UnresolvedLinkException,
                           java.io.IOException
        Specified by:
        renameInternal in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        org.apache.hadoop.fs.FileAlreadyExistsException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.ParentNotDirectoryException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • setOwner

        public void setOwner​(org.apache.hadoop.fs.Path p,
                     java.lang.String user,
                     java.lang.String group)
              throws org.apache.hadoop.security.AccessControlException,
                     java.io.FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     java.io.IOException
        Specified by:
        setOwner in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • setPermission

        public void setPermission​(org.apache.hadoop.fs.Path p,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws org.apache.hadoop.security.AccessControlException,
                          java.io.FileNotFoundException,
                          org.apache.hadoop.fs.UnresolvedLinkException,
                          java.io.IOException
        Specified by:
        setPermission in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • setReplication

        public boolean setReplication​(org.apache.hadoop.fs.Path p,
                              short replication)
                       throws org.apache.hadoop.security.AccessControlException,
                              java.io.FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              java.io.IOException
        Specified by:
        setReplication in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • setTimes

        public void setTimes​(org.apache.hadoop.fs.Path p,
                     long mtime,
                     long atime)
              throws org.apache.hadoop.security.AccessControlException,
                     java.io.FileNotFoundException,
                     org.apache.hadoop.fs.UnresolvedLinkException,
                     java.io.IOException
        Specified by:
        setTimes in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.FileNotFoundException
        org.apache.hadoop.fs.UnresolvedLinkException
        java.io.IOException

      • setVerifyChecksum

        public void setVerifyChecksum​(boolean verifyChecksum)
                       throws org.apache.hadoop.security.AccessControlException,
                              java.io.IOException
        Specified by:
        setVerifyChecksum in class org.apache.hadoop.fs.AbstractFileSystem
        Throws:
        org.apache.hadoop.security.AccessControlException
        java.io.IOException