java.io.Closeable, java.lang.AutoCloseable, Configurable, BulkDeleteSource, PathCapabilities, DelegationTokenIssuer@LimitedPrivate({"MapReduce","HBase","Hive"})
@Evolving
public class ViewFileSystemOverloadScheme
extends ViewFileSystem
This class is extended from the ViewFileSystem for the overloaded scheme file system. Mount link configurations and in-memory mount table building behaviors are inherited from ViewFileSystem. Unlike ViewFileSystem scheme (viewfs://), the users would be able to use any scheme.
To use this class, the following configurations need to be added in
core-site.xml file.
1) fs.<scheme>.impl
= org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme
2) fs.viewfs.overload.scheme.target.<scheme>.impl
= <hadoop compatible file system implementation class name
for the <scheme>>
Here <scheme> can be any scheme, but with that scheme there should be a hadoop compatible file system available. Second configuration value should be the respective scheme's file system implementation class. Example: if scheme is configured with "hdfs", then the 2nd configuration class name will be org.apache.hadoop.hdfs.DistributedFileSystem. if scheme is configured with "s3a", then the 2nd configuration class name will be org.apache.hadoop.fs.s3a.S3AFileSystem.
Use Case 1:
===========
If users want some of their existing cluster (hdfs://Cluster)
data to mount with other hdfs and object store clusters(hdfs://NN1,
o3fs://bucket1.volume1/, s3a://bucket1/)
fs.viewfs.mounttable.Cluster.link./user = hdfs://NN1/user
fs.viewfs.mounttable.Cluster.link./data = o3fs://bucket1.volume1/data
fs.viewfs.mounttable.Cluster.link./backup = s3a://bucket1/backup/
Op1: Create file hdfs://Cluster/user/fileA will go to hdfs://NN1/user/fileA
Op2: Create file hdfs://Cluster/data/datafile will go to
o3fs://bucket1.volume1/data/datafile
Op3: Create file hdfs://Cluster/backup/data.zip will go to
s3a://bucket1/backup/data.zip
Use Case 2:
===========
If users want some of their existing cluster (s3a://bucketA/)
data to mount with other hdfs and object store clusters
(hdfs://NN1, o3fs://bucket1.volume1/)
fs.viewfs.mounttable.bucketA.link./user = hdfs://NN1/user
fs.viewfs.mounttable.bucketA.link./data = o3fs://bucket1.volume1/data
fs.viewfs.mounttable.bucketA.link./salesDB = s3a://bucketA/salesDB/
Op1: Create file s3a://bucketA/user/fileA will go to hdfs://NN1/user/fileA
Op2: Create file s3a://bucketA/data/datafile will go to
o3fs://bucket1.volume1/data/datafile
Op3: Create file s3a://bucketA/salesDB/dbfile will go to
s3a://bucketA/salesDB/dbfile
Note:
(1) In ViewFileSystemOverloadScheme, by default the mount links will be
represented as non-symlinks. If you want to change this behavior, please see
ViewFileSystem.listStatus(Path)
(2) In ViewFileSystemOverloadScheme, only the initialized uri's hostname will
be considered as the mount table name. When the passed uri has hostname:port,
it will simply ignore the port number and only hostname will be considered as
the mount table name.
(3) If there are no mount links configured with the initializing uri's
hostname as the mount table name, then it will automatically consider the
current uri as fallback( ex:
fs.viewfs.mounttable.<mycluster>.linkFallback) target fs uri.
| Modifier and Type | Class | Description |
|---|---|---|
static class |
ViewFileSystemOverloadScheme.MountPathInfo<T> |
A class to maintain the target file system and a path to pass to the target
file system.
|
ViewFileSystem.MountPointFileSystem.DirectoryEntries, FileSystem.DirListingIterator<T extends FileStatus>, FileSystem.StatisticsDEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIXTOKEN_LOG| Constructor | Description |
|---|---|
ViewFileSystemOverloadScheme() |
| Modifier and Type | Method | Description |
|---|---|---|
java.net.URI |
canonicalizeUri(java.net.URI uri) |
Canonicalize the given URI.
|
protected FsGetter |
fsGetter() |
This method is overridden because in ViewFileSystemOverloadScheme if
overloaded scheme matches with mounted target fs scheme, file system
should be created without going into fs.<scheme>.impl based
resolution.
|
FileSystem |
getFallbackFileSystem() |
|
ViewFileSystemOverloadScheme.MountPathInfo<FileSystem> |
getMountPathInfo(Path path,
Configuration conf) |
Gets the mount path info, which contains the target file system and
remaining path to pass to the target file system.
|
FileSystem |
getRawFileSystem(Path path,
Configuration conf) |
This is an admin only API to give access to its child raw file system, if
the path is link.
|
java.lang.String |
getScheme() |
Return the protocol scheme for the FileSystem.
|
void |
initialize(java.net.URI theUri,
Configuration conf) |
Called after a new FileSystem instance is constructed.
|
void |
setSupportAutoAddingFallbackOnNoMounts(boolean addAutoFallbackOnNoMounts) |
Sets whether to add fallback automatically when no mount points found.
|
boolean |
supportAutoAddingFallbackOnNoMounts() |
By default returns false as ViewFileSystemOverloadScheme supports auto
adding fallback on no mounts.
|
access, append, close, create, createNonRecursive, createSnapshot, delete, delete, deleteSnapshot, getAclStatus, getAllStoragePolicies, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getEnclosingRoot, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileStatus, getHomeDirectory, getLinkTarget, getMountPoints, getQuotaUsage, getServerDefaults, getServerDefaults, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUri, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, hasPathCapability, listLocatedStatus, listStatus, listXAttrs, mkdirs, mkdirs, modifyAclEntries, open, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setOwner, setPermission, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWorkingDirectory, setWriteChecksum, setXAttr, truncate, unsetStoragePolicyappend, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFid, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createPathId, createSnapshot, createSymlink, deleteFid, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, get, getAdditionalTokenIssuers, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getDefaultPort, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLocal, getName, getNamed, getPathHandle, getReplication, getStatistics, getStatistics, getStorageStatistics, getUsed, getZkConnectString, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, makeQualified, mkdirs, mkdirsFid, mkdirsFid, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFid, openFid, openFid2, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, rename, resolveLink, setDefaultUri, setDefaultUri, setOwnerFid, setQuota, setQuotaByStorageType, setXAttr, startLocalOutput, supportsSymlinksgetConf, setConfclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitaddDelegationTokenspublic ViewFileSystemOverloadScheme()
throws java.io.IOException
java.io.IOExceptionpublic java.lang.String getScheme()
ViewFileSystemgetScheme in class ViewFileSystemviewfspublic boolean supportAutoAddingFallbackOnNoMounts()
public void setSupportAutoAddingFallbackOnNoMounts(boolean addAutoFallbackOnNoMounts)
addAutoFallbackOnNoMounts - addAutoFallbackOnNoMounts.public void initialize(java.net.URI theUri,
Configuration conf)
throws java.io.IOException
ViewFileSysteminitialize in class ViewFileSystemtheUri - a uri whose authority section names the host, port, etc. for
this FileSystemconf - the configurationjava.io.IOException - on any failure to initialize this instance.protected FsGetter fsGetter()
fsGetter in class ViewFileSystempublic FileSystem getRawFileSystem(Path path, Configuration conf) throws java.io.IOException
path - - fs uri pathconf - - configurationjava.io.IOException - raised on errors performing I/O.public ViewFileSystemOverloadScheme.MountPathInfo<FileSystem> getMountPathInfo(Path path, Configuration conf) throws java.io.IOException
path - the path.conf - configuration.java.io.IOException - raised on errors performing I/O.public FileSystem getFallbackFileSystem()
@LimitedPrivate("HDFS")
public java.net.URI canonicalizeUri(java.net.URI uri)
FileSystemFileSystem.getDefaultPort() returns a
default port.canonicalizeUri in class FileSystemuri - url.NetUtils.getCanonicalUri(URI, int)Copyright © 2008–2025 Apache Software Foundation. All rights reserved.