Package org.apache.hadoop.fs.adl

Class AdlFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.adl.AdlFileSystem

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@Public
@Evolving
public class AdlFileSystem
extends org.apache.hadoop.fs.FileSystem

A FileSystem to access Azure Data Lake Store.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	SCHEME

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

TOKEN_LOG

Constructor Summary

Constructors

Constructor	Description
AdlFileSystem()

Method Summary

Modifier and Type	Method	Description
void	access(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsAction mode)	Checks if the user can access a path.
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)	Append to an existing file (optional operation).
void	concat(org.apache.hadoop.fs.Path trg, org.apache.hadoop.fs.Path[] srcs)	Concat existing files together.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)	Create call semantic is handled differently in case of ADL.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)	Deprecated. API only for 0.20-append
boolean	delete(org.apache.hadoop.fs.Path path, boolean recursive)	Delete a file.
static java.lang.String	getAccountNameFromFQDN(java.lang.String accountFQDN)	Gets ADL account name from ADL FQDN.
org.apache.hadoop.fs.permission.AclStatus	getAclStatus(org.apache.hadoop.fs.Path path)	Gets the ACL of a file or directory.
com.microsoft.azure.datalake.store.ADLStoreClient	getAdlClient()
long	getBlockSize(org.apache.hadoop.fs.Path f)	Deprecated. Use getFileStatus() instead
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)	Return the ContentSummary of a given Path.
protected AzureADTokenProvider	getCustomAccessTokenProvider(org.apache.hadoop.conf.Configuration conf)	This method is provided for convenience for derived classes to define custom AzureADTokenProvider instance.
long	getDefaultBlockSize()	Deprecated. use getDefaultBlockSize(Path) instead
long	getDefaultBlockSize(org.apache.hadoop.fs.Path f)	Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
int	getDefaultPort()
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)	Return a file status object that represents the path.
org.apache.hadoop.fs.Path	getHomeDirectory()	Constructing home directory locally is fine as long as Hadoop local user name and ADL user name relationship story is not fully baked yet.
short	getReplication(org.apache.hadoop.fs.Path src)	Deprecated. Use getFileStatus() instead
java.lang.String	getScheme()
protected java.lang.String	getTransportScheme()
java.net.URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()	Get the current working directory for the given file system.
boolean	hasPathCapability(org.apache.hadoop.fs.Path path, java.lang.String capability)
void	initialize(java.net.URI storeUri, org.apache.hadoop.conf.Configuration originalConf)	Called after a new FileSystem instance is constructed.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)	List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)	Make the given file and all non-existent parents into directories.
void	modifyAclEntries(org.apache.hadoop.fs.Path path, java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)	Modifies ACL entries of files and directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int buffersize)	Open call semantic is handled differently in case of ADL.
static org.apache.hadoop.conf.Configuration	propagateAccountOptions(org.apache.hadoop.conf.Configuration source, java.lang.String accountName)	Propagates account-specific settings into generic ADL configuration keys.
void	removeAcl(org.apache.hadoop.fs.Path path)	Removes all but the base ACL entries of files and directories.
void	removeAclEntries(org.apache.hadoop.fs.Path path, java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)	Removes ACL entries from files and directories.
void	removeDefaultAcl(org.apache.hadoop.fs.Path path)	Removes all default ACL entries from files and directories.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)	Renames Path src to Path dst.
void	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, org.apache.hadoop.fs.Options.Rename... options)	Deprecated.
void	setAcl(org.apache.hadoop.fs.Path path, java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)	Fully replaces ACL of files and directories, discarding all existing entries.
void	setOwner(org.apache.hadoop.fs.Path path, java.lang.String owner, java.lang.String group)	Set owner of a path (i.e.
void	setPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)	Set permission of a path.
boolean	setReplication(org.apache.hadoop.fs.Path p, short replication)	Azure data lake does not support user configuration for data replication hence not leaving system to query on azure data lake.
void	setUserGroupRepresentationAsUPN(boolean enableUPN)
void	setWorkingDirectory(org.apache.hadoop.fs.Path dir)	Set the current working directory for the given file system.
boolean	supportsSymlinks()

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFid, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createPathId, createSnapshot, createSnapshot, createSymlink, delete, deleteFid, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, getZkConnectString, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, mkdirsFid, mkdirsFid, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFid, openFid, openFid2, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeXAttr, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setDefaultUri, setDefaultUri, setOwnerFid, setQuota, setQuotaByStorageType, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens

Field Detail

SCHEME

public static final java.lang.String SCHEME

See Also:

Constant Field Values

Constructor Detail

AdlFileSystem

public AdlFileSystem()

Method Detail

getScheme

public java.lang.String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

getUri

public java.net.URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

getDefaultPort

public int getDefaultPort()

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

supportsSymlinks

public boolean supportsSymlinks()

Overrides:

supportsSymlinks in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(java.net.URI storeUri,
                       org.apache.hadoop.conf.Configuration originalConf)
                throws java.io.IOException

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

storeUri - a uri whose authority section names the host, port, etc. for this FileSystem

originalConf - the configuration to use for the FS. The account- specific options are patched over the base ones before any use is made of the config.

Throws:

java.io.IOException

getCustomAccessTokenProvider

protected AzureADTokenProvider getCustomAccessTokenProvider(org.apache.hadoop.conf.Configuration conf)
                                                     throws java.io.IOException

This method is provided for convenience for derived classes to define custom AzureADTokenProvider instance. In order to ensure secure hadoop infrastructure and user context for which respective AdlFileSystem instance is initialized, Loading AzureADTokenProvider is not sufficient. The order of loading AzureADTokenProvider is to first invoke getCustomAccessTokenProvider(Configuration), If method return null which means no implementation provided by derived classes, then configuration object is loaded to retrieve token configuration as specified is documentation. Custom token management takes the higher precedence during initialization.

Parameters:

conf - Configuration object

Returns:

null if the no custom AzureADTokenProvider token management is specified.

Throws:

java.io.IOException - if failed to initialize token provider.

getAdlClient

@VisibleForTesting
public com.microsoft.azure.datalake.store.ADLStoreClient getAdlClient()

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

Constructing home directory locally is fine as long as Hadoop local user name and ADL user name relationship story is not fully baked yet.

Overrides:

getHomeDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

Hadoop local user home directory.

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

Create call semantic is handled differently in case of ADL. Create semantics is translated to Create/Append semantics. 1. No dedicated connection to server. 2. Buffering is locally done, Once buffer is full or flush is invoked on the by the caller. All the pending data is pushed to ADL as APPEND operation code. 3. On close - Additional call is send to server to close the stream, and release lock from the stream. Necessity of Create/Append semantics is 1. ADL backend server does not allow idle connection for longer duration . In case of slow writer scenario, observed connection timeout/Connection reset causing occasional job failures. 2. Performance boost to jobs which are slow writer, avoided network latency 3. ADL equally better performing with multiple of 4MB chunk as append calls.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - File path

permission - Access permission for the newly created file

overwrite - Remove existing file and recreate new one if true otherwise throw error if file exist

bufferSize - Buffer size, ADL backend does not honour

replication - Replication count, ADL backend does not honour

blockSize - Block size, ADL backend does not honour

progress - Progress indicator

Returns:

FSDataOutputStream OutputStream on which application can push stream of bytes

Throws:

java.io.IOException - when system error, internal server error or user error

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws java.io.IOException

Deprecated.

API only for 0.20-append

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

permission - Access permission for the newly created file

flags - CreateFlags to use for this stream.

bufferSize - the size of the buffer to be used. ADL backend does not honour

replication - required block replication for the file. ADL backend does not honour

blockSize - Block size, ADL backend does not honour

progress - Progress indicator

Throws:

java.io.IOException - when system error, internal server error or user error

See Also:

setPermission(Path, FsPermission)

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

Append to an existing file (optional operation).

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used. ADL backend does not honour

progress - Progress indicator

Throws:

java.io.IOException - when system error, internal server error or user error

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path p,
                              short replication)
                       throws java.io.IOException

Azure data lake does not support user configuration for data replication hence not leaving system to query on azure data lake. Stub implementation

Overrides:

setReplication in class org.apache.hadoop.fs.FileSystem

Parameters:

p - Not honoured

replication - Not honoured

Returns:

True hard coded since ADL file system does not support replication configuration

Throws:

java.io.IOException - No exception would not thrown in this case however aligning with parent api definition.

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                   int buffersize)
                                            throws java.io.IOException

Open call semantic is handled differently in case of ADL. Instead of network stream is returned to the user, Overridden FsInputStream is returned.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

f - File path

buffersize - Buffer size, Not honoured

Returns:

FSDataInputStream InputStream on which application can read stream of bytes

Throws:

java.io.IOException - when system error, internal server error or user error

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws java.io.IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

java.io.IOException - when the path does not exist or any other error; IOException see specific implementation

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws java.io.IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

java.io.IOException - when the path does not exist or any other error; IOException see specific implementation

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws java.io.IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS. ADLS support POSIX standard for rename operation.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

java.io.IOException - on failure

rename

@Deprecated
public void rename(org.apache.hadoop.fs.Path src,
                   org.apache.hadoop.fs.Path dst,
                   org.apache.hadoop.fs.Options.Rename... options)
            throws java.io.IOException

Deprecated.

Overrides:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

concat

public void concat(org.apache.hadoop.fs.Path trg,
                   org.apache.hadoop.fs.Path[] srcs)
            throws java.io.IOException

Concat existing files together.

Overrides:

concat in class org.apache.hadoop.fs.FileSystem

Parameters:

trg - the path to the target destination.

srcs - the paths to the sources to use for the concatenation.

Throws:

java.io.IOException - when system error, internal server error or user error

delete

public boolean delete(org.apache.hadoop.fs.Path path,
                      boolean recursive)
               throws java.io.IOException

Delete a file.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

java.io.IOException - when system error, internal server error or user error

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws java.io.IOException

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - to apply to path

Throws:

java.io.IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path path,
                     java.lang.String owner,
                     java.lang.String group)
              throws java.io.IOException

Set owner of a path (i.e. a file or a directory). The parameters owner and group cannot both be null.

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path

owner - If it is null, the original username remains unchanged.

group - If it is null, the original groupname remains unchanged.

Throws:

java.io.IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path path,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws java.io.IOException

Set permission of a path.

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path

permission - Access permission

Throws:

java.io.IOException

modifyAclEntries

public void modifyAclEntries(org.apache.hadoop.fs.Path path,
                             java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws java.io.IOException

Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)

Overrides:

modifyAclEntries in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing modifications

Throws:

java.io.IOException - if an ACL could not be modified

removeAclEntries

public void removeAclEntries(org.apache.hadoop.fs.Path path,
                             java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws java.io.IOException

Removes ACL entries from files and directories. Other ACL entries are retained.

Overrides:

removeAclEntries in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing entries to remove

Throws:

java.io.IOException - if an ACL could not be modified

removeDefaultAcl

public void removeDefaultAcl(org.apache.hadoop.fs.Path path)
                      throws java.io.IOException

Removes all default ACL entries from files and directories.

Overrides:

removeDefaultAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

Throws:

java.io.IOException - if an ACL could not be modified

removeAcl

public void removeAcl(org.apache.hadoop.fs.Path path)
               throws java.io.IOException

Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.

Overrides:

removeAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

Throws:

java.io.IOException - if an ACL could not be removed

setAcl

public void setAcl(org.apache.hadoop.fs.Path path,
                   java.util.List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
            throws java.io.IOException

Fully replaces ACL of files and directories, discarding all existing entries.

Overrides:

setAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing modifications, must include entries for user, group, and others for compatibility with permission bits.

Throws:

java.io.IOException - if an ACL could not be modified

getAclStatus

public org.apache.hadoop.fs.permission.AclStatus getAclStatus(org.apache.hadoop.fs.Path path)
                                                       throws java.io.IOException

Gets the ACL of a file or directory.

Overrides:

getAclStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to get

Returns:

AclStatus describing the ACL of the file or directory

Throws:

java.io.IOException - if an ACL could not be read

access

public void access(org.apache.hadoop.fs.Path path,
                   org.apache.hadoop.fs.permission.FsAction mode)
            throws java.io.IOException

Checks if the user can access a path. The mode specifies which access checks to perform. If the requested permissions are granted, then the method returns normally. If access is denied, then the method throws an AccessControlException.

Overrides:

access in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to check

mode - type of access to check

Throws:

org.apache.hadoop.security.AccessControlException - if access is denied

java.io.FileNotFoundException - if the path does not exist

java.io.IOException - see specific implementation

getContentSummary

public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                      throws java.io.IOException

Return the ContentSummary of a given Path.

Overrides:

getContentSummary in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path to use

Throws:

java.io.IOException

getTransportScheme

@VisibleForTesting
protected java.lang.String getTransportScheme()

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Get the current working directory for the given file system.

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

the directory pathname

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path dir)

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

dir - Working directory path.

getDefaultBlockSize

@Deprecated
public long getDefaultBlockSize()

Deprecated.

use getDefaultBlockSize(Path) instead

Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getDefaultBlockSize

public long getDefaultBlockSize(org.apache.hadoop.fs.Path f)

Return the number of bytes that large input files should be optimally be split into to minimize i/o time. The given path will be used to locate the actual filesystem. The full path does not have to exist.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path of file

Returns:

the default block size for the path's filesystem

getBlockSize

@Deprecated
public long getBlockSize(org.apache.hadoop.fs.Path f)
                  throws java.io.IOException

Deprecated.

Use getFileStatus() instead

Overrides:

getBlockSize in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

getReplication

@Deprecated
public short getReplication(org.apache.hadoop.fs.Path src)

Deprecated.

Use getFileStatus() instead

Get replication.

Overrides:

getReplication in class org.apache.hadoop.fs.FileSystem

Parameters:

src - file name

Returns:

file replication

setUserGroupRepresentationAsUPN

@VisibleForTesting
public void setUserGroupRepresentationAsUPN(boolean enableUPN)

getAccountNameFromFQDN

public static java.lang.String getAccountNameFromFQDN(java.lang.String accountFQDN)

Gets ADL account name from ADL FQDN.

Parameters:

accountFQDN - ADL account fqdn

Returns:

ADL account name

propagateAccountOptions

public static org.apache.hadoop.conf.Configuration propagateAccountOptions(org.apache.hadoop.conf.Configuration source,
                                                                           java.lang.String accountName)

Propagates account-specific settings into generic ADL configuration keys. This is done by propagating the values of the form fs.adl.account.${account_name}.key to fs.adl.key, for all values of "key" The source of the updated property is set to the key name of the account property, to aid in diagnostics of where things came from. Returns a new configuration. Why the clone? You can use the same conf for different filesystems, and the original values are not updated.

Parameters:

source - Source Configuration object

accountName - account name. Must not be empty

Returns:

a (potentially) patched clone of the original

hasPathCapability

public boolean hasPathCapability(org.apache.hadoop.fs.Path path,
                                 java.lang.String capability)
                          throws java.io.IOException

Specified by:

hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities

Overrides:

hasPathCapability in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException