Package org.apache.hadoop.fs.azure

Class NativeAzureFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.azure.NativeAzureFileSystem

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

Direct Known Subclasses:

NativeAzureFileSystem.Secure

@Public
@Stable
public class NativeAzureFileSystem
extends org.apache.hadoop.fs.FileSystem

A FileSystem for reading and writing files stored on Windows Azure. This implementation is blob-based and stores files on Azure in their native form so they can be read by other Azure tools.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	NativeAzureFileSystem.FolderRenamePending	A description of a folder rename operation, including the source and destination keys, and descriptions of the files in the source folder.
class	NativeAzureFileSystem.NativeAzureFsOutputStream	Azure output stream; wraps an inner stream of different types.
static class	NativeAzureFileSystem.Secure	A FileSystem for reading and writing files stored on Windows Azure.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	APPEND_SUPPORT_ENABLE_PROPERTY_NAME
static java.lang.String	AZURE_CHMOD_USERLIST_PROPERTY_NAME	Configuration property used to specify list of users that can perform chmod operation when authorization is enabled in WASB.
static java.lang.String	AZURE_CHOWN_USERLIST_PROPERTY_NAME	Configuration property used to specify list of users that can perform chown operation when authorization is enabled in WASB.
static java.lang.String	AZURE_DAEMON_USERLIST_PROPERTY_NAME	Configuration property used to specify list of daemon users that can perform chmod operation when authorization is enabled in WASB.
static java.lang.String	AZURE_DELETE_THREADS	The configuration property to set number of threads to be used for delete operation.
static java.lang.String	AZURE_RENAME_THREADS	The configuration property to set number of threads to be used for rename operation.
static int	DEFAULT_AZURE_DELETE_THREADS	The default number of threads to be used for delete operation.
static int	DEFAULT_AZURE_RENAME_THREADS	The default number of threads to be used for rename operation.
static java.lang.String	KEY_AZURE_AUTHORIZATION	Configuration key to enable authorization support in WASB.
static org.slf4j.Logger	LOG
static java.lang.String	RETURN_URI_AS_CANONICAL_SERVICE_NAME_PROPERTY_NAME
static java.lang.String	SKIP_AZURE_METRICS_PROPERTY_NAME

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

TOKEN_LOG

Constructor Summary

Constructors

Constructor	Description
NativeAzureFileSystem()
NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

Method Summary

Modifier and Type	Method	Description
void	access(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsAction mode)
SelfRenewingLease	acquireLease(org.apache.hadoop.fs.Path path)	Get a self-renewing lease on the specified file.
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)	This optional operation is not yet supported.
protected void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
protected org.apache.hadoop.fs.FSDataOutputStream	createInternal(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, SelfRenewingLease parentFolderLease)	This is the version of the create call that is meant for internal usage.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path path)	Deprecated.
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive, boolean skipParentFolderLastModifiedTimeUpdate)	Delete the specified file or folder.
void	deleteFilesWithDanglingTempData(org.apache.hadoop.fs.Path root)	Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
protected boolean	existsInternal(org.apache.hadoop.fs.Path f)	Checks if a given path exists in the filesystem.
protected void	finalize()
java.lang.String	getCanonicalServiceName()	If fs.azure.override.canonical.service.name is set as true, return URI of the WASB filesystem, otherwise use the default implementation.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(java.lang.String renewer)	Get a delegation token from remote service endpoint if 'fs.azure.enable.kerberos.support' is set to 'true'.
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.Path	getHomeDirectory()
AzureFileSystemInstrumentation	getInstrumentation()	Gets the metrics source for this file system.
java.lang.String	getOwnerForPath(org.apache.hadoop.fs.Path absolutePath)
java.lang.String	getScheme()
AzureNativeFileSystemStore	getStore()	For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.
org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor	getThreadPoolExecutor(int threadCount, java.lang.String threadNamePrefix, java.lang.String operation, java.lang.String key, java.lang.String config)
java.net.URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
byte[]	getXAttr(org.apache.hadoop.fs.Path path, java.lang.String xAttrName)	Get the value of an attribute for a path.
boolean	hasPathCapability(org.apache.hadoop.fs.Path path, java.lang.String capability)
void	initialize(java.net.URI uri, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)	Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.
org.apache.hadoop.fs.Path	makeAbsolute(org.apache.hadoop.fs.Path path)	Get the absolute version of the path (fully qualified).
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean noUmask)
static java.lang.String	newMetricsSourceName()	Creates a new metrics source name that's unique within this process.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize)
protected java.util.concurrent.CompletableFuture<org.apache.hadoop.fs.FSDataInputStream>	openFileWithOptions(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.impl.OpenFileParameters parameters)
java.lang.String	pathToKey(org.apache.hadoop.fs.Path path)	Convert the path to a key.
void	recoverFilesWithDanglingTempData(org.apache.hadoop.fs.Path root, org.apache.hadoop.fs.Path destination)	Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path p, java.lang.String username, java.lang.String groupname)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
void	setWorkingDirectory(org.apache.hadoop.fs.Path newDir)	Set the working directory to the given directory.
void	setXAttr(org.apache.hadoop.fs.Path path, java.lang.String xAttrName, byte[] value, java.util.EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flag)	Set the value of an attribute for a path.
void	updateWasbAuthorizer(WasbAuthorizerInterface authorizer)

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFid, createFile, createMultipartUploader, createNewFile, createPathHandle, createPathId, createSnapshot, createSnapshot, createSymlink, deleteFid, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttrs, getXAttrs, getZkConnectString, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, mkdirsFid, mkdirsFid, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFid, openFid, openFid2, openFile, openFile, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwnerFid, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens

Field Detail

LOG

public static final org.slf4j.Logger LOG

AZURE_CHOWN_USERLIST_PROPERTY_NAME

public static final java.lang.String AZURE_CHOWN_USERLIST_PROPERTY_NAME

Configuration property used to specify list of users that can perform chown operation when authorization is enabled in WASB.

See Also:

Constant Field Values

AZURE_DAEMON_USERLIST_PROPERTY_NAME

public static final java.lang.String AZURE_DAEMON_USERLIST_PROPERTY_NAME

Configuration property used to specify list of daemon users that can perform chmod operation when authorization is enabled in WASB.

See Also:

Constant Field Values

AZURE_CHMOD_USERLIST_PROPERTY_NAME

public static final java.lang.String AZURE_CHMOD_USERLIST_PROPERTY_NAME

Configuration property used to specify list of users that can perform chmod operation when authorization is enabled in WASB.

See Also:

Constant Field Values

SKIP_AZURE_METRICS_PROPERTY_NAME

public static final java.lang.String SKIP_AZURE_METRICS_PROPERTY_NAME

See Also:

Constant Field Values

APPEND_SUPPORT_ENABLE_PROPERTY_NAME

public static final java.lang.String APPEND_SUPPORT_ENABLE_PROPERTY_NAME

See Also:

Constant Field Values

RETURN_URI_AS_CANONICAL_SERVICE_NAME_PROPERTY_NAME

public static final java.lang.String RETURN_URI_AS_CANONICAL_SERVICE_NAME_PROPERTY_NAME

See Also:

Constant Field Values

AZURE_RENAME_THREADS

public static final java.lang.String AZURE_RENAME_THREADS

The configuration property to set number of threads to be used for rename operation.

See Also:

Constant Field Values

DEFAULT_AZURE_RENAME_THREADS

public static final int DEFAULT_AZURE_RENAME_THREADS

The default number of threads to be used for rename operation.

See Also:

Constant Field Values

AZURE_DELETE_THREADS

public static final java.lang.String AZURE_DELETE_THREADS

The configuration property to set number of threads to be used for delete operation.

See Also:

Constant Field Values

DEFAULT_AZURE_DELETE_THREADS

public static final int DEFAULT_AZURE_DELETE_THREADS

The default number of threads to be used for delete operation.

See Also:

Constant Field Values

KEY_AZURE_AUTHORIZATION

public static final java.lang.String KEY_AZURE_AUTHORIZATION

Configuration key to enable authorization support in WASB.

See Also:

Constant Field Values

Constructor Detail

NativeAzureFileSystem

public NativeAzureFileSystem()

NativeAzureFileSystem

public NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

Method Detail

getScheme

public java.lang.String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

getCanonicalServiceName

public java.lang.String getCanonicalServiceName()

If fs.azure.override.canonical.service.name is set as true, return URI of the WASB filesystem, otherwise use the default implementation.

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

Returns:

a service string that uniquely identifies this file system

newMetricsSourceName

@VisibleForTesting
public static java.lang.String newMetricsSourceName()

Creates a new metrics source name that's unique within this process.

Returns:

metric source name

checkPath

protected void checkPath(org.apache.hadoop.fs.Path path)

Overrides:

checkPath in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(java.net.URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws java.io.IOException,
                       java.lang.IllegalArgumentException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

java.lang.IllegalArgumentException

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

Overrides:

getHomeDirectory in class org.apache.hadoop.fs.FileSystem

updateWasbAuthorizer

@VisibleForTesting
public void updateWasbAuthorizer(WasbAuthorizerInterface authorizer)

pathToKey

@VisibleForTesting
public java.lang.String pathToKey(org.apache.hadoop.fs.Path path)

Convert the path to a key. By convention, any leading or trailing slash is removed, except for the special case of a single slash.

Parameters:

path - path converted to a key

Returns:

key string

makeAbsolute

@VisibleForTesting
public org.apache.hadoop.fs.Path makeAbsolute(org.apache.hadoop.fs.Path path)

Get the absolute version of the path (fully qualified). This is public for testing purposes.

Parameters:

path - path to be absolute path.

Returns:

fully qualified path

getStore

@VisibleForTesting
public AzureNativeFileSystemStore getStore()

For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.

Returns:

The store object.

getInstrumentation

public AzureFileSystemInstrumentation getInstrumentation()

Gets the metrics source for this file system. This is mainly here for unit testing purposes.

Returns:

the metrics source.

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

This optional operation is not yet supported.

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

acquireLease

public SelfRenewingLease acquireLease(org.apache.hadoop.fs.Path path)
                               throws AzureException

Get a self-renewing lease on the specified file.

Parameters:

path - path whose lease to be renewed.

Returns:

Lease

Throws:

AzureException - when not being able to acquire a lease on the path

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws java.io.IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws java.io.IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws java.io.IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

createInternal

protected org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path f,
                                                                 org.apache.hadoop.fs.permission.FsPermission permission,
                                                                 boolean overwrite,
                                                                 SelfRenewingLease parentFolderLease)
                                                          throws org.apache.hadoop.fs.FileAlreadyExistsException,
                                                                 java.io.IOException

This is the version of the create call that is meant for internal usage. This version is not public facing and does not perform authorization checks. It is used by the public facing create call and by FolderRenamePending to create the internal -RenamePending.json file.

Parameters:

f - the path to a file to be created.

permission - for the newly created file.

overwrite - specifies if the file should be overwritten.

parentFolderLease - lease on the parent folder.

Returns:

the output stream used to write data into the newly created file .

Throws:

java.io.IOException - if an IO error occurs while attempting to delete the path.

org.apache.hadoop.fs.FileAlreadyExistsException

delete

@Deprecated
public boolean delete(org.apache.hadoop.fs.Path path)
               throws java.io.IOException

Deprecated.

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive)
               throws java.io.IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive,
                      boolean skipParentFolderLastModifiedTimeUpdate)
               throws java.io.IOException

Delete the specified file or folder. The parameter skipParentFolderLastModifiedTimeUpdate is used in the case of atomic folder rename redo. In that case, there is a lease on the parent folder, so (without reworking the code) modifying the parent folder update time will fail because of a conflict with the lease. Since we are going to delete the folder soon anyway so accurate modified time is not necessary, it's easier to just skip the modified time update.

Parameters:

f - file path to be deleted.

recursive - specify deleting recursively or not.

skipParentFolderLastModifiedTimeUpdate - If true, don't update the folder last modified time.

Returns:

true if and only if the file is deleted

Throws:

java.io.IOException - Thrown when fail to delete file or directory.

getThreadPoolExecutor

public org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor getThreadPoolExecutor(int threadCount,
                                                                                          java.lang.String threadNamePrefix,
                                                                                          java.lang.String operation,
                                                                                          java.lang.String key,
                                                                                          java.lang.String config)

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws java.io.FileNotFoundException,
                                                     java.io.IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.FileNotFoundException

java.io.IOException

existsInternal

protected boolean existsInternal(org.apache.hadoop.fs.Path f)
                          throws java.io.IOException

Checks if a given path exists in the filesystem. Calls getFileStatusInternal and has the same costs as the public facing exists call. This internal version of the exists call does not perform authorization checks, and is used internally by various filesystem operations that need to check if the parent/ancestor/path exist. The idea is to avoid having to configure authorization policies for these internal calls.

Parameters:

f - the path to a file or directory.

Returns:

true if path exists; otherwise false.

Throws:

java.io.IOException - if an IO error occurs while attempting to check for existence of the path.

getUri

public java.net.URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws java.io.FileNotFoundException,
                                                    java.io.IOException

Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.FileNotFoundException

java.io.IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws java.io.IOException

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission,
                      boolean noUmask)
               throws java.io.IOException

Throws:

java.io.IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                   int bufferSize)
                                            throws java.io.FileNotFoundException,
                                                   java.io.IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.FileNotFoundException

java.io.IOException

openFileWithOptions

protected java.util.concurrent.CompletableFuture<org.apache.hadoop.fs.FSDataInputStream> openFileWithOptions(org.apache.hadoop.fs.Path path,
                                                                                                             org.apache.hadoop.fs.impl.OpenFileParameters parameters)
                                                                                                      throws java.io.IOException

Overrides:

openFileWithOptions in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws java.io.FileNotFoundException,
                      java.io.IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.FileNotFoundException

java.io.IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)

Set the working directory to the given directory.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

setPermission

public void setPermission(org.apache.hadoop.fs.Path p,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws java.io.FileNotFoundException,
                          java.io.IOException

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.FileNotFoundException

java.io.IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path p,
                     java.lang.String username,
                     java.lang.String groupname)
              throws java.io.IOException

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

setXAttr

public void setXAttr(org.apache.hadoop.fs.Path path,
                     java.lang.String xAttrName,
                     byte[] value,
                     java.util.EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flag)
              throws java.io.IOException

Set the value of an attribute for a path.

Overrides:

setXAttr in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path on which to set the attribute

xAttrName - The attribute to set

value - The byte value of the attribute to set (encoded in utf-8)

flag - The mode in which to set the attribute

Throws:

java.io.IOException - If there was an issue setting the attribute on Azure

getXAttr

public byte[] getXAttr(org.apache.hadoop.fs.Path path,
                       java.lang.String xAttrName)
                throws java.io.IOException

Get the value of an attribute for a path.

Overrides:

getXAttr in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path on which to get the attribute

xAttrName - The attribute to get

Returns:

The bytes of the attribute's value (encoded in utf-8) or null if the attribute does not exist

Throws:

java.io.IOException - If there was an issue getting the attribute from Azure

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

getDelegationToken

public org.apache.hadoop.security.token.Token<?> getDelegationToken(java.lang.String renewer)
                                                             throws java.io.IOException

Get a delegation token from remote service endpoint if 'fs.azure.enable.kerberos.support' is set to 'true'.

Specified by:

getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getDelegationToken in class org.apache.hadoop.fs.FileSystem

Parameters:

renewer - the account name that is allowed to renew the token.

Returns:

delegation token

Throws:

java.io.IOException - thrown when getting the current user.

access

public void access(org.apache.hadoop.fs.Path path,
                   org.apache.hadoop.fs.permission.FsAction mode)
            throws java.io.IOException

Overrides:

access in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

recoverFilesWithDanglingTempData

public void recoverFilesWithDanglingTempData(org.apache.hadoop.fs.Path root,
                                             org.apache.hadoop.fs.Path destination)
                                      throws java.io.IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we move them to the destination given.

Parameters:

root - The root path to consider.

destination - The destination path to move any recovered files to.

Throws:

java.io.IOException - Thrown when fail to recover files.

deleteFilesWithDanglingTempData

public void deleteFilesWithDanglingTempData(org.apache.hadoop.fs.Path root)
                                     throws java.io.IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we delete them.

Parameters:

root - The root path to consider.

Throws:

java.io.IOException - Thrown when fail to delete.

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable

getOwnerForPath

@VisibleForTesting
public java.lang.String getOwnerForPath(org.apache.hadoop.fs.Path absolutePath)
                                 throws java.io.IOException

Throws:

java.io.IOException

hasPathCapability

public boolean hasPathCapability(org.apache.hadoop.fs.Path path,
                                 java.lang.String capability)
                          throws java.io.IOException

Specified by:

hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities

Overrides:

hasPathCapability in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException