Package org.apache.hadoop.fs.azure

Class AzureNativeFileSystemStore

java.lang.Object

org.apache.hadoop.fs.azure.AzureNativeFileSystemStore

@Private
@VisibleForTesting
public class AzureNativeFileSystemStore
extends java.lang.Object

Core implementation of Windows Azure Filesystem for Hadoop. Provides the bridging logic between Hadoop's abstract filesystem and Azure Storage

Field Summary

Fields

Modifier and Type	Field	Description
static int	DEFAULT_DOWNLOAD_BLOCK_SIZE
static boolean	DEFAULT_ENABLE_FLAT_LISTING	Enable flat listing of blobs as default option.
static long	DEFAULT_HADOOP_BLOCK_SIZE
static int	DEFAULT_UPLOAD_BLOCK_SIZE
static boolean	DEFAULT_USE_SECURE_MODE	Default values to control SAS Key mode.
static java.lang.String	FS_AZURE_BLOCK_BLOB_BUFFERED_PREAD_DISABLE	Optional config to enable a lock free pread which will bypass buffer in BlockBlobInputStream.
static java.lang.String	KEY_ATOMIC_RENAME_DIRECTORIES	Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.
static java.lang.String	KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES	Configuration key to indicate the set of directories in WASB where we should store files as block blobs with block compaction enabled.
static java.lang.String	KEY_ENABLE_FLAT_LISTING	Configuration key to enable flat listing of blobs.
static java.lang.String	KEY_PAGE_BLOB_DIRECTORIES	Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs.
static java.lang.String	KEY_USE_LOCAL_SAS_KEY_MODE	By default the SAS Key mode is expected to run in Romote key mode.
static java.lang.String	KEY_USE_SECURE_MODE	Configuration keys to identify if WASB needs to run in Secure mode.
static org.slf4j.Logger	LOG
static java.lang.String	NO_ACCESS_TO_CONTAINER_MSG	The error message template when container is not accessible.

Constructor Summary

Constructors

Constructor	Description
AzureNativeFileSystemStore()

Method Summary

Modifier and Type	Method	Description
SelfRenewingLease	acquireLease(java.lang.String key)	Get a lease on the blob identified by key.
void	changePermissionStatus(java.lang.String key, org.apache.hadoop.fs.permission.PermissionStatus newPermission)	Changes the permission status on the given key.
void	close()
boolean	delete(java.lang.String key)	API implementation to delete a blob in the back end azure storage.
boolean	delete(java.lang.String key, SelfRenewingLease lease)	API implementation to delete a blob in the back end azure storage.
void	dump()
boolean	explicitFileExists(java.lang.String key)	Checks whether an explicit file/folder exists.
protected void	finalize()
static java.lang.String	getAccountKeyFromConfiguration(java.lang.String accountName, org.apache.hadoop.conf.Configuration conf)
BandwidthGaugeUpdater	getBandwidthGaugeUpdater()
long	getHadoopBlockSize()	Returns the file block size.
java.lang.String	getLinkInFileMetadata(java.lang.String key)	If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file.
void	initialize(java.net.URI uri, org.apache.hadoop.conf.Configuration conf, AzureFileSystemInstrumentation instrumentation)	Method for the URI and configuration object necessary to create a storage session with an Azure session.
boolean	isAtomicRenameKey(java.lang.String key)	Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.
boolean	isBlockBlobWithCompactionKey(java.lang.String key)	Checks if the given key in Azure Storage should be stored as a block blobs with compaction enabled instead of normal block blob.
boolean	isKeyForDirectorySet(java.lang.String key, java.util.Set<java.lang.String> dirSet)
boolean	isPageBlobKey(java.lang.String key)	Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.
org.apache.hadoop.fs.azure.FileMetadata[]	list(java.lang.String prefix, int maxListingCount, int maxListingDepth)
void	purge(java.lang.String prefix)
void	rename(java.lang.String srcKey, java.lang.String dstKey)
void	rename(java.lang.String srcKey, java.lang.String dstKey, boolean acquireLease, SelfRenewingLease existingLease)
void	rename(java.lang.String srcKey, java.lang.String dstKey, boolean acquireLease, SelfRenewingLease existingLease, boolean overwriteDestination)
java.io.InputStream	retrieve(java.lang.String key)
java.io.InputStream	retrieve(java.lang.String key, long startByteOffset)
java.io.InputStream	retrieve(java.lang.String key, long startByteOffset, java.util.Optional<org.apache.hadoop.conf.Configuration> options)
java.io.DataOutputStream	retrieveAppendStream(java.lang.String key, int bufferSize)
byte[]	retrieveAttribute(java.lang.String key, java.lang.String attribute)
org.apache.hadoop.fs.azure.FileMetadata	retrieveMetadata(java.lang.String key)
void	storeAttribute(java.lang.String key, java.lang.String attribute, byte[] value)
void	storeEmptyFolder(java.lang.String key, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
void	storeEmptyLinkFile(java.lang.String key, java.lang.String tempBlobKey, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)	Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.
java.io.DataOutputStream	storefile(java.lang.String keyEncoded, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus, java.lang.String key)
void	updateFolderLastModifiedTime(java.lang.String key, java.util.Date lastModified, SelfRenewingLease folderLease)
void	updateFolderLastModifiedTime(java.lang.String key, SelfRenewingLease folderLease)

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

KEY_USE_SECURE_MODE

@VisibleForTesting
public static final java.lang.String KEY_USE_SECURE_MODE

Configuration keys to identify if WASB needs to run in Secure mode. In Secure mode all interactions with Azure storage is performed using SAS uris. There are two sub modes within the Secure mode , one is remote SAS key mode where the SAS keys are generated from a remote process and local mode where SAS keys are generated within WASB.

See Also:

Constant Field Values

KEY_USE_LOCAL_SAS_KEY_MODE

public static final java.lang.String KEY_USE_LOCAL_SAS_KEY_MODE

By default the SAS Key mode is expected to run in Romote key mode. This flags sets it to run on the local mode.

See Also:

Constant Field Values

KEY_PAGE_BLOB_DIRECTORIES

public static final java.lang.String KEY_PAGE_BLOB_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs. Entries should be plain directory names (i.e. not URIs) with no leading or trailing slashes. Delimit the entries with commas.

See Also:

Constant Field Values

KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES

public static final java.lang.String KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should store files as block blobs with block compaction enabled. Entries can be directory paths relative to the container (e.g. "/path") or fully qualified wasb:// URIs (e.g. wasb://container@example.blob.core.windows.net/path)

See Also:

Constant Field Values

KEY_ATOMIC_RENAME_DIRECTORIES

public static final java.lang.String KEY_ATOMIC_RENAME_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.

See Also:

Constant Field Values

KEY_ENABLE_FLAT_LISTING

public static final java.lang.String KEY_ENABLE_FLAT_LISTING

Configuration key to enable flat listing of blobs. This config is useful only if listing depth is AZURE_UNBOUNDED_DEPTH.

See Also:

Constant Field Values

FS_AZURE_BLOCK_BLOB_BUFFERED_PREAD_DISABLE

public static final java.lang.String FS_AZURE_BLOCK_BLOB_BUFFERED_PREAD_DISABLE

Optional config to enable a lock free pread which will bypass buffer in BlockBlobInputStream. This is not a config which can be set at cluster level. It can be used as an option on FutureDataInputStreamBuilder.

See Also:

FileSystem.openFile(org.apache.hadoop.fs.Path), Constant Field Values

DEFAULT_DOWNLOAD_BLOCK_SIZE

public static final int DEFAULT_DOWNLOAD_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_UPLOAD_BLOCK_SIZE

public static final int DEFAULT_UPLOAD_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_HADOOP_BLOCK_SIZE

public static final long DEFAULT_HADOOP_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_USE_SECURE_MODE

public static final boolean DEFAULT_USE_SECURE_MODE

Default values to control SAS Key mode. By default we set the values to false.

See Also:

Constant Field Values

DEFAULT_ENABLE_FLAT_LISTING

public static final boolean DEFAULT_ENABLE_FLAT_LISTING

Enable flat listing of blobs as default option. This is useful only if listing depth is AZURE_UNBOUNDED_DEPTH.

See Also:

Constant Field Values

NO_ACCESS_TO_CONTAINER_MSG

public static final java.lang.String NO_ACCESS_TO_CONTAINER_MSG

The error message template when container is not accessible.

See Also:

Constant Field Values

Constructor Detail

AzureNativeFileSystemStore

public AzureNativeFileSystemStore()

Method Detail

getBandwidthGaugeUpdater

@VisibleForTesting
public BandwidthGaugeUpdater getBandwidthGaugeUpdater()

initialize

public void initialize(java.net.URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       AzureFileSystemInstrumentation instrumentation)
                throws java.lang.IllegalArgumentException,
                       AzureException,
                       java.io.IOException

Method for the URI and configuration object necessary to create a storage session with an Azure session. It parses the scheme to ensure it matches the storage protocol supported by this file system.

Parameters:

uri - - URI for target storage blob.

conf - - reference to configuration object.

instrumentation - - the metrics source that will keep track of operations here.

Throws:

java.lang.IllegalArgumentException - if URI or job object is null, or invalid scheme.

AzureException

java.io.IOException

getAccountKeyFromConfiguration

@VisibleForTesting
public static java.lang.String getAccountKeyFromConfiguration(java.lang.String accountName,
                                                              org.apache.hadoop.conf.Configuration conf)
                                                       throws KeyProviderException

Throws:

KeyProviderException

isPageBlobKey

public boolean isPageBlobKey(java.lang.String key)

Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.

isBlockBlobWithCompactionKey

public boolean isBlockBlobWithCompactionKey(java.lang.String key)

Checks if the given key in Azure Storage should be stored as a block blobs with compaction enabled instead of normal block blob.

Parameters:

key - blob name

Returns:

true, if the file is in directory with block compaction enabled.

isAtomicRenameKey

public boolean isAtomicRenameKey(java.lang.String key)

Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.

isKeyForDirectorySet

public boolean isKeyForDirectorySet(java.lang.String key,
                                    java.util.Set<java.lang.String> dirSet)

getHadoopBlockSize

public long getHadoopBlockSize()

Returns the file block size. This is a fake value used for integration of the Azure store with Hadoop.

storefile

public java.io.DataOutputStream storefile(java.lang.String keyEncoded,
                                          org.apache.hadoop.fs.permission.PermissionStatus permissionStatus,
                                          java.lang.String key)
                                   throws AzureException

Throws:

AzureException

storeEmptyFolder

public void storeEmptyFolder(java.lang.String key,
                             org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
                      throws AzureException

Throws:

AzureException

storeEmptyLinkFile

public void storeEmptyLinkFile(java.lang.String key,
                               java.lang.String tempBlobKey,
                               org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
                        throws AzureException

Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.

Throws:

AzureException

getLinkInFileMetadata

public java.lang.String getLinkInFileMetadata(java.lang.String key)
                                       throws AzureException

If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file. Otherwise, returns null.

Throws:

AzureException

retrieveMetadata

public org.apache.hadoop.fs.azure.FileMetadata retrieveMetadata(java.lang.String key)
                                                         throws java.io.IOException

Throws:

java.io.IOException

retrieveAttribute

public byte[] retrieveAttribute(java.lang.String key,
                                java.lang.String attribute)
                         throws java.io.IOException

Throws:

java.io.IOException

storeAttribute

public void storeAttribute(java.lang.String key,
                           java.lang.String attribute,
                           byte[] value)
                    throws java.io.IOException

Throws:

java.io.IOException

retrieve

public java.io.InputStream retrieve(java.lang.String key)
                             throws AzureException,
                                    java.io.IOException

Throws:

AzureException

java.io.IOException

retrieve

public java.io.InputStream retrieve(java.lang.String key,
                                    long startByteOffset)
                             throws AzureException,
                                    java.io.IOException

Throws:

AzureException

java.io.IOException

retrieve

public java.io.InputStream retrieve(java.lang.String key,
                                    long startByteOffset,
                                    java.util.Optional<org.apache.hadoop.conf.Configuration> options)
                             throws AzureException,
                                    java.io.IOException

Throws:

AzureException

java.io.IOException

list

public org.apache.hadoop.fs.azure.FileMetadata[] list(java.lang.String prefix,
                                                      int maxListingCount,
                                                      int maxListingDepth)
                                               throws java.io.IOException

Throws:

java.io.IOException

delete

public boolean delete(java.lang.String key,
                      SelfRenewingLease lease)
               throws java.io.IOException

API implementation to delete a blob in the back end azure storage.

Throws:

java.io.IOException

delete

public boolean delete(java.lang.String key)
               throws java.io.IOException

API implementation to delete a blob in the back end azure storage.

Throws:

java.io.IOException

rename

public void rename(java.lang.String srcKey,
                   java.lang.String dstKey)
            throws java.io.IOException

Throws:

java.io.IOException

rename

public void rename(java.lang.String srcKey,
                   java.lang.String dstKey,
                   boolean acquireLease,
                   SelfRenewingLease existingLease)
            throws java.io.IOException

Throws:

java.io.IOException

rename

public void rename(java.lang.String srcKey,
                   java.lang.String dstKey,
                   boolean acquireLease,
                   SelfRenewingLease existingLease,
                   boolean overwriteDestination)
            throws java.io.IOException

Throws:

java.io.IOException

explicitFileExists

public boolean explicitFileExists(java.lang.String key)
                           throws AzureException

Checks whether an explicit file/folder exists. This is used by redo of atomic rename. There was a bug(apache jira HADOOP-12780) during atomic rename if process crashes after an inner directory has been renamed but still there are file under that directory to be renamed then after the process comes again it tries to redo the renames. It checks whether the directory exists or not by calling filesystem.exist. But filesystem.Exists will treat that directory as implicit directory and return true as file exists under that directory. So It will try try to rename that directory and will fail as the corresponding blob does not exist. So this method explicitly checks for the blob.

Throws:

AzureException

changePermissionStatus

public void changePermissionStatus(java.lang.String key,
                                   org.apache.hadoop.fs.permission.PermissionStatus newPermission)
                            throws AzureException

Changes the permission status on the given key.

Throws:

AzureException

purge

public void purge(java.lang.String prefix)
           throws java.io.IOException

Throws:

java.io.IOException

acquireLease

public SelfRenewingLease acquireLease(java.lang.String key)
                               throws AzureException

Get a lease on the blob identified by key. This lease will be renewed indefinitely by a background thread.

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(java.lang.String key,
                                         java.util.Date lastModified,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(java.lang.String key,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

dump

public void dump()
          throws java.io.IOException

Throws:

java.io.IOException

close

public void close()

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable

retrieveAppendStream

public java.io.DataOutputStream retrieveAppendStream(java.lang.String key,
                                                     int bufferSize)
                                              throws java.io.IOException

Throws:

java.io.IOException