Package org.apache.hadoop.fs.azure.metrics

Class AzureFileSystemInstrumentation

java.lang.Object

org.apache.hadoop.fs.azure.metrics.AzureFileSystemInstrumentation

All Implemented Interfaces:

org.apache.hadoop.metrics2.MetricsSource

@Metrics(about="Metrics for WASB",
         context="azureFileSystem")
@Public
@Evolving
public final class AzureFileSystemInstrumentation
extends java.lang.Object
implements org.apache.hadoop.metrics2.MetricsSource

A metrics source for the WASB file system to track all the metrics we care about for getting a clear picture of the performance/reliability/interaction of the Hadoop cluster with Azure Storage.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	METRIC_TAG_ACCOUNT_NAME
static java.lang.String	METRIC_TAG_CONTAINTER_NAME
static java.lang.String	METRIC_TAG_FILESYSTEM_ID
static java.lang.String	WASB_BYTES_READ
static java.lang.String	WASB_BYTES_WRITTEN
static java.lang.String	WASB_CLIENT_ERRORS
static java.lang.String	WASB_DIRECTORIES_CREATED
static java.lang.String	WASB_DIRECTORIES_DELETED
static java.lang.String	WASB_DOWNLOAD_LATENCY
static java.lang.String	WASB_DOWNLOAD_RATE
static java.lang.String	WASB_FILES_CREATED
static java.lang.String	WASB_FILES_DELETED
static java.lang.String	WASB_RAW_BYTES_DOWNLOADED
static java.lang.String	WASB_RAW_BYTES_UPLOADED
static java.lang.String	WASB_SERVER_ERRORS
static java.lang.String	WASB_UPLOAD_LATENCY
static java.lang.String	WASB_UPLOAD_RATE
static java.lang.String	WASB_WEB_RESPONSES

Constructor Summary

Constructors

Constructor	Description
AzureFileSystemInstrumentation(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
void	blockDownloaded(long latency)	Indicate that we just downloaded a block and record its latency.
void	blockUploaded(long latency)	Indicate that we just uploaded a block and record its latency.
void	clientErrorEncountered()	Indicate that we just encountered a client-side error.
void	currentDownloadBytesPerSecond(long bytesPerSecond)	Record the current bytes-per-second download rate seen.
void	currentUploadBytesPerSecond(long bytesPerSecond)	Record the current bytes-per-second upload rate seen.
void	directoryCreated()	Indicate that we just created a directory through WASB.
void	directoryDeleted()	Indicate that we just deleted a directory through WASB.
void	fileCreated()	Indicate that we just created a file through WASB.
void	fileDeleted()	Indicate that we just deleted a file through WASB.
long	getBlockDownloadLatency()	Get the current rolling average of the download latency.
long	getBlockUploadLatency()	Get the current rolling average of the upload latency.
long	getCurrentMaximumDownloadBandwidth()	Get the current maximum download bandwidth.
long	getCurrentMaximumUploadBandwidth()	Get the current maximum upload bandwidth.
long	getCurrentWebResponses()	Gets the current number of web responses obtained from Azure Storage.
java.util.UUID	getFileSystemInstanceId()	The unique identifier for this file system in the metrics.
void	getMetrics(org.apache.hadoop.metrics2.MetricsCollector builder, boolean all)
org.apache.hadoop.metrics2.MetricsInfo	getMetricsRegistryInfo()	Get the metrics registry information.
void	rawBytesDownloaded(long numberOfBytes)	Indicate that we just downloaded some data to Azure storage.
void	rawBytesUploaded(long numberOfBytes)	Indicate that we just uploaded some data to Azure storage.
void	serverErrorEncountered()	Indicate that we just encountered a server-caused error.
void	setAccountName(java.lang.String accountName)	Sets the account name to tag all the metrics with.
void	setContainerName(java.lang.String containerName)	Sets the container name to tag all the metrics with.
void	updateBytesReadInLastSecond(long currentBytesRead)	Sets the current gauge value for how many bytes were read in the last second.
void	updateBytesWrittenInLastSecond(long currentBytesWritten)	Sets the current gauge value for how many bytes were written in the last second.
void	webResponse()	Indicate that we just got a web response from Azure Storage.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METRIC_TAG_FILESYSTEM_ID

public static final java.lang.String METRIC_TAG_FILESYSTEM_ID

See Also:

Constant Field Values

METRIC_TAG_ACCOUNT_NAME

public static final java.lang.String METRIC_TAG_ACCOUNT_NAME

See Also:

Constant Field Values

METRIC_TAG_CONTAINTER_NAME

public static final java.lang.String METRIC_TAG_CONTAINTER_NAME

See Also:

Constant Field Values

WASB_WEB_RESPONSES

public static final java.lang.String WASB_WEB_RESPONSES

See Also:

Constant Field Values

WASB_BYTES_WRITTEN

public static final java.lang.String WASB_BYTES_WRITTEN

See Also:

Constant Field Values

WASB_BYTES_READ

public static final java.lang.String WASB_BYTES_READ

See Also:

Constant Field Values

WASB_RAW_BYTES_UPLOADED

public static final java.lang.String WASB_RAW_BYTES_UPLOADED

See Also:

Constant Field Values

WASB_RAW_BYTES_DOWNLOADED

public static final java.lang.String WASB_RAW_BYTES_DOWNLOADED

See Also:

Constant Field Values

WASB_FILES_CREATED

public static final java.lang.String WASB_FILES_CREATED

See Also:

Constant Field Values

WASB_FILES_DELETED

public static final java.lang.String WASB_FILES_DELETED

See Also:

Constant Field Values

WASB_DIRECTORIES_CREATED

public static final java.lang.String WASB_DIRECTORIES_CREATED

See Also:

Constant Field Values

WASB_DIRECTORIES_DELETED

public static final java.lang.String WASB_DIRECTORIES_DELETED

See Also:

Constant Field Values

WASB_UPLOAD_RATE

public static final java.lang.String WASB_UPLOAD_RATE

See Also:

Constant Field Values

WASB_DOWNLOAD_RATE

public static final java.lang.String WASB_DOWNLOAD_RATE

See Also:

Constant Field Values

WASB_UPLOAD_LATENCY

public static final java.lang.String WASB_UPLOAD_LATENCY

See Also:

Constant Field Values

WASB_DOWNLOAD_LATENCY

public static final java.lang.String WASB_DOWNLOAD_LATENCY

See Also:

Constant Field Values

WASB_CLIENT_ERRORS

public static final java.lang.String WASB_CLIENT_ERRORS

See Also:

Constant Field Values

WASB_SERVER_ERRORS

public static final java.lang.String WASB_SERVER_ERRORS

See Also:

Constant Field Values

Constructor Detail

AzureFileSystemInstrumentation

public AzureFileSystemInstrumentation(org.apache.hadoop.conf.Configuration conf)

Method Detail

getFileSystemInstanceId

public java.util.UUID getFileSystemInstanceId()

The unique identifier for this file system in the metrics.

Returns:

The unique identifier.

getMetricsRegistryInfo

public org.apache.hadoop.metrics2.MetricsInfo getMetricsRegistryInfo()

Get the metrics registry information.

Returns:

The metrics registry information.

setAccountName

public void setAccountName(java.lang.String accountName)

Sets the account name to tag all the metrics with.

Parameters:

accountName - The account name.

setContainerName

public void setContainerName(java.lang.String containerName)

Sets the container name to tag all the metrics with.

Parameters:

containerName - The container name.

webResponse

public void webResponse()

Indicate that we just got a web response from Azure Storage. This should be called for every web request/response we do (to get accurate metrics of how we're hitting the storage service).

getCurrentWebResponses

public long getCurrentWebResponses()

Gets the current number of web responses obtained from Azure Storage.

Returns:

The number of web responses.

fileCreated

public void fileCreated()

Indicate that we just created a file through WASB.

fileDeleted

public void fileDeleted()

Indicate that we just deleted a file through WASB.

directoryCreated

public void directoryCreated()

Indicate that we just created a directory through WASB.

directoryDeleted

public void directoryDeleted()

Indicate that we just deleted a directory through WASB.

updateBytesWrittenInLastSecond

public void updateBytesWrittenInLastSecond(long currentBytesWritten)

Sets the current gauge value for how many bytes were written in the last second.

Parameters:

currentBytesWritten - The number of bytes.

updateBytesReadInLastSecond

public void updateBytesReadInLastSecond(long currentBytesRead)

Sets the current gauge value for how many bytes were read in the last second.

Parameters:

currentBytesRead - The number of bytes.

currentUploadBytesPerSecond

public void currentUploadBytesPerSecond(long bytesPerSecond)

Record the current bytes-per-second upload rate seen.

Parameters:

bytesPerSecond - The bytes per second.

currentDownloadBytesPerSecond

public void currentDownloadBytesPerSecond(long bytesPerSecond)

Record the current bytes-per-second download rate seen.

Parameters:

bytesPerSecond - The bytes per second.

rawBytesUploaded

public void rawBytesUploaded(long numberOfBytes)

Indicate that we just uploaded some data to Azure storage.

Parameters:

numberOfBytes - The raw number of bytes uploaded (including overhead).

rawBytesDownloaded

public void rawBytesDownloaded(long numberOfBytes)

Indicate that we just downloaded some data to Azure storage.

Parameters:

numberOfBytes - The raw number of bytes downloaded (including overhead).

blockUploaded

public void blockUploaded(long latency)

Indicate that we just uploaded a block and record its latency.

Parameters:

latency - The latency in milliseconds.

blockDownloaded

public void blockDownloaded(long latency)

Indicate that we just downloaded a block and record its latency.

Parameters:

latency - The latency in milliseconds.

clientErrorEncountered

public void clientErrorEncountered()

Indicate that we just encountered a client-side error.

serverErrorEncountered

public void serverErrorEncountered()

Indicate that we just encountered a server-caused error.

getBlockUploadLatency

public long getBlockUploadLatency()

Get the current rolling average of the upload latency.

Returns:

rolling average of upload latency in milliseconds.

getBlockDownloadLatency

public long getBlockDownloadLatency()

Get the current rolling average of the download latency.

Returns:

rolling average of download latency in milliseconds.

getCurrentMaximumUploadBandwidth

public long getCurrentMaximumUploadBandwidth()

Get the current maximum upload bandwidth.

Returns:

maximum upload bandwidth in bytes per second.

getCurrentMaximumDownloadBandwidth

public long getCurrentMaximumDownloadBandwidth()

Get the current maximum download bandwidth.

Returns:

maximum download bandwidth in bytes per second.

getMetrics

public void getMetrics(org.apache.hadoop.metrics2.MetricsCollector builder,
                       boolean all)

Specified by:

getMetrics in interface org.apache.hadoop.metrics2.MetricsSource