Package org.apache.hadoop.hdfs.server.datanode.fsdataset

Interface FsDatasetSpi<V extends FsVolumeSpi>

All Superinterfaces:

FSDatasetMBean, org.apache.hadoop.metrics2.MetricsSource

@Private
public interface FsDatasetSpi<V extends FsVolumeSpi>
extends FSDatasetMBean

This is a service provider interface for the underlying storage that stores replicas for a data node. The default implementation stores replicas on local drives.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	FsDatasetSpi.Factory<D extends FsDatasetSpi<?>>	A factory for creating FsDatasetSpi objects.
static class	FsDatasetSpi.FsVolumeReferences	It behaviors as an unmodifiable list of FsVolume.

Method Summary

Modifier and Type	Method	Description
DataNodeLockManager<? extends AutoCloseDataSetLock>	acquireDatasetLockManager()	Acquire lock Manager for the data set.
void	addBlockPool(java.lang.String bpid, org.apache.hadoop.conf.Configuration conf)	add new block pool ID
void	addVolume(StorageLocation location, java.util.List<NamespaceInfo> nsInfos)	Add a new volume to the FsDataset.
void	adjustCrcChannelPosition(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, ReplicaOutputStreams outs, int checksumSize)	Sets the file pointer of the checksum stream so that the last checksum will be overwritten
ReplicaHandler	append(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen)	Append to a finalized replica and returns the meta info of the replica.
void	cache(java.lang.String bpid, long[] blockIds)	Caches the specified block
void	checkAndUpdate(java.lang.String bpid, FsVolumeSpi.ScanInfo info)	Check whether the in-memory block record matches the block on the disk, and, in case that they are not matched, update the record or mark it as corrupted.
void	checkBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long minLength, HdfsServerConstants.ReplicaState state)	Check if a block is valid.
void	clearRollingUpgradeMarker(java.lang.String bpid)	Delete the rolling upgrade marker file if it exists.
void	clearTrash(java.lang.String bpid)	Clear trash
boolean	contains(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)	Does the dataset contain the block?
ReplicaInPipeline	convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock temporary)	Covert a temporary replica to a RBW.
ReplicaHandler	createRbw(org.apache.hadoop.fs.StorageType storageType, java.lang.String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean allowLazyPersist)	Creates a RBW replica and returns the meta info of the replica
ReplicaHandler	createRbw(org.apache.hadoop.fs.StorageType storageType, java.lang.String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean allowLazyPersist, long newGS)	Creates a RBW replica and returns the meta info of the replica
ReplicaHandler	createTemporary(org.apache.hadoop.fs.StorageType storageType, java.lang.String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean isTransfer)	Creates a temporary replica and returns the meta information of the replica .
java.util.Set<? extends Replica>	deepCopyReplica(java.lang.String bpid)	Deep copy the replica info belonging to given block pool.
void	deleteBlockPool(java.lang.String bpid, boolean force)	Deletes the block pool directories.
void	enableTrash(java.lang.String bpid)	Enable 'trash' for the given dataset.
void	finalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean fsyncDir)	Finalizes the block previously opened for writing using writeToBlock.
java.io.InputStream	getBlockInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long seekOffset)	Returns an input stream at specified offset of the specified block.
org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo	getBlockLocalPathInfo(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)	Get BlockLocalPathInfo for the given block.
java.util.Map<org.apache.hadoop.hdfs.server.protocol.DatanodeStorage,BlockListAsLongs>	getBlockReports(java.lang.String bpid)	Returns one block report per volume.
java.util.List<java.lang.Long>	getCacheReport(java.lang.String bpid)	Returns the cache report - the full list of cached block IDs of a block pool.
java.util.List<ReplicaInfo>	getFinalizedBlocks(java.lang.String bpid)	Gets a list of references to the finalized blocks for the given block pool.
FsDatasetSpi.FsVolumeReferences	getFsVolumeReferences()	Returns a list of FsVolumes that hold reference counts.
long	getLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)	Returns the specified block's on-disk length (excluding metadata).
LengthInputStream	getMetaDataInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
MountVolumeMap	getMountVolumeMap()	Get relationship between disk mount and FsVolume.
boolean	getPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)	Check whether the block was pinned
Replica	getReplica(java.lang.String bpid, long blockId)	Deprecated.
java.lang.String	getReplicaString(java.lang.String bpid, long blockId)
long	getReplicaVisibleLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)	Get visible length of the specified replica.
org.apache.hadoop.hdfs.server.protocol.DatanodeStorage	getStorage(java.lang.String storageUuid)
org.apache.hadoop.hdfs.server.protocol.StorageReport[]	getStorageReports(java.lang.String bpid)
org.apache.hadoop.hdfs.protocol.Block	getStoredBlock(java.lang.String bpid, long blkid)
ReplicaInputStreams	getTmpInputStreams(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long blkoff, long ckoff)	Returns an input stream at specified offset of the specified block.
V	getVolume(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
VolumeFailureSummary	getVolumeFailureSummary()	Returns info about volume failures.
java.util.Map<java.lang.String,java.lang.Object>	getVolumeInfoMap()
java.util.List<FsVolumeImpl>	getVolumeList()	Get the volume list.
void	handleVolumeFailures(java.util.Set<FsVolumeSpi> failedVolumes)	Check if all the data directories are healthy
boolean	hasEnoughResource()	Checks how many valid storage volumes there are in the DataNode.
ReplicaRecoveryInfo	initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)	Initialize a replica recovery.
void	invalidate(java.lang.String bpid, org.apache.hadoop.hdfs.protocol.Block[] invalidBlks)	Invalidates the specified blocks.
void	invalidateMissingBlock(java.lang.String bpid, org.apache.hadoop.hdfs.protocol.Block block)	Invalidate a block which is not found on disk.
boolean	isCached(java.lang.String bpid, long blockId)	Determine if the specified block is cached.
boolean	isDeletingBlock(java.lang.String bpid, long blockId)	Confirm whether the block is deleting
boolean	isValidBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)	Is the block valid?
boolean	isValidRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)	Is the block a valid RBW?
ReplicaInfo	moveBlockAcrossStorage(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, org.apache.hadoop.fs.StorageType targetStorageType, java.lang.String storageId)	Move block from one storage to another storage
ReplicaInfo	moveBlockAcrossVolumes(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, FsVolumeSpi destination)	Moves a given block from one volume to another volume.
void	onCompleteLazyPersist(java.lang.String bpId, long blockId, long creationTime, java.io.File[] savedFiles, V targetVolume)	Callback from RamDiskAsyncLazyPersistService upon async lazy persist task end
void	onFailLazyPersist(java.lang.String bpId, long blockId)	Callback from RamDiskAsyncLazyPersistService upon async lazy persist task fail
ReplicaHandler	recoverAppend(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen)	Recover a failed append to a finalized replica and returns the meta info of the replica.
Replica	recoverClose(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen)	Recover a failed pipeline close.
ReplicaHandler	recoverRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd)	Recovers a RBW replica and returns the meta info of the replica.
void	removeVolumes(java.util.Collection<StorageLocation> volumes, boolean clearFailure)	Removes a collection of volumes from FsDataset.
default void	setLastDirScannerFinishTime(long time)	Set the last time in milliseconds when the directory scanner successfully ran.
void	setPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)	Set a block to be pinned on this datanode so that it cannot be moved by Balancer/Mover.
void	setRollingUpgradeMarker(java.lang.String bpid)	Create a marker file indicating that a rolling upgrade is in progress.
void	shutdown()	Shutdown the FSDataset
void	shutdownBlockPool(java.lang.String bpid)	Shutdown and remove the block pool from underlying storage.
void	submitBackgroundSyncFileRangeRequest(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaOutputStreams outs, long offset, long nbytes, int flags)	submit a sync_file_range request to AsyncDiskService.
boolean	trashEnabled(java.lang.String bpid)
void	uncache(java.lang.String bpid, long[] blockIds)	Uncaches the specified blocks
void	unfinalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)	Unfinalizes the block previously opened for writing using writeToBlock.
Replica	updateReplicaUnderRecovery(org.apache.hadoop.hdfs.protocol.ExtendedBlock oldBlock, long recoveryId, long newBlockId, long newLength)	Update replica's generation stamp and length and finalize it.

Methods inherited from interface org.apache.hadoop.hdfs.server.datanode.metrics.FSDatasetMBean

getBlockPoolUsed, getCacheCapacity, getCacheUsed, getCapacity, getDfsUsed, getEstimatedCapacityLostTotal, getFailedStorageLocations, getLastDirScannerFinishTime, getLastVolumeFailureDate, getNumBlocksCached, getNumBlocksFailedToCache, getNumBlocksFailedToUncache, getNumFailedVolumes, getPendingAsyncDeletions, getRemaining, getStorageInfo

Methods inherited from interface org.apache.hadoop.metrics2.MetricsSource

getMetrics

Method Detail

getFsVolumeReferences

FsDatasetSpi.FsVolumeReferences getFsVolumeReferences()

Returns a list of FsVolumes that hold reference counts. The caller must release the reference of each volume by calling FsDatasetSpi.FsVolumeReferences.close().

addVolume

void addVolume(StorageLocation location,
               java.util.List<NamespaceInfo> nsInfos)
        throws java.io.IOException

Add a new volume to the FsDataset. If the FSDataset supports block scanning, this function registers the new volume with the block scanner.

Parameters:

location - The storage location for the new volume.

nsInfos - Namespace information for the new volume.

Throws:

java.io.IOException

removeVolumes

void removeVolumes(java.util.Collection<StorageLocation> volumes,
                   boolean clearFailure)

Removes a collection of volumes from FsDataset. If the FSDataset supports block scanning, this function removes the volumes from the block scanner.

Parameters:

volumes - The paths of the volumes to be removed.

clearFailure - set true to clear the failure information about the volumes.

getStorage

org.apache.hadoop.hdfs.server.protocol.DatanodeStorage getStorage(java.lang.String storageUuid)

Returns:

a storage with the given storage ID

getStorageReports

org.apache.hadoop.hdfs.server.protocol.StorageReport[] getStorageReports(java.lang.String bpid)
                                                                  throws java.io.IOException

Returns:

one or more storage reports for attached volumes.

Throws:

java.io.IOException

getVolume

V getVolume(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)

Returns:

the volume that contains a replica of the block.

getVolumeInfoMap

java.util.Map<java.lang.String,java.lang.Object> getVolumeInfoMap()

Returns:

a volume information map (name => info).

getVolumeFailureSummary

VolumeFailureSummary getVolumeFailureSummary()

Returns info about volume failures.

Returns:

info about volume failures, possibly null

getFinalizedBlocks

java.util.List<ReplicaInfo> getFinalizedBlocks(java.lang.String bpid)

Gets a list of references to the finalized blocks for the given block pool.

Callers of this function should call acquireDatasetLockManager() to avoid blocks' status being changed during list iteration.

Returns:

a list of references to the finalized blocks for the given block pool.

checkAndUpdate

void checkAndUpdate(java.lang.String bpid,
                    FsVolumeSpi.ScanInfo info)
             throws java.io.IOException

Check whether the in-memory block record matches the block on the disk, and, in case that they are not matched, update the record or mark it as corrupted.

Throws:

java.io.IOException

getMetaDataInputStream

LengthInputStream getMetaDataInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
                                  throws java.io.IOException

Parameters:

b - - the block

Returns:

a stream if the meta-data of the block exists; otherwise, return null.

Throws:

java.io.IOException

getLength

long getLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
        throws java.io.IOException

Returns the specified block's on-disk length (excluding metadata).

Returns:

the specified block's on-disk length (excluding metadta)

Throws:

java.io.IOException - on error

getReplica

@Deprecated
Replica getReplica(java.lang.String bpid,
                   long blockId)

Deprecated.

Get reference to the replica meta info in the replicasMap. To be called from methods that are synchronized on implementations of FsDatasetSpi

Returns:

replica from the replicas map

getReplicaString

java.lang.String getReplicaString(java.lang.String bpid,
                                  long blockId)

Returns:

replica meta information

getStoredBlock

org.apache.hadoop.hdfs.protocol.Block getStoredBlock(java.lang.String bpid,
                                                     long blkid)
                                              throws java.io.IOException

Returns:

the generation stamp stored with the block.

Throws:

java.io.IOException

getBlockInputStream

java.io.InputStream getBlockInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                                        long seekOffset)
                                 throws java.io.IOException

Returns an input stream at specified offset of the specified block.

Parameters:

b - block

seekOffset - offset with in the block to seek to

Returns:

an input stream to read the contents of the specified block, starting at the offset

Throws:

java.io.IOException

getTmpInputStreams

ReplicaInputStreams getTmpInputStreams(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                                       long blkoff,
                                       long ckoff)
                                throws java.io.IOException

Returns an input stream at specified offset of the specified block. The block is still in the tmp directory and is not finalized

Returns:

an input stream to read the contents of the specified block, starting at the offset

Throws:

java.io.IOException

createTemporary

ReplicaHandler createTemporary(org.apache.hadoop.fs.StorageType storageType,
                               java.lang.String storageId,
                               org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                               boolean isTransfer)
                        throws java.io.IOException

Creates a temporary replica and returns the meta information of the replica .

Parameters:

b - block

Returns:

the meta info of the replica which is being written to

Throws:

java.io.IOException - if an error occurs

createRbw

ReplicaHandler createRbw(org.apache.hadoop.fs.StorageType storageType,
                         java.lang.String storageId,
                         org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                         boolean allowLazyPersist)
                  throws java.io.IOException

Creates a RBW replica and returns the meta info of the replica

Parameters:

b - block

Returns:

the meta info of the replica which is being written to

Throws:

java.io.IOException - if an error occurs

createRbw

ReplicaHandler createRbw(org.apache.hadoop.fs.StorageType storageType,
                         java.lang.String storageId,
                         org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                         boolean allowLazyPersist,
                         long newGS)
                  throws java.io.IOException

Creates a RBW replica and returns the meta info of the replica

Parameters:

b - block

Returns:

the meta info of the replica which is being written to

Throws:

java.io.IOException - if an error occurs

recoverRbw

ReplicaHandler recoverRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                          long newGS,
                          long minBytesRcvd,
                          long maxBytesRcvd)
                   throws java.io.IOException

Recovers a RBW replica and returns the meta info of the replica.

Parameters:

b - block

newGS - the new generation stamp for the replica

minBytesRcvd - the minimum number of bytes that the replica could have

maxBytesRcvd - the maximum number of bytes that the replica could have

Returns:

the meta info of the replica which is being written to

Throws:

java.io.IOException - if an error occurs

convertTemporaryToRbw

ReplicaInPipeline convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock temporary)
                                 throws java.io.IOException

Covert a temporary replica to a RBW.

Parameters:

temporary - the temporary replica being converted

Returns:

the result RBW

Throws:

java.io.IOException

append

ReplicaHandler append(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                      long newGS,
                      long expectedBlockLen)
               throws java.io.IOException

Append to a finalized replica and returns the meta info of the replica.

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Returns:

the meata info of the replica which is being written to

Throws:

java.io.IOException

recoverAppend

ReplicaHandler recoverAppend(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                             long newGS,
                             long expectedBlockLen)
                      throws java.io.IOException

Recover a failed append to a finalized replica and returns the meta info of the replica.

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Returns:

the meta info of the replica which is being written to

Throws:

java.io.IOException

recoverClose

Replica recoverClose(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                     long newGS,
                     long expectedBlockLen)
              throws java.io.IOException

Recover a failed pipeline close. It bumps the replica's generation stamp and finalize it if RBW replica

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Returns:

the storage uuid of the replica.

Throws:

java.io.IOException

finalizeBlock

void finalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                   boolean fsyncDir)
            throws java.io.IOException

Finalizes the block previously opened for writing using writeToBlock. The block size is what is in the parameter b and it must match the amount of data written

Parameters:

b - Block to be finalized

fsyncDir - whether to sync the directory changes to durable device.

Throws:

java.io.IOException

org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException - if the replica can not be found when the block is been finalized. For instance, the block resides on an HDFS volume that has been removed.

unfinalizeBlock

void unfinalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
              throws java.io.IOException

Unfinalizes the block previously opened for writing using writeToBlock. The temporary file associated with this block is deleted.

Throws:

java.io.IOException

getBlockReports

java.util.Map<org.apache.hadoop.hdfs.server.protocol.DatanodeStorage,BlockListAsLongs> getBlockReports(java.lang.String bpid)

Returns one block report per volume.

Parameters:

bpid - Block Pool Id

Returns:

- a map of DatanodeStorage to block report for the volume.

getCacheReport

java.util.List<java.lang.Long> getCacheReport(java.lang.String bpid)

Returns the cache report - the full list of cached block IDs of a block pool.

Parameters:

bpid - Block Pool Id

Returns:

the cache report - the full list of cached block IDs.

contains

boolean contains(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)

Does the dataset contain the block?

checkBlock

void checkBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                long minLength,
                HdfsServerConstants.ReplicaState state)
         throws org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException,
                UnexpectedReplicaStateException,
                java.io.FileNotFoundException,
                java.io.EOFException,
                java.io.IOException

Check if a block is valid.

Parameters:

b - The block to check.

minLength - The minimum length that the block must have. May be 0.

state - If this is null, it is ignored. If it is non-null, we will check that the replica has this state.

Throws:

org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException - If the replica is not found

UnexpectedReplicaStateException - If the replica is not in the expected state.

java.io.FileNotFoundException - If the block file is not found or there was an error locating it.

java.io.EOFException - If the replica length is too short.

java.io.IOException - May be thrown from the methods called.

isValidBlock

boolean isValidBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)

Is the block valid?

Returns:

- true if the specified block is valid

isValidRbw

boolean isValidRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)

Is the block a valid RBW?

Returns:

- true if the specified block is a valid RBW

invalidate

void invalidate(java.lang.String bpid,
                org.apache.hadoop.hdfs.protocol.Block[] invalidBlks)
         throws java.io.IOException

Invalidates the specified blocks.

Parameters:

bpid - Block pool Id

invalidBlks - - the blocks to be invalidated

Throws:

java.io.IOException

invalidateMissingBlock

void invalidateMissingBlock(java.lang.String bpid,
                            org.apache.hadoop.hdfs.protocol.Block block)
                     throws java.io.IOException

Invalidate a block which is not found on disk.

Parameters:

bpid - the block pool ID.

block - The block to be invalidated.

Throws:

java.io.IOException

cache

void cache(java.lang.String bpid,
           long[] blockIds)

Caches the specified block

Parameters:

bpid - Block pool id

blockIds - - block ids to cache

uncache

void uncache(java.lang.String bpid,
             long[] blockIds)

Uncaches the specified blocks

Parameters:

bpid - Block pool id

blockIds - - blocks ids to uncache

isCached

boolean isCached(java.lang.String bpid,
                 long blockId)

Determine if the specified block is cached.

Parameters:

bpid - Block pool id

blockId - - block id

Returns:

true if the block is cached

handleVolumeFailures

void handleVolumeFailures(java.util.Set<FsVolumeSpi> failedVolumes)

Check if all the data directories are healthy

Parameters:

failedVolumes -

shutdown

void shutdown()

Shutdown the FSDataset

adjustCrcChannelPosition

void adjustCrcChannelPosition(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
                              ReplicaOutputStreams outs,
                              int checksumSize)
                       throws java.io.IOException

Sets the file pointer of the checksum stream so that the last checksum will be overwritten

Parameters:

b - block

outs - The streams for the data file and checksum file

checksumSize - number of bytes each checksum has

Throws:

java.io.IOException

hasEnoughResource

boolean hasEnoughResource()

Checks how many valid storage volumes there are in the DataNode.

Returns:

true if more than the minimum number of valid volumes are left in the FSDataSet.

getReplicaVisibleLength

long getReplicaVisibleLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)
                      throws java.io.IOException

Get visible length of the specified replica.

Throws:

java.io.IOException

initReplicaRecovery

ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                 throws java.io.IOException

Initialize a replica recovery.

Returns:

actual state of the replica on this data-node or null if data-node does not have the replica.

Throws:

java.io.IOException

updateReplicaUnderRecovery

Replica updateReplicaUnderRecovery(org.apache.hadoop.hdfs.protocol.ExtendedBlock oldBlock,
                                   long recoveryId,
                                   long newBlockId,
                                   long newLength)
                            throws java.io.IOException

Update replica's generation stamp and length and finalize it.

Returns:

the ID of storage that stores the block

Throws:

java.io.IOException

addBlockPool

void addBlockPool(java.lang.String bpid,
                  org.apache.hadoop.conf.Configuration conf)
           throws java.io.IOException

add new block pool ID

Parameters:

bpid - Block pool Id

conf - Configuration

Throws:

java.io.IOException

shutdownBlockPool

void shutdownBlockPool(java.lang.String bpid)

Shutdown and remove the block pool from underlying storage.

Parameters:

bpid - Block pool Id to be removed

deleteBlockPool

void deleteBlockPool(java.lang.String bpid,
                     boolean force)
              throws java.io.IOException

Deletes the block pool directories. If force is false, directories are deleted only if no block files exist for the block pool. If force is true entire directory for the blockpool is deleted along with its contents.

Parameters:

bpid - BlockPool Id to be deleted.

force - If force is false, directories are deleted only if no block files exist for the block pool, otherwise entire directory for the blockpool is deleted along with its contents.

Throws:

java.io.IOException

getBlockLocalPathInfo

org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo getBlockLocalPathInfo(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
                                                                  throws java.io.IOException

Get BlockLocalPathInfo for the given block.

Throws:

java.io.IOException

enableTrash

void enableTrash(java.lang.String bpid)

Enable 'trash' for the given dataset. When trash is enabled, files are moved to a separate trash directory instead of being deleted immediately. This can be useful for example during rolling upgrades.

clearTrash

void clearTrash(java.lang.String bpid)

Clear trash

trashEnabled

boolean trashEnabled(java.lang.String bpid)

Returns:

true when trash is enabled

setRollingUpgradeMarker

void setRollingUpgradeMarker(java.lang.String bpid)
                      throws java.io.IOException

Create a marker file indicating that a rolling upgrade is in progress.

Throws:

java.io.IOException

clearRollingUpgradeMarker

void clearRollingUpgradeMarker(java.lang.String bpid)
                        throws java.io.IOException

Delete the rolling upgrade marker file if it exists.

Parameters:

bpid -

Throws:

java.io.IOException

submitBackgroundSyncFileRangeRequest

void submitBackgroundSyncFileRangeRequest(org.apache.hadoop.hdfs.protocol.ExtendedBlock block,
                                          ReplicaOutputStreams outs,
                                          long offset,
                                          long nbytes,
                                          int flags)

submit a sync_file_range request to AsyncDiskService.

onCompleteLazyPersist

void onCompleteLazyPersist(java.lang.String bpId,
                           long blockId,
                           long creationTime,
                           java.io.File[] savedFiles,
                           V targetVolume)

Callback from RamDiskAsyncLazyPersistService upon async lazy persist task end

onFailLazyPersist

void onFailLazyPersist(java.lang.String bpId,
                       long blockId)

Callback from RamDiskAsyncLazyPersistService upon async lazy persist task fail

moveBlockAcrossStorage

ReplicaInfo moveBlockAcrossStorage(org.apache.hadoop.hdfs.protocol.ExtendedBlock block,
                                   org.apache.hadoop.fs.StorageType targetStorageType,
                                   java.lang.String storageId)
                            throws java.io.IOException

Move block from one storage to another storage

Throws:

java.io.IOException

setPinning

void setPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)
         throws java.io.IOException

Set a block to be pinned on this datanode so that it cannot be moved by Balancer/Mover. It is a no-op when dfs.datanode.block-pinning.enabled is set to false.

Throws:

java.io.IOException

getPinning

boolean getPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)
            throws java.io.IOException

Check whether the block was pinned

Throws:

java.io.IOException

isDeletingBlock

boolean isDeletingBlock(java.lang.String bpid,
                        long blockId)

Confirm whether the block is deleting

moveBlockAcrossVolumes

ReplicaInfo moveBlockAcrossVolumes(org.apache.hadoop.hdfs.protocol.ExtendedBlock block,
                                   FsVolumeSpi destination)
                            throws java.io.IOException

Moves a given block from one volume to another volume. This is used by disk balancer.

Parameters:

block - - ExtendedBlock

destination - - Destination volume

Returns:

Old replica info

Throws:

java.io.IOException

acquireDatasetLockManager

DataNodeLockManager<? extends AutoCloseDataSetLock> acquireDatasetLockManager()

Acquire lock Manager for the data set. This prevents other threads from modifying the volume map structure inside the datanode.

Returns:

The AutoClosable read lock instance.

deepCopyReplica

java.util.Set<? extends Replica> deepCopyReplica(java.lang.String bpid)
                                          throws java.io.IOException

Deep copy the replica info belonging to given block pool.

Parameters:

bpid - Specified block pool id.

Returns:

A set of replica info.

Throws:

java.io.IOException

getMountVolumeMap

MountVolumeMap getMountVolumeMap()
                          throws java.io.IOException

Get relationship between disk mount and FsVolume.

Returns:

Disk mount and FsVolume relationship.

Throws:

java.io.IOException

getVolumeList

java.util.List<FsVolumeImpl> getVolumeList()

Get the volume list.

setLastDirScannerFinishTime

default void setLastDirScannerFinishTime(long time)

Set the last time in milliseconds when the directory scanner successfully ran.

Parameters:

time - the last time in milliseconds when the directory scanner successfully ran.