Package org.apache.hadoop.hdfs.server.blockmanagement

Class DatanodeAdminManager

java.lang.Object

org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager

@Private
public class DatanodeAdminManager
extends java.lang.Object

Manages decommissioning and maintenance state for DataNodes. A background monitor thread periodically checks the status of DataNodes that are decommissioning or entering maintenance state.

A DataNode can be decommissioned in a few situations:

• If a DN is dead, it is decommissioned immediately.
• If a DN is alive, it is decommissioned after all of its blocks are sufficiently replicated. Merely under-replicated blocks do not block decommissioning as long as they are above a replication threshold.

In the second case, the DataNode transitions to a DECOMMISSION_INPROGRESS state and is tracked by the monitor thread. The monitor periodically scans through the list of insufficiently replicated blocks on these DataNodes to determine if they can be DECOMMISSIONED. The monitor also prunes this list as blocks become replicated, so monitor scans will become more efficient over time.

DECOMMISSION_INPROGRESS nodes that become dead do not progress to DECOMMISSIONED until they become live again. This prevents potential durability loss for singly-replicated blocks (see HDFS-6791).

DataNodes can also be put under maintenance state for any short duration maintenance operations. Unlike decommissioning, blocks are not always re-replicated for the DataNodes to enter maintenance state. When the blocks are replicated at least dfs.namenode.maintenance.replication.min, DataNodes transition to IN_MAINTENANCE state. Otherwise, just like decommissioning, DataNodes transition to ENTERING_MAINTENANCE state and wait for the blocks to be sufficiently replicated and then transition to IN_MAINTENANCE state. The block replication factor is relaxed for a maximum of maintenance expiry time. When DataNodes don't transition or join the cluster back by expiry time, blocks are re-replicated just as in decommissioning case as to avoid read or write performance degradation.

This class depends on the FSNamesystem lock for synchronization.

Method Summary

Modifier and Type	Method	Description
int	getBlocksPerLock()
int	getNumNodesChecked()
int	getNumPendingNodes()
int	getNumTrackedNodes()
java.util.Queue<DatanodeDescriptor>	getPendingNodes()
int	getPendingRepLimit()
protected boolean	isSufficient(BlockInfo block, BlockCollection bc, NumberReplicas numberReplicas, boolean isDecommission, boolean isMaintenance)	Checks whether a block is sufficiently replicated/stored for DECOMMISSION_INPROGRESS or ENTERING_MAINTENANCE datanodes.
protected void	logBlockReplicationInfo(BlockInfo block, BlockCollection bc, DatanodeDescriptor srcNode, NumberReplicas num, java.lang.Iterable<DatanodeStorageInfo> storages)
void	refreshBlocksPerLock(int blocksPerLock, java.lang.String key)
void	refreshPendingRepLimit(int pendingRepLimit, java.lang.String key)
protected void	setDecommissioned(DatanodeDescriptor dn)
protected void	setInMaintenance(DatanodeDescriptor dn)
void	startDecommission(DatanodeDescriptor node)	Start decommissioning the specified datanode.
void	startMaintenance(DatanodeDescriptor node, long maintenanceExpireTimeInMS)	Start maintenance of the specified datanode.
void	stopDecommission(DatanodeDescriptor node)	Stop decommissioning the specified datanode.
void	stopMaintenance(DatanodeDescriptor node)	Stop maintenance of the specified datanode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

startDecommission

@VisibleForTesting
public void startDecommission(DatanodeDescriptor node)

Start decommissioning the specified datanode.

Parameters:

node -

stopDecommission

@VisibleForTesting
public void stopDecommission(DatanodeDescriptor node)

Stop decommissioning the specified datanode.

Parameters:

node -

startMaintenance

@VisibleForTesting
public void startMaintenance(DatanodeDescriptor node,
                             long maintenanceExpireTimeInMS)

Start maintenance of the specified datanode.

Parameters:

node -

stopMaintenance

@VisibleForTesting
public void stopMaintenance(DatanodeDescriptor node)

Stop maintenance of the specified datanode.

Parameters:

node -

setDecommissioned

protected void setDecommissioned(DatanodeDescriptor dn)

setInMaintenance

protected void setInMaintenance(DatanodeDescriptor dn)

isSufficient

protected boolean isSufficient(BlockInfo block,
                               BlockCollection bc,
                               NumberReplicas numberReplicas,
                               boolean isDecommission,
                               boolean isMaintenance)

Checks whether a block is sufficiently replicated/stored for DECOMMISSION_INPROGRESS or ENTERING_MAINTENANCE datanodes. For replicated blocks or striped blocks, full-strength replication or storage is not always necessary, hence "sufficient".

Returns:

true if sufficient, else false.

logBlockReplicationInfo

protected void logBlockReplicationInfo(BlockInfo block,
                                       BlockCollection bc,
                                       DatanodeDescriptor srcNode,
                                       NumberReplicas num,
                                       java.lang.Iterable<DatanodeStorageInfo> storages)

getNumPendingNodes

@VisibleForTesting
public int getNumPendingNodes()

getNumTrackedNodes

@VisibleForTesting
public int getNumTrackedNodes()

getNumNodesChecked

@VisibleForTesting
public int getNumNodesChecked()

getPendingNodes

@VisibleForTesting
public java.util.Queue<DatanodeDescriptor> getPendingNodes()

refreshPendingRepLimit

public void refreshPendingRepLimit(int pendingRepLimit,
                                   java.lang.String key)

getPendingRepLimit

@VisibleForTesting
public int getPendingRepLimit()

refreshBlocksPerLock

public void refreshBlocksPerLock(int blocksPerLock,
                                 java.lang.String key)

getBlocksPerLock

@VisibleForTesting
public int getBlocksPerLock()