Package org.apache.hadoop.hdfs.server.blockmanagement

Class BlockPlacementPolicyWithUpgradeDomain

java.lang.Object

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithUpgradeDomain

@Private
@Evolving
public class BlockPlacementPolicyWithUpgradeDomain
extends BlockPlacementPolicyDefault

The class is responsible for choosing the desired number of targets for placing block replicas that honors upgrade domain policy. Here is the replica placement strategy. If the writer is on a datanode, the 1st replica is placed on the local machine, otherwise a random datanode. The 2nd replica is placed on a datanode that is on a different rack. The 3rd replica is placed on a datanode which is on a different node of the rack as the second replica. All 3 replicas have unique upgrade domains.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy

BlockPlacementPolicy.NotEnoughReplicasException

Field Summary

Fields inherited from class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault

clusterMap, considerLoad, considerLoadFactor, heartbeatInterval, host2datanodeMap, tolerateHeartbeatMultiplier

Fields inherited from class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy

LOG

Constructor Summary

Constructors

Constructor	Description
BlockPlacementPolicyWithUpgradeDomain()

Method Summary

Modifier and Type	Method	Description
java.lang.String	getUpgradeDomainWithDefaultValue(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo)
void	initialize(org.apache.hadoop.conf.Configuration conf, FSClusterStats stats, org.apache.hadoop.net.NetworkTopology clusterMap, org.apache.hadoop.hdfs.server.blockmanagement.Host2NodesMap host2datanodeMap)	Used to setup a BlockPlacementPolicy object.
protected boolean	isGoodDatanode(DatanodeDescriptor node, int maxTargetPerRack, boolean considerLoad, java.util.List<DatanodeStorageInfo> results, boolean avoidStaleNodes)
boolean	isMovable(java.util.Collection<org.apache.hadoop.hdfs.protocol.DatanodeInfo> locs, org.apache.hadoop.hdfs.protocol.DatanodeInfo source, org.apache.hadoop.hdfs.protocol.DatanodeInfo target)	Check if the move is allowed.
protected java.util.Collection<DatanodeStorageInfo>	pickupReplicaSet(java.util.Collection<DatanodeStorageInfo> moreThanOne, java.util.Collection<DatanodeStorageInfo> exactlyOne, java.util.Map<java.lang.String,java.util.List<DatanodeStorageInfo>> rackMap)	Pick up replica node set for deleting replica as over-replicated.
BlockPlacementStatus	verifyBlockPlacement(org.apache.hadoop.hdfs.protocol.DatanodeInfo[] locs, int numberOfReplicas)	Verify if the block's placement meets requirement of placement policy, i.e.

Methods inherited from class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault

addToExcludedNodes, chooseDataNode, chooseDataNode, chooseFavouredNodes, chooseLocalOrFavoredStorage, chooseLocalRack, chooseLocalStorage, chooseLocalStorage, chooseRandom, chooseRandom, chooseRemoteRack, chooseReplicasToDelete, chooseReplicaToDelete, chooseTarget, chooseTarget, chooseTargetInOrder, getExcludeSlowNodesEnabled, getMaxNodesPerRack, getMinBlocksForWrite, setExcludeSlowNodesEnabled, setMinBlocksForWrite

Methods inherited from class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy

adjustSetsWithChosenReplica, getDatanodeInfo, getRack, splitNodesWithRack

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockPlacementPolicyWithUpgradeDomain

public BlockPlacementPolicyWithUpgradeDomain()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
                       FSClusterStats stats,
                       org.apache.hadoop.net.NetworkTopology clusterMap,
                       org.apache.hadoop.hdfs.server.blockmanagement.Host2NodesMap host2datanodeMap)

Description copied from class: BlockPlacementPolicy

Used to setup a BlockPlacementPolicy object. This should be defined by all implementations of a BlockPlacementPolicy.

Overrides:

initialize in class BlockPlacementPolicyDefault

Parameters:

conf - the configuration object

stats - retrieve cluster status from here

clusterMap - cluster topology

isGoodDatanode

protected boolean isGoodDatanode(DatanodeDescriptor node,
                                 int maxTargetPerRack,
                                 boolean considerLoad,
                                 java.util.List<DatanodeStorageInfo> results,
                                 boolean avoidStaleNodes)

getUpgradeDomainWithDefaultValue

public java.lang.String getUpgradeDomainWithDefaultValue(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo)

verifyBlockPlacement

public BlockPlacementStatus verifyBlockPlacement(org.apache.hadoop.hdfs.protocol.DatanodeInfo[] locs,
                                                 int numberOfReplicas)

Description copied from class: BlockPlacementPolicy

Verify if the block's placement meets requirement of placement policy, i.e. replicas are placed on no less than minRacks racks in the system.

Overrides:

verifyBlockPlacement in class BlockPlacementPolicyDefault

Parameters:

locs - block with locations

numberOfReplicas - replica number of file to be verified

Returns:

the result of verification

pickupReplicaSet

protected java.util.Collection<DatanodeStorageInfo> pickupReplicaSet(java.util.Collection<DatanodeStorageInfo> moreThanOne,
                                                                     java.util.Collection<DatanodeStorageInfo> exactlyOne,
                                                                     java.util.Map<java.lang.String,java.util.List<DatanodeStorageInfo>> rackMap)

Description copied from class: BlockPlacementPolicyDefault

Pick up replica node set for deleting replica as over-replicated. First set contains replica nodes on rack with more than one replica while second set contains remaining replica nodes. If only 1 rack, pick all. If 2 racks, pick all that have more than 1 replicas on the same rack; if no such replicas, pick all. If 3 or more racks, pick all.

Overrides:

pickupReplicaSet in class BlockPlacementPolicyDefault

isMovable

public boolean isMovable(java.util.Collection<org.apache.hadoop.hdfs.protocol.DatanodeInfo> locs,
                         org.apache.hadoop.hdfs.protocol.DatanodeInfo source,
                         org.apache.hadoop.hdfs.protocol.DatanodeInfo target)

Description copied from class: BlockPlacementPolicy

Check if the move is allowed. Used by balancer and other tools.

Overrides:

isMovable in class BlockPlacementPolicyDefault

Parameters:

locs - all replicas including source and target

source - source replica of the move

target - target replica of the move