NetworkTopology (Apache Hadoop Common 3.4.1.100-eep-941 API)

java.lang.Object
- org.apache.hadoop.net.NetworkTopology

Direct Known Subclasses:

NetworkTopologyWithNodeGroup
```
@LimitedPrivate({"HDFS","MapReduce"})
@Unstable
public class NetworkTopology
extends java.lang.Object
```
The class represents a cluster of computer with a tree hierarchical network topology. For example, a cluster may be consists of many data centers filled with racks of computers. In a network topology, leaves represent data nodes (computers) and inner nodes represent switches/routers that manage traffic in/out of data centers or racks.

Nested Class Summary

Nested Classes
Modifier and Type Class Description

static class NetworkTopology.InvalidTopologyException

Nested Classes
Modifier and Type	Class	Description
`static class`	`NetworkTopology.InvalidTopologyException`

Field Summary

Fields
Modifier and Type	Field	Description
`static java.lang.String`	`DEFAULT_RACK`
`static org.slf4j.Logger`	`LOG`
`protected java.util.concurrent.locks.ReadWriteLock`	`netlock`	the lock used to manage access
`protected int`	`numOfRacks`	rack counter

Constructor Summary

Constructors
Constructor Description

NetworkTopology()

Constructors
Constructor	Description
`NetworkTopology()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`add(Node node)`	Add a leaf node Update node counter & rack counter if necessary
`Node`	`chooseRandom(java.lang.String scope)`	Randomly choose a node.
`protected Node`	`chooseRandom(java.lang.String scope, java.lang.String excludedScope, java.util.Collection<Node> excludedNodes)`
`Node`	`chooseRandom(java.lang.String scope, java.util.Collection<Node> excludedNodes)`	Randomly choose one node from scope.
`boolean`	`contains(Node node)`	Check if the tree contains node node
`int`	`countNumOfAvailableNodes(java.lang.String scope, java.util.Collection<Node> excludedNodes)`	return the number of leaves in scope but not in excludedNodes if scope starts with ~, return the number of nodes that are not in scope and excludedNodes;
`void`	`decommissionNode(Node node)`	Update empty rack number when remove a node like decommission.
`java.util.List<Node>`	`getDatanodesInRack(java.lang.String loc)`	Given a string representation of a rack, return its children
`int`	`getDistance(Node node1, Node node2)`	Return the distance between two nodes It is assumed that the distance from one node to its parent is 1 The distance between two nodes is calculated by summing up their distances to their closest common ancestor.
`static int`	`getDistanceByPath(Node node1, Node node2)`	Return the distance between two nodes by comparing their network paths without checking if they belong to the same ancestor node by reference.
`static java.lang.String`	`getFirstHalf(java.lang.String networkLocation)`
`static NetworkTopology`	`getInstance(Configuration conf)`	Get an instance of NetworkTopology based on the value of the configuration parameter net.topology.impl.
`static NetworkTopology`	`getInstance(Configuration conf, InnerNode.Factory factory)`
`static java.lang.String`	`getLastHalf(java.lang.String networkLocation)`
`java.util.List<Node>`	`getLeaves(java.lang.String scope)`	return leaves in scope
`Node`	`getNode(java.lang.String loc)`	Given a string representation of a node, return its reference
`protected Node`	`getNodeForNetworkLocation(Node node)`	Return a reference to the node given its string representation.
`int`	`getNumOfLeaves()`
`int`	`getNumOfNonEmptyRacks()`
`int`	`getNumOfRacks()`
`java.lang.String`	`getRack(java.lang.String loc)`	Given a string representation of a rack for a specific network location To be overridden in subclasses for specific NetworkTopology implementations, as alternative to overriding the full `getRack(String)` method.
`protected int`	`getWeight(Node reader, Node node)`	Returns an integer weight which specifies how far away {node} is away from {reader}.
`protected static int`	`getWeightUsingNetworkLocation(Node reader, Node node)`	Returns an integer weight which specifies how far away node is from reader.
`boolean`	`hasClusterEverBeenMultiRack()`
`protected void`	`incrementRacks()`
`protected NetworkTopology`	`init(InnerNode.Factory factory)`
`protected static boolean`	`isChildScope(java.lang.String parentScope, java.lang.String childScope)`	Checks whether one scope is contained in the other scope.
`boolean`	`isNodeGroupAware()`
`protected static boolean`	`isNodeInScope(Node node, java.lang.String scope)`	Checks whether a node belongs to the scope.
`boolean`	`isOnSameNodeGroup(Node node1, Node node2)`
`boolean`	`isOnSameRack(Node node1, Node node2)`	Check if two nodes are on the same rack
`protected boolean`	`isSameParents(Node node1, Node node2)`	Compare the parents of each node for equality
`void`	`recommissionNode(Node node)`	Update empty rack number when add a node like recommission.
`void`	`remove(Node node)`	Remove a node Update node counter and rack counter if necessary
`void`	`sortByDistance(Node reader, Node[] nodes, int activeLen)`	Sort nodes array by network distance to reader.
`<T extends Node> void`	`sortByDistance(Node reader, T[] nodes, int activeLen, java.util.function.Consumer<java.util.List<T>> secondarySort)`	Sort nodes array by network distance to reader with secondary sort.
`void`	`sortByDistanceUsingNetworkLocation(Node reader, Node[] nodes, int activeLen)`	Sort nodes array by network distance to reader with secondary sort.
`<T extends Node> void`	`sortByDistanceUsingNetworkLocation(Node reader, T[] nodes, int activeLen, java.util.function.Consumer<java.util.List<T>> secondarySort)`	Sort nodes array by network distance to reader.
`java.lang.String`	`toString()`	convert a network tree to a string.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - DEFAULT_RACK
```
public static final java.lang.String DEFAULT_RACK
```
    See Also:
    
    Constant Field Values
  - LOG
```
public static final org.slf4j.Logger LOG
```
  - numOfRacks
```
protected int numOfRacks
```
    rack counter
  - netlock
```
protected java.util.concurrent.locks.ReadWriteLock netlock
```
    the lock used to manage access
- Constructor Detail
  - NetworkTopology
```
public NetworkTopology()
```
- Method Detail
  - getInstance
```
public static NetworkTopology getInstance(Configuration conf)
```
    Get an instance of NetworkTopology based on the value of the configuration parameter net.topology.impl.
    
    Parameters:
    
    conf - the configuration to be used
    
    Returns:
    
    an instance of NetworkTopology
  - getInstance
```
public static NetworkTopology getInstance(Configuration conf,
                                          InnerNode.Factory factory)
```
  - init
```
protected NetworkTopology init(InnerNode.Factory factory)
```
  - add
```
public void add(Node node)
```
    Add a leaf node Update node counter & rack counter if necessary
    
    Parameters:
    
    node - node to be added; can be null
    
    Throws:
    
    java.lang.IllegalArgumentException - if add a node to a leave or node to be added is not a leaf
  - incrementRacks
```
protected void incrementRacks()
```
  - getNodeForNetworkLocation
```
protected Node getNodeForNetworkLocation(Node node)
```
    Return a reference to the node given its string representation. Default implementation delegates to getNode(String).
    To be overridden in subclasses for specific NetworkTopology implementations, as alternative to overriding the full add(Node) method.
    
    Parameters:
    
    node - The string representation of this node's network location is used to retrieve a Node object.
    
    Returns:
    
    a reference to the node; null if the node is not in the tree
    
    See Also:
    
    add(Node), getNode(String)
  - getDatanodesInRack
```
public java.util.List<Node> getDatanodesInRack(java.lang.String loc)
```
    Given a string representation of a rack, return its children
    
    Parameters:
    
    loc - a path-like string representation of a rack
    
    Returns:
    
    a newly allocated list with all the node's children
  - remove
```
public void remove(Node node)
```
    Remove a node Update node counter and rack counter if necessary
    
    Parameters:
    
    node - node to be removed; can be null
  - contains
```
public boolean contains(Node node)
```
    Check if the tree contains node node
    
    Parameters:
    
    node - a node
    
    Returns:
    
    true if node is already in the tree; false otherwise
  - getNode
```
public Node getNode(java.lang.String loc)
```
    Given a string representation of a node, return its reference
    
    Parameters:
    
    loc - a path-like string representation of a node
    
    Returns:
    
    a reference to the node; null if the node is not in the tree
  - hasClusterEverBeenMultiRack
```
public boolean hasClusterEverBeenMultiRack()
```
    Returns:
    
    true if this cluster has ever consisted of multiple racks, even if it is not now a multi-rack cluster.
  - getRack
```
public java.lang.String getRack(java.lang.String loc)
```
    Given a string representation of a rack for a specific network location To be overridden in subclasses for specific NetworkTopology implementations, as alternative to overriding the full getRack(String) method.
    
    Parameters:
    
    loc - a path-like string representation of a network location
    
    Returns:
    
    a rack string
  - getNumOfRacks
```
public int getNumOfRacks()
```
    Returns:
    
    the total number of racks
  - getNumOfLeaves
```
public int getNumOfLeaves()
```
    Returns:
    
    the total number of leaf nodes
  - getDistance
```
public int getDistance(Node node1,
                       Node node2)
```
    Return the distance between two nodes It is assumed that the distance from one node to its parent is 1 The distance between two nodes is calculated by summing up their distances to their closest common ancestor.
    
    Parameters:
    
    node1 - one node
    
    node2 - another node
    
    Returns:
    
    the distance between node1 and node2 which is zero if they are the same or Integer.MAX_VALUE if node1 or node2 do not belong to the cluster
  - getDistanceByPath
```
public static int getDistanceByPath(Node node1,
                                    Node node2)
```
    Return the distance between two nodes by comparing their network paths without checking if they belong to the same ancestor node by reference. It is assumed that the distance from one node to its parent is 1 The distance between two nodes is calculated by summing up their distances to their closest common ancestor.
    
    Parameters:
    
    node1 - one node
    
    node2 - another node
    
    Returns:
    
    the distance between node1 and node2
  - isOnSameRack
```
public boolean isOnSameRack(Node node1,
                            Node node2)
```
    Check if two nodes are on the same rack
    
    Parameters:
    
    node1 - one node (can be null)
    
    node2 - another node (can be null)
    
    Returns:
    
    true if node1 and node2 are on the same rack; false otherwise
    
    Throws:
    
    java.lang.IllegalArgumentException - when either node1 or node2 is null, or node1 or node2 do not belong to the cluster
  - isNodeGroupAware
```
public boolean isNodeGroupAware()
```
    Returns:
    
    Check if network topology is aware of NodeGroup.
  - isOnSameNodeGroup
```
public boolean isOnSameNodeGroup(Node node1,
                                 Node node2)
```
    Parameters:
    
    node1 - input node1.
    
    node2 - input node2.
    
    Returns:
    
    Return false directly as not aware of NodeGroup, to be override in sub-class.
  - isSameParents
```
protected boolean isSameParents(Node node1,
                                Node node2)
```
    Compare the parents of each node for equality
    To be overridden in subclasses for specific NetworkTopology implementations, as alternative to overriding the full isOnSameRack(Node, Node) method.
    
    Parameters:
    
    node1 - the first node to compare
    
    node2 - the second node to compare
    
    Returns:
    
    true if their parents are equal, false otherwise
    
    See Also:
    
    isOnSameRack(Node, Node)
  - chooseRandom
```
public Node chooseRandom(java.lang.String scope)
```
    Randomly choose a node.
    
    Parameters:
    
    scope - range of nodes from which a node will be chosen
    
    Returns:
    
    the chosen node
    
    See Also:
    
    chooseRandom(String, Collection)
  - chooseRandom
```
public Node chooseRandom(java.lang.String scope,
                         java.util.Collection<Node> excludedNodes)
```
    Randomly choose one node from scope. If scope starts with ~, choose one from the all nodes except for the ones in scope; otherwise, choose one from scope. If excludedNodes is given, choose a node that's not in excludedNodes.
    
    Parameters:
    
    scope - range of nodes from which a node will be chosen
    
    excludedNodes - nodes to be excluded from
    
    Returns:
    
    the chosen node
  - chooseRandom
```
protected Node chooseRandom(java.lang.String scope,
                            java.lang.String excludedScope,
                            java.util.Collection<Node> excludedNodes)
```
  - getLeaves
```
public java.util.List<Node> getLeaves(java.lang.String scope)
```
    return leaves in scope
    
    Parameters:
    
    scope - a path string
    
    Returns:
    
    leaves nodes under specific scope
  - countNumOfAvailableNodes
```
@VisibleForTesting
public int countNumOfAvailableNodes(java.lang.String scope,
                                    java.util.Collection<Node> excludedNodes)
```
    return the number of leaves in scope but not in excludedNodes if scope starts with ~, return the number of nodes that are not in scope and excludedNodes;
    
    Parameters:
    
    scope - a path string that may start with ~
    
    excludedNodes - a list of nodes
    
    Returns:
    
    number of available nodes
  - toString
```
public java.lang.String toString()
```
    convert a network tree to a string.
    
    Overrides:
    
    toString in class java.lang.Object
  - getFirstHalf
```
public static java.lang.String getFirstHalf(java.lang.String networkLocation)
```
    Parameters:
    
    networkLocation - input networkLocation.
    
    Returns:
    
    Divide networklocation string into two parts by last separator, and get the first part here.
  - getLastHalf
```
public static java.lang.String getLastHalf(java.lang.String networkLocation)
```
    Parameters:
    
    networkLocation - input networkLocation.
    
    Returns:
    
    Divide networklocation string into two parts by last separator, and get the second part here.
  - getWeight
```
@VisibleForTesting
protected int getWeight(Node reader,
                        Node node)
```
    Returns an integer weight which specifies how far away {node} is away from {reader}. A lower value signifies that a node is closer.
    
    Parameters:
    
    reader - Node where data will be read
    
    node - Replica of data
    
    Returns:
    
    weight
  - getWeightUsingNetworkLocation
```
@VisibleForTesting
protected static int getWeightUsingNetworkLocation(Node reader,
                                                   Node node)
```
    Returns an integer weight which specifies how far away node is from reader. A lower value signifies that a node is closer. It uses network location to calculate the weight
    
    Parameters:
    
    reader - Node where data will be read
    
    node - Replica of data
    
    Returns:
    
    weight
  - sortByDistance
```
public void sortByDistance(Node reader,
                           Node[] nodes,
                           int activeLen)
```
    Sort nodes array by network distance to reader.
    In a three-level topology, a node can be either local, on the same rack, or on a different rack from the reader. Sorting the nodes based on network distance from the reader reduces network traffic and improves performance.
    As an additional twist, we also randomize the nodes at each network distance. This helps with load balancing when there is data skew.
    
    Parameters:
    
    reader - Node where data will be read
    
    nodes - Available replicas with the requested data
    
    activeLen - Number of active nodes at the front of the array
  - sortByDistance
```
public <T extends Node> void sortByDistance(Node reader,
                                            T[] nodes,
                                            int activeLen,
                                            java.util.function.Consumer<java.util.List<T>> secondarySort)
```
    Sort nodes array by network distance to reader with secondary sort.
    In a three-level topology, a node can be either local, on the same rack, or on a different rack from the reader. Sorting the nodes based on network distance from the reader reduces network traffic and improves performance.
    As an additional twist, we also randomize the nodes at each network distance. This helps with load balancing when there is data skew.
    
    Type Parameters:
    
    T - Generics Type T
    
    Parameters:
    
    reader - Node where data will be read
    
    nodes - Available replicas with the requested data
    
    activeLen - Number of active nodes at the front of the array
    
    secondarySort - a secondary sorting strategy which can inject into that point from outside to help sort the same distance.
  - sortByDistanceUsingNetworkLocation
```
public void sortByDistanceUsingNetworkLocation(Node reader,
                                               Node[] nodes,
                                               int activeLen)
```
    Sort nodes array by network distance to reader with secondary sort.
    using network location. This is used when the reader is not a datanode. Sorting the nodes based on network distance from the reader reduces network traffic and improves performance.
    
    Parameters:
    
    reader - Node where data will be read
    
    nodes - Available replicas with the requested data
    
    activeLen - Number of active nodes at the front of the array
  - sortByDistanceUsingNetworkLocation
```
public <T extends Node> void sortByDistanceUsingNetworkLocation(Node reader,
                                                                T[] nodes,
                                                                int activeLen,
                                                                java.util.function.Consumer<java.util.List<T>> secondarySort)
```
    Sort nodes array by network distance to reader.
    using network location. This is used when the reader is not a datanode. Sorting the nodes based on network distance from the reader reduces network traffic and improves performance.
    
    Type Parameters:
    
    T - Generics Type T.
    
    Parameters:
    
    reader - Node where data will be read
    
    nodes - Available replicas with the requested data
    
    activeLen - Number of active nodes at the front of the array
    
    secondarySort - a secondary sorting strategy which can inject into that point from outside to help sort the same distance.
  - isChildScope
```
protected static boolean isChildScope(java.lang.String parentScope,
                                      java.lang.String childScope)
```
    Checks whether one scope is contained in the other scope.
    
    Parameters:
    
    parentScope - the parent scope to check
    
    childScope - the child scope which needs to be checked.
    
    Returns:
    
    true if childScope is contained within the parentScope
  - isNodeInScope
```
protected static boolean isNodeInScope(Node node,
                                       java.lang.String scope)
```
    Checks whether a node belongs to the scope.
    
    Parameters:
    
    node - the node to check.
    
    scope - scope to check.
    
    Returns:
    
    true if node lies within the scope
  - getNumOfNonEmptyRacks
```
public int getNumOfNonEmptyRacks()
```
    Returns:
    
    the number of nonempty racks
  - recommissionNode
```
public void recommissionNode(Node node)
```
    Update empty rack number when add a node like recommission.
    
    Parameters:
    
    node - node to be added; can be null
  - decommissionNode
```
public void decommissionNode(Node node)
```
    Update empty rack number when remove a node like decommission.
    
    Parameters:
    
    node - node to be added; can be null

Class NetworkTopology

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_RACK

LOG

numOfRacks

netlock

Constructor Detail

NetworkTopology

Method Detail

getInstance

getInstance

init

add

incrementRacks

getNodeForNetworkLocation

getDatanodesInRack

remove

contains

getNode

hasClusterEverBeenMultiRack

getRack

getNumOfRacks

getNumOfLeaves

getDistance

getDistanceByPath

isOnSameRack

isNodeGroupAware

isOnSameNodeGroup

isSameParents

chooseRandom

chooseRandom

chooseRandom

getLeaves

countNumOfAvailableNodes

toString

getFirstHalf

getLastHalf

getWeight

getWeightUsingNetworkLocation

sortByDistance

sortByDistance

sortByDistanceUsingNetworkLocation

sortByDistanceUsingNetworkLocation

isChildScope

isNodeInScope

getNumOfNonEmptyRacks

recommissionNode

decommissionNode