Package org.apache.hadoop.yarn.server.nodemanager.util

Class NodeManagerHardwareUtils

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils

@Private
@Unstable
public class NodeManagerHardwareUtils
extends java.lang.Object

Helper class to determine hardware related characteristics such as the number of processors and the amount of memory on the node.

Constructor Summary

Constructors

Constructor	Description
NodeManagerHardwareUtils()

Method Summary

Modifier and Type	Method	Description
static long	getContainerMemoryMB(org.apache.hadoop.conf.Configuration conf)	Function to return how much memory we should set aside for YARN containers.
static long	getContainerMemoryMB(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)	Function to return how much memory we should set aside for YARN containers.
static float	getContainersCPUs(org.apache.hadoop.conf.Configuration conf)	Returns the fraction of CPUs that should be used for YARN containers.
static float	getContainersCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)	Returns the fraction of CPUs that should be used for YARN containers.
static int	getNodeCpuPercentage(org.apache.hadoop.conf.Configuration conf)	Gets the percentage of physical CPU that is configured for YARN containers.
static int	getNodeCPUs(org.apache.hadoop.conf.Configuration conf)	Returns the number of CPUs on the node.
static int	getNodeCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)	Returns the number of CPUs on the node.
static org.apache.hadoop.yarn.api.records.Resource	getNodeResources(org.apache.hadoop.conf.Configuration configuration)	Get the resources for the node.
static int	getVCores(org.apache.hadoop.conf.Configuration conf)	Function to return the number of vcores on the system that can be used for YARN containers.
static int	getVCores(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin, org.apache.hadoop.conf.Configuration conf)	Function to return the number of vcores on the system that can be used for YARN containers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NodeManagerHardwareUtils

public NodeManagerHardwareUtils()

Method Detail

getNodeCPUs

public static int getNodeCPUs(org.apache.hadoop.conf.Configuration conf)

Returns the number of CPUs on the node. This value depends on the configuration setting which decides whether to count logical processors (such as hyperthreads) as cores or not.

Parameters:

conf - - Configuration object

Returns:

Number of CPUs

getNodeCPUs

public static int getNodeCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin,
                              org.apache.hadoop.conf.Configuration conf)

Returns the number of CPUs on the node. This value depends on the configuration setting which decides whether to count logical processors (such as hyperthreads) as cores or not.

Parameters:

plugin - - ResourceCalculatorPlugin object to determine hardware specs

conf - - Configuration object

Returns:

Number of CPU cores on the node.

getContainersCPUs

public static float getContainersCPUs(org.apache.hadoop.conf.Configuration conf)

Returns the fraction of CPUs that should be used for YARN containers. The number is derived based on various configuration params such as YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT

Parameters:

conf - - Configuration object

Returns:

Fraction of CPUs to be used for YARN containers

getContainersCPUs

public static float getContainersCPUs(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin,
                                      org.apache.hadoop.conf.Configuration conf)

Returns the fraction of CPUs that should be used for YARN containers. The number is derived based on various configuration params such as YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT

Parameters:

plugin - - ResourceCalculatorPlugin object to determine hardware specs

conf - - Configuration object

Returns:

Fraction of CPUs to be used for YARN containers

getNodeCpuPercentage

public static int getNodeCpuPercentage(org.apache.hadoop.conf.Configuration conf)

Gets the percentage of physical CPU that is configured for YARN containers. This is percent > 0 and <= 100 based on YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT

Parameters:

conf - Configuration object

Returns:

percent > 0 and <= 100

getVCores

public static int getVCores(org.apache.hadoop.conf.Configuration conf)

Function to return the number of vcores on the system that can be used for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager cores. 2. If the config variable yarn.nodemanager.cpu.use_logical_processors is set to true, it returns the logical processor count(count hyperthreads as cores), else it returns the physical cores count.

Parameters:

conf - - the configuration for the NodeManager

Returns:

the number of cores to be used for YARN containers

getVCores

public static int getVCores(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin,
                            org.apache.hadoop.conf.Configuration conf)

Function to return the number of vcores on the system that can be used for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager cores. 2. If the config variable yarn.nodemanager.cpu.use_logical_processors is set to true, it returns the logical processor count(count hyperthreads as cores), else it returns the physical cores count.

Parameters:

plugin - - ResourceCalculatorPlugin object to determine hardware specs

conf - - the configuration for the NodeManager

Returns:

the number of cores to be used for YARN containers

getContainerMemoryMB

public static long getContainerMemoryMB(org.apache.hadoop.conf.Configuration conf)

Function to return how much memory we should set aside for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager physical memory. 2. If the OS has a ResourceCalculatorPlugin implemented, the calculation is 0.8 * (RAM - 2 * JVM-memory) i.e. use 80% of the memory after accounting for memory used by the DataNode and the NodeManager. If the number is less than 1GB, log a warning message.

Parameters:

conf - - the configuration for the NodeManager

Returns:

the amount of memory that will be used for YARN containers in MB.

getContainerMemoryMB

public static long getContainerMemoryMB(org.apache.hadoop.yarn.util.ResourceCalculatorPlugin plugin,
                                        org.apache.hadoop.conf.Configuration conf)

Function to return how much memory we should set aside for YARN containers. If a number is specified in the configuration file, then that number is returned. If nothing is specified - 1. If the OS is an "unknown" OS(one for which we don't have ResourceCalculatorPlugin implemented), return the default NodeManager physical memory. 2. If the OS has a ResourceCalculatorPlugin implemented, the calculation is 0.8 * (RAM - 2 * JVM-memory) i.e. use 80% of the memory after accounting for memory used by the DataNode and the NodeManager. If the number is less than 1GB, log a warning message.

Parameters:

plugin - - ResourceCalculatorPlugin object to determine hardware specs

conf - - the configuration for the NodeManager

Returns:

the amount of memory that will be used for YARN containers in MB.

getNodeResources

public static org.apache.hadoop.yarn.api.records.Resource getNodeResources(org.apache.hadoop.conf.Configuration configuration)

Get the resources for the node.

Parameters:

configuration - configuration file

Returns:

the resources for the node