Package org.apache.hadoop.yarn.util.resource

Class ResourceUtils

java.lang.Object

org.apache.hadoop.yarn.util.resource.ResourceUtils

public class ResourceUtils
extends java.lang.Object

Helper class to read the resource-types to be supported by the system.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	EXTERNAL_VOLUME_RESOURCE_TAG
static java.lang.String	MAXIMUM_ALLOCATION
static java.lang.String	MINIMUM_ALLOCATION
static java.util.regex.Pattern	RESOURCE_REQUEST_VALUE_PATTERN
static java.lang.String	TAGS
static java.lang.String	TYPE
static java.lang.String	UNITS
static java.lang.String	YARN_IO_OPTIONAL

Method Summary

Modifier and Type	Method	Description
static void	areMandatoryResourcesAvailable(Resource res)	Are mandatory resources like memory-mb, vcores available? If not, throw exceptions.
static Resource	createResourceFromString(java.lang.String resourceStr, java.util.List<ResourceTypeInfo> resourceTypeInfos)
static ResourceInformation[]	createResourceTypesArray(java.util.Map<java.lang.String,java.lang.Long> res)	Create an array of ResourceInformation objects corresponding to the passed in map of names to values.
static Resource	createResourceWithSameValue(long value)	Return a new Resource instance with all resource values initialized to value.
static Resource	fetchMaximumAllocationFromConfig(org.apache.hadoop.conf.Configuration conf)	Get maximum allocation from config, *THIS WILL NOT UPDATE INTERNAL DATA.
static java.lang.StringBuilder	getCustomResourcesStrings(Resource resource)
static java.lang.String	getDefaultUnit(java.lang.String resourceType)	Get default unit by given resource type.
static java.util.Map<java.lang.String,ResourceInformation>	getNodeResourceInformation(org.apache.hadoop.conf.Configuration conf)	Function to get the resources for a node.
static int	getNumberOfCountableResourceTypes()
static int	getNumberOfKnownResourceTypes()
static java.util.List<ResourceInformation>	getRequestedResourcesFromConfig(org.apache.hadoop.conf.Configuration configuration, java.lang.String prefix)	From a given configuration get all entries representing requested resources: entries that match the {prefix}{resourceName}={value}[{units}] pattern.
static java.util.List<ResourceTypeInfo>	getResourcesTypeInfo()	Get all resource types information from known resource types.
static java.util.Map<java.lang.String,java.lang.Integer>	getResourceTypeIndex()	Get associate index of resource types such memory, cpu etc.
static java.util.Map<java.lang.String,ResourceInformation>	getResourceTypes()	Get the resource types to be supported by the system.
static ResourceInformation[]	getResourceTypesArray()
static Resource	getResourceTypesMaximumAllocation()	Get a Resource object with for the maximum allocation possible.
static Resource	getResourceTypesMinimumAllocation()
static java.lang.String	getUnits(java.lang.String resourceValue)
static long	getValue(java.lang.String resourceValue)
static void	initializeResourcesFromResourceInformationMap(java.util.Map<java.lang.String,ResourceInformation> resourceInformationMap)	This method is visible for testing, unit test can construct a resourceInformationMap and pass it to this method to initialize multiple resources.
static Resource	multiplyFloor(Resource resource, double multiplier)
static Resource	multiplyRound(Resource resource, double multiplier)
static java.lang.String[]	parseResourceValue(java.lang.String resourceValue)	Extract unit and actual value from resource value.
static void	reinitializeResources(java.util.List<ResourceTypeInfo> resourceTypeInfo)	Reinitialize all resource types from external source (in case of client, server will send the updated list and local resourceutils cache will be updated as per server's list of resources).
static void	resetNodeResources()
static void	resetResourceTypes()
static java.util.Map<java.lang.String,ResourceInformation>	resetResourceTypes(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UNITS

public static final java.lang.String UNITS

See Also:

Constant Field Values

TYPE

public static final java.lang.String TYPE

See Also:

Constant Field Values

TAGS

public static final java.lang.String TAGS

See Also:

Constant Field Values

MINIMUM_ALLOCATION

public static final java.lang.String MINIMUM_ALLOCATION

See Also:

Constant Field Values

MAXIMUM_ALLOCATION

public static final java.lang.String MAXIMUM_ALLOCATION

See Also:

Constant Field Values

EXTERNAL_VOLUME_RESOURCE_TAG

public static final java.lang.String EXTERNAL_VOLUME_RESOURCE_TAG

See Also:

Constant Field Values

RESOURCE_REQUEST_VALUE_PATTERN

public static final java.util.regex.Pattern RESOURCE_REQUEST_VALUE_PATTERN

YARN_IO_OPTIONAL

public static final java.lang.String YARN_IO_OPTIONAL

See Also:

Constant Field Values

Method Detail

fetchMaximumAllocationFromConfig

public static Resource fetchMaximumAllocationFromConfig(org.apache.hadoop.conf.Configuration conf)

Get maximum allocation from config, *THIS WILL NOT UPDATE INTERNAL DATA.

Parameters:

conf - config

Returns:

maximum allocation

initializeResourcesFromResourceInformationMap

@VisibleForTesting
public static void initializeResourcesFromResourceInformationMap(java.util.Map<java.lang.String,ResourceInformation> resourceInformationMap)

This method is visible for testing, unit test can construct a resourceInformationMap and pass it to this method to initialize multiple resources.

Parameters:

resourceInformationMap - constructed resource information map.

getResourceTypeIndex

public static java.util.Map<java.lang.String,java.lang.Integer> getResourceTypeIndex()

Get associate index of resource types such memory, cpu etc. This could help to access each resource types in a resource faster.

Returns:

Index map for all Resource Types.

getResourceTypes

public static java.util.Map<java.lang.String,ResourceInformation> getResourceTypes()

Get the resource types to be supported by the system.

Returns:

A map of the resource name to a ResourceInformation object which contains details such as the unit.

getResourceTypesArray

public static ResourceInformation[] getResourceTypesArray()

getNumberOfKnownResourceTypes

public static int getNumberOfKnownResourceTypes()

getNumberOfCountableResourceTypes

public static int getNumberOfCountableResourceTypes()

resetResourceTypes

@VisibleForTesting
public static void resetResourceTypes()

resetResourceTypes

@VisibleForTesting
public static java.util.Map<java.lang.String,ResourceInformation> resetResourceTypes(org.apache.hadoop.conf.Configuration conf)

getUnits

public static java.lang.String getUnits(java.lang.String resourceValue)

parseResourceValue

public static java.lang.String[] parseResourceValue(java.lang.String resourceValue)

Extract unit and actual value from resource value.

Parameters:

resourceValue - Value of the resource

Returns:

Array containing unit and value. [0]=unit, [1]=value

Throws:

java.lang.IllegalArgumentException - if units contain non alpha characters

getValue

public static long getValue(java.lang.String resourceValue)

getNodeResourceInformation

public static java.util.Map<java.lang.String,ResourceInformation> getNodeResourceInformation(org.apache.hadoop.conf.Configuration conf)

Function to get the resources for a node. This function will look at the file YarnConfiguration.NODE_RESOURCES_CONFIGURATION_FILE to determine the node resources.

Parameters:

conf - configuration file

Returns:

a map to resource name to the ResourceInformation object. The map is guaranteed to have entries for memory and vcores

resetNodeResources

@VisibleForTesting
public static void resetNodeResources()

getResourceTypesMinimumAllocation

public static Resource getResourceTypesMinimumAllocation()

getResourceTypesMaximumAllocation

public static Resource getResourceTypesMaximumAllocation()

Get a Resource object with for the maximum allocation possible.

Returns:

a Resource object with the maximum allocation for the scheduler

getDefaultUnit

public static java.lang.String getDefaultUnit(java.lang.String resourceType)

Get default unit by given resource type.

Parameters:

resourceType - resourceType

Returns:

default unit

getResourcesTypeInfo

public static java.util.List<ResourceTypeInfo> getResourcesTypeInfo()

Get all resource types information from known resource types.

Returns:

List of ResourceTypeInfo

reinitializeResources

public static void reinitializeResources(java.util.List<ResourceTypeInfo> resourceTypeInfo)

Reinitialize all resource types from external source (in case of client, server will send the updated list and local resourceutils cache will be updated as per server's list of resources).

Parameters:

resourceTypeInfo - List of resource types

getRequestedResourcesFromConfig

public static java.util.List<ResourceInformation> getRequestedResourcesFromConfig(org.apache.hadoop.conf.Configuration configuration,
                                                                                  java.lang.String prefix)

From a given configuration get all entries representing requested resources: entries that match the {prefix}{resourceName}={value}[{units}] pattern.

Parameters:

configuration - The configuration

prefix - Keys with this prefix are considered from the configuration

Returns:

The list of requested resources as described by the configuration

areMandatoryResourcesAvailable

public static void areMandatoryResourcesAvailable(Resource res)

Are mandatory resources like memory-mb, vcores available? If not, throw exceptions. On availability, ensure those values are within boundary.

Parameters:

res - resource

Throws:

java.lang.IllegalArgumentException - if mandatory resource is not available or value is not within boundary

createResourceTypesArray

public static ResourceInformation[] createResourceTypesArray(java.util.Map<java.lang.String,java.lang.Long> res)

Create an array of ResourceInformation objects corresponding to the passed in map of names to values. The array will be ordered according to the order returned by getResourceTypesArray(). The value of each resource type in the returned array will either be the value given for that resource in the res parameter or, if none is given, 0.

Parameters:

res - the map of resource type values

Returns:

an array of ResourceInformation instances

createResourceWithSameValue

@Private
@Unstable
public static Resource createResourceWithSameValue(long value)

Return a new Resource instance with all resource values initialized to value.

Parameters:

value - the value to use for all resources

Returns:

a new Resource instance

multiplyFloor

public static Resource multiplyFloor(Resource resource,
                                     double multiplier)

multiplyRound

public static Resource multiplyRound(Resource resource,
                                     double multiplier)

createResourceFromString

@Private
@Unstable
public static Resource createResourceFromString(java.lang.String resourceStr,
                                                java.util.List<ResourceTypeInfo> resourceTypeInfos)

getCustomResourcesStrings

public static java.lang.StringBuilder getCustomResourcesStrings(Resource resource)