Package org.apache.hadoop.yarn.server.resourcemanager

Class RMServerUtils

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils

public class RMServerUtils
extends java.lang.Object

Utility methods to aid serving RM data through the REST and RPC APIs

Field Summary

Fields

Modifier and Type	Field	Description
static org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport	DUMMY_APPLICATION_RESOURCE_USAGE_REPORT	Statically defined dummy ApplicationResourceUsageREport.
protected static org.apache.hadoop.yarn.factories.RecordFactory	RECORD_FACTORY
static java.lang.String	RESOURCE_OUTSIDE_ALLOWED_RANGE
static java.lang.String	UPDATE_OUTSTANDING_ERROR

Constructor Summary

Constructors

Constructor	Description
RMServerUtils()

Method Summary

Modifier and Type	Method	Description
static void	checkSchedContainerChangeRequest(SchedContainerChangeRequest request, boolean increase)	Validate increase/decrease request.
static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState	convertRmAppAttemptStateToYarnApplicationAttemptState(RMAppAttemptState currentState, RMAppAttemptState previousState)
static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState	createApplicationAttemptState(RMAppAttemptState rmAppAttemptState)
static org.apache.hadoop.yarn.api.records.YarnApplicationState	createApplicationState(RMAppState rmAppState)
static int	getApplicableNodeCountForAM(RMContext rmContext, org.apache.hadoop.conf.Configuration conf, java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> amReqs)	Get applicable Node count for AM.
static java.lang.Long	getOrDefault(java.util.Map<java.lang.String,java.lang.Long> map, java.lang.String key, java.lang.Long defaultValue)
static void	normalizeAndValidateRequests(java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, java.lang.String queueName, YarnScheduler scheduler, RMContext rmContext, boolean nodeLabelsEnabled)	Utility method to validate a list resource requests, by ensuring that the requested memory/vcore is non-negative and not greater than max.
static void	processRMProxyUsersConf(org.apache.hadoop.conf.Configuration conf)	Find all configs whose name starts with YarnConfiguration.RM_PROXY_USER_PREFIX, and add a record for each one by replacing the prefix with ProxyUsers.CONF_HADOOP_PROXYUSER.
static java.util.List<RMNode>	queryRMNodes(RMContext context, java.util.EnumSet<org.apache.hadoop.yarn.api.records.NodeState> acceptedStates)
static ContainerUpdates	validateAndSplitUpdateResourceRequests(RMContext rmContext, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, java.util.List<org.apache.hadoop.yarn.api.records.UpdateContainerError> updateErrors)	Check if we have: - Request for same containerId and different target resource.
static void	validateApplicationTimeouts(java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.Long> timeouts)
static void	validateBlacklistRequest(org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest blacklistRequest)
static void	validateContainerReleaseRequest(java.util.List<org.apache.hadoop.yarn.api.records.ContainerId> containerReleaseList, org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)	It will validate to make sure all the containers belong to correct application attempt id.
static java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.Long>	validateISO8601AndConvertToLocalTimeEpoch(java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.String> timeoutsInISO8601)	Validate ISO8601 format with epoch time.
static org.apache.hadoop.security.UserGroupInformation	verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer, java.lang.String method, java.lang.String module, org.slf4j.Logger LOG)	Utility method to verify if the current user has access based on the passed AccessControlList
static org.apache.hadoop.security.UserGroupInformation	verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer, java.lang.String method, org.slf4j.Logger LOG)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UPDATE_OUTSTANDING_ERROR

public static final java.lang.String UPDATE_OUTSTANDING_ERROR

See Also:

Constant Field Values

RESOURCE_OUTSIDE_ALLOWED_RANGE

public static final java.lang.String RESOURCE_OUTSIDE_ALLOWED_RANGE

See Also:

Constant Field Values

RECORD_FACTORY

protected static final org.apache.hadoop.yarn.factories.RecordFactory RECORD_FACTORY

DUMMY_APPLICATION_RESOURCE_USAGE_REPORT

public static final org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport DUMMY_APPLICATION_RESOURCE_USAGE_REPORT

Statically defined dummy ApplicationResourceUsageREport. Used as a return value when a valid report cannot be found.

Constructor Detail

RMServerUtils

public RMServerUtils()

Method Detail

queryRMNodes

public static java.util.List<RMNode> queryRMNodes(RMContext context,
                                                  java.util.EnumSet<org.apache.hadoop.yarn.api.records.NodeState> acceptedStates)

validateAndSplitUpdateResourceRequests

public static ContainerUpdates validateAndSplitUpdateResourceRequests(RMContext rmContext,
                                                                      org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request,
                                                                      org.apache.hadoop.yarn.api.records.Resource maximumAllocation,
                                                                      java.util.List<org.apache.hadoop.yarn.api.records.UpdateContainerError> updateErrors)

Check if we have: - Request for same containerId and different target resource. - If targetResources violates maximum/minimumAllocation.

Parameters:

rmContext - RM context.

request - Allocate Request.

maximumAllocation - Maximum Allocation.

updateErrors - Container update errors.

Returns:

ContainerUpdateRequests.

normalizeAndValidateRequests

public static void normalizeAndValidateRequests(java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask,
                                                org.apache.hadoop.yarn.api.records.Resource maximumAllocation,
                                                java.lang.String queueName,
                                                YarnScheduler scheduler,
                                                RMContext rmContext,
                                                boolean nodeLabelsEnabled)
                                         throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

Utility method to validate a list resource requests, by ensuring that the requested memory/vcore is non-negative and not greater than max.

Parameters:

ask - resource request.

maximumAllocation - Maximum Allocation.

queueName - queue name.

scheduler - YarnScheduler.

rmContext - RMContext.

nodeLabelsEnabled - the node labels feature enabled.

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException - when there is invalid request.

checkSchedContainerChangeRequest

public static void checkSchedContainerChangeRequest(SchedContainerChangeRequest request,
                                                    boolean increase)
                                             throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

Validate increase/decrease request.

 - Throw exception when any other error happens
 

Parameters:

request - SchedContainerChangeRequest.

increase - true, add container; false, decrease container.

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException - when there is invalid request.

validateBlacklistRequest

public static void validateBlacklistRequest(org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest blacklistRequest)
                                     throws org.apache.hadoop.yarn.exceptions.InvalidResourceBlacklistRequestException

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceBlacklistRequestException

validateContainerReleaseRequest

public static void validateContainerReleaseRequest(java.util.List<org.apache.hadoop.yarn.api.records.ContainerId> containerReleaseList,
                                                   org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
                                            throws org.apache.hadoop.yarn.exceptions.InvalidContainerReleaseException

It will validate to make sure all the containers belong to correct application attempt id. If not then it will throw InvalidContainerReleaseException

Parameters:

containerReleaseList - containers to be released as requested by application master.

appAttemptId - Application attempt Id

Throws:

org.apache.hadoop.yarn.exceptions.InvalidContainerReleaseException - an Application Master tries to release containers not belonging to it using.

verifyAdminAccess

public static org.apache.hadoop.security.UserGroupInformation verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer,
                                                                                java.lang.String method,
                                                                                org.slf4j.Logger LOG)
                                                                         throws java.io.IOException

Throws:

java.io.IOException

verifyAdminAccess

public static org.apache.hadoop.security.UserGroupInformation verifyAdminAccess(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer,
                                                                                java.lang.String method,
                                                                                java.lang.String module,
                                                                                org.slf4j.Logger LOG)
                                                                         throws java.io.IOException

Utility method to verify if the current user has access based on the passed AccessControlList

Parameters:

authorizer - the AccessControlList to check against

method - the method name to be logged

module - like AdminService or NodeLabelManager

LOG - the logger to use

Returns:

UserGroupInformation of the current user

Throws:

java.io.IOException - an I/O exception has occurred.

createApplicationState

public static org.apache.hadoop.yarn.api.records.YarnApplicationState createApplicationState(RMAppState rmAppState)

convertRmAppAttemptStateToYarnApplicationAttemptState

public static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState convertRmAppAttemptStateToYarnApplicationAttemptState(RMAppAttemptState currentState,
                                                                                                                                   RMAppAttemptState previousState)

createApplicationAttemptState

public static org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState createApplicationAttemptState(RMAppAttemptState rmAppAttemptState)

processRMProxyUsersConf

public static void processRMProxyUsersConf(org.apache.hadoop.conf.Configuration conf)

Find all configs whose name starts with YarnConfiguration.RM_PROXY_USER_PREFIX, and add a record for each one by replacing the prefix with ProxyUsers.CONF_HADOOP_PROXYUSER.

Parameters:

conf - Configuration.

validateApplicationTimeouts

public static void validateApplicationTimeouts(java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.Long> timeouts)
                                        throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

validateISO8601AndConvertToLocalTimeEpoch

public static java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.Long> validateISO8601AndConvertToLocalTimeEpoch(java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationTimeoutType,java.lang.String> timeoutsInISO8601)
                                                                                                                                               throws org.apache.hadoop.yarn.exceptions.YarnException

Validate ISO8601 format with epoch time.

Parameters:

timeoutsInISO8601 - format

Returns:

expire time in local epoch

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if given application timeout value is lesser than current time.

getApplicableNodeCountForAM

public static int getApplicableNodeCountForAM(RMContext rmContext,
                                              org.apache.hadoop.conf.Configuration conf,
                                              java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> amReqs)

Get applicable Node count for AM.

Parameters:

rmContext - context

conf - configuration

amReqs - am resource requests

Returns:

applicable node count

getOrDefault

public static java.lang.Long getOrDefault(java.util.Map<java.lang.String,java.lang.Long> map,
                                          java.lang.String key,
                                          java.lang.Long defaultValue)