Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class AbstractParentQueue

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue

All Implemented Interfaces:

CSQueue, Queue, SchedulerQueue<CSQueue>

Direct Known Subclasses:

AbstractManagedParentQueue, ParentQueue

public abstract class AbstractParentQueue
extends AbstractCSQueue

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	AbstractParentQueue.QueueCapacityType

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

AbstractCSQueue.CapacityConfigType

Field Summary

Fields

Modifier and Type	Field	Description
protected java.util.List<CSQueue>	childQueues

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

activitiesManager, authorizer, capacityConfigType, configuredCapacityVectors, configuredMaxCapacityVectors, queueAllocationSettings, queueContext, queueEntity, queueNodeLabelsSettings, queuePath, readLock, writeLock

Constructor Summary

Constructors

Constructor	Description
AbstractParentQueue(CapacitySchedulerQueueContext queueContext, java.lang.String queueName, CSQueue parent, CSQueue old)
AbstractParentQueue(CapacitySchedulerQueueContext queueContext, java.lang.String queueName, CSQueue parent, CSQueue old, boolean isDynamic)

Method Summary

Modifier and Type	Method	Description
void	apply(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)
CSAssignment	assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, CandidateNodeSet<FiCaSchedulerNode> candidates, ResourceLimits resourceLimits, SchedulingMode schedulingMode)	Assign containers to applications in the queue or it's children (if any).
void	attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer)	Attach a container to this queue
void	collectSchedulerApplications(java.util.Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)	Adds all applications in the queue and its subqueues to the given collection.
void	completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, FiCaSchedulerNode node, RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event, CSQueue completedChildQueue, boolean sortQueues)	A container assigned to the queue has completed.
CSQueue	createNewQueue(java.lang.String childQueuePath, boolean isLeaf)
void	detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer)	Detach a container from this queue
void	finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId application, java.lang.String user)	An application submitted to this queue has finished.
void	finishApplicationAttempt(FiCaSchedulerApp application, java.lang.String queue)	An application attempt submitted to this queue has finished.
ActiveUsersManager	getAbstractUsersManager()	Get the AbstractUsersManager for the queue.
AutoCreatedQueueTemplate	getAutoCreatedQueueTemplate()
AbstractParentQueue.QueueCapacityType	getCapacityConfigurationTypeForQueues(java.util.Collection<CSQueue> queues)
java.util.List<CSQueue>	getChildQueues()	Get child queues
java.util.List<CSQueue>	getChildQueuesByTryLock()	Get child queues By tryLock.
int	getNumApplications()	Get the number of applications in the queue.
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(boolean includeChildQueues, boolean recursive)	Get queue information
QueueOrderingPolicy	getQueueOrderingPolicy()
java.util.List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>	getQueueUserAclInfo(org.apache.hadoop.security.UserGroupInformation user)	Get queue ACLs for given user.
ResourceLimits	getResourceLimitsOfChild(CSQueue child, org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits parentLimits, java.lang.String nodePartition, boolean netLimit)
boolean	hasChildQueues()
boolean	isEligibleForAutoDeletion()	Checks whether this queue is a dynamic queue and could be deleted.
boolean	isEligibleForAutoQueueCreation()	Check whether this queue supports adding additional child queues dynamically.
boolean	isEligibleForLegacyAutoQueueCreation()	Check whether this queue supports legacy(v1) dynamic child queue creation.
void	recoverContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, SchedulerApplicationAttempt attempt, RMContainer rmContainer)	Recover the state of the queue for a given container.
void	refreshAfterResourceCalculation(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)
void	reinitialize(CSQueue newlyParsedQueue, org.apache.hadoop.yarn.api.records.Resource clusterResource)	Reinitialize the queue.
void	removeChildQueue(CSQueue queue)
protected void	setDynamicQueueACLProperties()
protected void	setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource)	Initialize queue properties that are based on configuration.
void	stopQueue()	Stop the queue.
void	submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, java.lang.String user, java.lang.String queue)	Submit a new application to the queue.
void	submitApplicationAttempt(FiCaSchedulerApp application, java.lang.String userName)	Submit an application attempt to the queue.
void	submitApplicationAttempt(FiCaSchedulerApp application, java.lang.String userName, boolean isMoveApp)	Submit an application attempt to the queue.
java.lang.String	toString()
void	updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)	Update the cluster resource for queues as we add/remove nodes
void	updateClusterResourceLegacyMode(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)
void	validateSubmitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, java.lang.String userName, java.lang.String queue)	Validate submitApplication api so that moveApplication do a pre-check.

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

accept, activateQueue, appFinished, assignContainers, checkConfigTypeIsAbsoluteResource, decPendingResource, decReservedResource, decUsedResource, getAbsoluteCapacity, getAbsoluteMaximumCapacity, getAbsoluteUsedCapacity, getAccessibleNodeLabels, getACLs, getCapacity, getCapacityConfigType, getCapacityOrWeightString, getConfiguredCapacityVector, getConfiguredMaxCapacityVector, getConfiguredNodeLabels, getDefaultApplicationLifetime, getDefaultApplicationPriority, getDefaultAppLifetimeWasSpecifiedInConfig, getDefaultNodeLabelExpression, getEffectiveCapacity, getEffectiveCapacityDown, getEffectiveMaxCapacity, getEffectiveMaxCapacityDown, getIntraQueuePreemptionDisabled, getIntraQueuePreemptionDisabledInHierarchy, getKillableContainers, getLabel, getMaximumAbsoluteResource, getMaximumAllocation, getMaximumApplicationLifetime, getMaximumCapacity, getMaxParallelApps, getMetrics, getMinimumAbsoluteResource, getMinimumAllocation, getMultiNodeSortingPolicyName, getNodeLabelsForQueue, getNumContainers, getParent, getPreemptionDisabled, getPriority, getPrivilegedEntity, getQueueCapacities, getQueueContext, getQueueInfo, getQueueName, getQueuePath, getQueuePathObject, getQueueResourceQuotas, getQueueResourceUsage, getQueueShortName, getReadLock, getState, getTotalKillableResource, getUsedCapacity, getUsedResources, getUserWeights, getWriteLock, hasAccess, incPendingResource, incReservedResource, incUsedResource, isDynamicQueue, isInactiveDynamicQueue, isReservationsContinueLooking, overrideCapacityVectorsForSpecialQueues, parseAndSetDynamicTemplates, recoverDrainingState, releaseResource, setConfiguredMaxCapacityVector, setConfiguredMinCapacityVector, setDynamicQueue, setMaxParallelApps, setMultiNodeSortingPolicyName, setParent, setupConfigurableCapacities, updateAbsoluteCapacities, updateCapacityConfigType, updateConfigurableResourceLimits, updateQueueState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

childQueues

protected final java.util.List<CSQueue> childQueues

Constructor Detail

AbstractParentQueue

public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
                           java.lang.String queueName,
                           CSQueue parent,
                           CSQueue old)
                    throws java.io.IOException

Throws:

java.io.IOException

AbstractParentQueue

public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
                           java.lang.String queueName,
                           CSQueue parent,
                           CSQueue old,
                           boolean isDynamic)
                    throws java.io.IOException

Throws:

java.io.IOException

Method Detail

setupQueueConfigs

protected void setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource)
                          throws java.io.IOException

Description copied from class: AbstractCSQueue

Initialize queue properties that are based on configuration.

Overrides:

setupQueueConfigs in class AbstractCSQueue

Parameters:

clusterResource - overall resource of the cluster

Throws:

java.io.IOException - if configuration is set in a way that is inconsistent

setDynamicQueueACLProperties

protected void setDynamicQueueACLProperties()

Overrides:

setDynamicQueueACLProperties in class AbstractCSQueue

getCapacityConfigurationTypeForQueues

public AbstractParentQueue.QueueCapacityType getCapacityConfigurationTypeForQueues(java.util.Collection<CSQueue> queues)
                                                                            throws java.io.IOException

Throws:

java.io.IOException

getQueueInfo

public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(boolean includeChildQueues,
                                                                 boolean recursive)

Description copied from interface: Queue

Get queue information

Parameters:

includeChildQueues - include child queues?

recursive - recursively get child queue information?

Returns:

queue information

getQueueUserAclInfo

public java.util.List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo(org.apache.hadoop.security.UserGroupInformation user)

Description copied from interface: Queue

Get queue ACLs for given user.

Parameters:

user - username

Returns:

queue ACLs for user

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

createNewQueue

public CSQueue createNewQueue(java.lang.String childQueuePath,
                              boolean isLeaf)
                       throws SchedulerDynamicEditException

Throws:

SchedulerDynamicEditException

removeChildQueue

public void removeChildQueue(CSQueue queue)
                      throws SchedulerDynamicEditException

Throws:

SchedulerDynamicEditException

isEligibleForAutoQueueCreation

public boolean isEligibleForAutoQueueCreation()

Check whether this queue supports adding additional child queues dynamically.

Returns:

true, if queue is eligible to create additional queues dynamically, false otherwise

isEligibleForLegacyAutoQueueCreation

public boolean isEligibleForLegacyAutoQueueCreation()

Check whether this queue supports legacy(v1) dynamic child queue creation.

Returns:

true if queue is eligible to create child queues dynamically using the legacy system, false otherwise

reinitialize

public void reinitialize(CSQueue newlyParsedQueue,
                         org.apache.hadoop.yarn.api.records.Resource clusterResource)
                  throws java.io.IOException

Description copied from interface: CSQueue

Reinitialize the queue.

Parameters:

newlyParsedQueue - new queue to re-initalize from

clusterResource - resources in the cluster

Throws:

java.io.IOException - an I/O exception has occurred.

submitApplication

public void submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId,
                              java.lang.String user,
                              java.lang.String queue)
                       throws org.apache.hadoop.security.AccessControlException

Description copied from interface: CSQueue

Submit a new application to the queue.

Parameters:

applicationId - the applicationId of the application being submitted

user - user who submitted the application

queue - queue to which the application is submitted

Throws:

org.apache.hadoop.security.AccessControlException - if any acl violation is there.

validateSubmitApplication

public void validateSubmitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId,
                                      java.lang.String userName,
                                      java.lang.String queue)
                               throws org.apache.hadoop.security.AccessControlException

Description copied from interface: CSQueue

Validate submitApplication api so that moveApplication do a pre-check.

Specified by:

validateSubmitApplication in interface CSQueue

Overrides:

validateSubmitApplication in class AbstractCSQueue

Parameters:

applicationId - Application ID

userName - User Name

queue - Queue Name

Throws:

org.apache.hadoop.security.AccessControlException - if any acl violation is there.

submitApplicationAttempt

public void submitApplicationAttempt(FiCaSchedulerApp application,
                                     java.lang.String userName)

Description copied from interface: CSQueue

Submit an application attempt to the queue.

Parameters:

application - application whose attempt is being submitted.

userName - userName who submitted the application.

submitApplicationAttempt

public void submitApplicationAttempt(FiCaSchedulerApp application,
                                     java.lang.String userName,
                                     boolean isMoveApp)

Description copied from interface: CSQueue

Submit an application attempt to the queue.

Parameters:

application - application whose attempt is being submitted

userName - user who submitted the application attempt

isMoveApp - is application being moved across the queue

finishApplicationAttempt

public void finishApplicationAttempt(FiCaSchedulerApp application,
                                     java.lang.String queue)

Description copied from interface: CSQueue

An application attempt submitted to this queue has finished.

Parameters:

application - application attempt.

queue - queue.

finishApplication

public void finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId application,
                              java.lang.String user)

Description copied from interface: CSQueue

An application submitted to this queue has finished.

Parameters:

application - applicationId.

user - user who submitted the application

assignContainers

public CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                     CandidateNodeSet<FiCaSchedulerNode> candidates,
                                     ResourceLimits resourceLimits,
                                     SchedulingMode schedulingMode)

Description copied from interface: CSQueue

Assign containers to applications in the queue or it's children (if any).

Parameters:

clusterResource - the resource of the cluster.

candidates - CandidateNodeSet the nodes that are considered for the current placement.

resourceLimits - how much overall resource of this queue can use.

schedulingMode - Type of exclusive check when assign container on a NodeManager, see SchedulingMode.

Returns:

the assignment

getResourceLimitsOfChild

public ResourceLimits getResourceLimitsOfChild(CSQueue child,
                                               org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                               ResourceLimits parentLimits,
                                               java.lang.String nodePartition,
                                               boolean netLimit)

completedContainer

public void completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                               FiCaSchedulerApp application,
                               FiCaSchedulerNode node,
                               RMContainer rmContainer,
                               org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                               RMContainerEventType event,
                               CSQueue completedChildQueue,
                               boolean sortQueues)

Description copied from interface: CSQueue

A container assigned to the queue has completed.

Parameters:

clusterResource - the resource of the cluster

application - application to which the container was assigned

node - node on which the container completed

rmContainer - completed container, null if it was just a reservation

containerStatus - ContainerStatus for the completed container

event - event to be sent to the container

completedChildQueue - CSQueue to reinsert in childQueues

sortQueues - indicates whether it should re-sort the queues

refreshAfterResourceCalculation

public void refreshAfterResourceCalculation(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                            ResourceLimits resourceLimits)

updateClusterResource

public void updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                  ResourceLimits resourceLimits)

Description copied from interface: CSQueue

Update the cluster resource for queues as we add/remove nodes

Parameters:

clusterResource - the current cluster resource

resourceLimits - the current ResourceLimits

updateClusterResourceLegacyMode

public void updateClusterResourceLegacyMode(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                            ResourceLimits resourceLimits)

hasChildQueues

public boolean hasChildQueues()

getChildQueues

public java.util.List<CSQueue> getChildQueues()

Description copied from interface: CSQueue

Get child queues

Returns:

child queues

getChildQueuesByTryLock

public java.util.List<CSQueue> getChildQueuesByTryLock()

Description copied from interface: CSQueue

Get child queues By tryLock.

Returns:

child queues

recoverContainer

public void recoverContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                             SchedulerApplicationAttempt attempt,
                             RMContainer rmContainer)

Description copied from interface: Queue

Recover the state of the queue for a given container.

Parameters:

clusterResource - the resource of the cluster

attempt - the application for which the container was allocated

rmContainer - the container that was recovered.

getAbstractUsersManager

public ActiveUsersManager getAbstractUsersManager()

Description copied from interface: CSQueue

Get the AbstractUsersManager for the queue.

Returns:

the AbstractUsersManager for the queue

collectSchedulerApplications

public void collectSchedulerApplications(java.util.Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)

Description copied from interface: CSQueue

Adds all applications in the queue and its subqueues to the given collection.

Parameters:

apps - the collection to add the applications to

attachContainer

public void attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                            FiCaSchedulerApp application,
                            RMContainer rmContainer)

Description copied from interface: CSQueue

Attach a container to this queue

Parameters:

clusterResource - the current cluster resource

application - application to which the container was assigned

rmContainer - the container to attach

detachContainer

public void detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                            FiCaSchedulerApp application,
                            RMContainer rmContainer)

Description copied from interface: CSQueue

Detach a container from this queue

Parameters:

clusterResource - the current cluster resource

application - application to which the container was assigned

rmContainer - the container to detach

getNumApplications

public int getNumApplications()

Description copied from interface: CSQueue

Get the number of applications in the queue.

Returns:

number of applications

apply

public void apply(org.apache.hadoop.yarn.api.records.Resource cluster,
                  ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)

stopQueue

public void stopQueue()

Description copied from interface: SchedulerQueue

Stop the queue.

getQueueOrderingPolicy

public QueueOrderingPolicy getQueueOrderingPolicy()

isEligibleForAutoDeletion

public boolean isEligibleForAutoDeletion()

Description copied from class: AbstractCSQueue

Checks whether this queue is a dynamic queue and could be deleted.

Overrides:

isEligibleForAutoDeletion in class AbstractCSQueue

Returns:

true if the dynamic queue could be deleted, false otherwise

getAutoCreatedQueueTemplate

public AutoCreatedQueueTemplate getAutoCreatedQueueTemplate()