Package org.apache.hadoop.yarn.server.resourcemanager.scheduler

Interface YarnScheduler

All Superinterfaces:

org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>

All Known Subinterfaces:

MutableConfScheduler, PreemptableResourceScheduler, ResourceScheduler

All Known Implementing Classes:

AbstractYarnScheduler, CapacityScheduler, FairScheduler, FifoScheduler

public interface YarnScheduler
extends org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>

This interface is used by the components to talk to the scheduler for allocating of resources, cleaning up resources.

Method Summary

Modifier and Type	Method	Description
void	addQueue(Queue newQueue)	Add to the scheduler a new Queue.
Allocation	allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId, java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, java.util.List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests, java.util.List<org.apache.hadoop.yarn.api.records.ContainerId> release, java.util.List<java.lang.String> blacklistAdditions, java.util.List<java.lang.String> blacklistRemovals, ContainerUpdates updateRequests)	The main API between the ApplicationMaster and the Scheduler.
boolean	checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI, org.apache.hadoop.yarn.api.records.QueueACL acl, java.lang.String queueName)	Check if the user has permission to perform the operation.
long	checkAndGetApplicationLifetime(java.lang.String queueName, long lifetime)	Verify whether a submitted application lifetime is valid as per configured Queue lifetime.
org.apache.hadoop.yarn.api.records.Priority	checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp, org.apache.hadoop.security.UserGroupInformation user, java.lang.String queuePath, org.apache.hadoop.yarn.api.records.ApplicationId applicationId)	Verify whether a submitted application priority is valid as per configured Queue
org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport	getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)	Get a resource usage report from a given app attempt ID.
java.util.List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId>	getAppsInQueue(java.lang.String queueName)	Gets the apps under a given queue
org.apache.hadoop.yarn.api.records.Resource	getClusterResource()	Get the whole resource capacity of the cluster.
org.apache.hadoop.yarn.api.records.Priority	getMaxClusterLevelAppPriority()	Get cluster max priority.
long	getMaximumApplicationLifetime(java.lang.String queueName)	Get maximum lifetime for a queue.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability()	Get maximum allocatable Resource at the cluster level.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability(java.lang.String queueName)	Get maximum allocatable Resource for the queue specified.
org.apache.hadoop.yarn.api.records.Resource	getMinimumResourceCapability()	Get minimum allocatable Resource.
SchedulerNodeReport	getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)	Get node resource usage report.
org.apache.hadoop.yarn.api.records.Resource	getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource requestedResource, org.apache.hadoop.yarn.api.records.Resource maxResourceCapability)	Normalize a resource request using scheduler level maximum resource or queue based maximum resource.
int	getNumClusterNodes()	Get the number of nodes available in the cluster.
java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest>	getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)	Get pending resource request for specified application attempt.
java.util.List<org.apache.hadoop.yarn.api.records.SchedulingRequest>	getPendingSchedulingRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)	Get pending scheduling request for specified application attempt.
java.util.Set<java.lang.String>	getPlanQueues()	Gets the list of names for queues managed by the Reservation System.
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(java.lang.String queueName, boolean includeChildQueues, boolean recursive)	Get queue information.
java.util.List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>	getQueueUserAclInfo()	Get acls for queues for current user.
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator()
RMContainer	getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)	Get the container for the given containerId.
QueueMetrics	getRootQueueMetrics()	Get the root queue for the scheduler.
SchedulerAppReport	getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)	Get the Scheduler app for a given app attempt Id.
SchedulerNode	getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)	Get SchedulerNode corresponds to nodeId.
java.util.EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>	getSchedulingResourceTypes()	Return a collection of the resource types that are considered when scheduling
java.util.List<org.apache.hadoop.yarn.api.records.Container>	getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)	Get previous attempts' live containers for work-preserving AM restart.
void	killAllAppsInQueue(java.lang.String queueName)	Terminate all applications in the specified queue.
void	moveAllApps(java.lang.String sourceQueue, java.lang.String destQueue)	Completely drain sourceQueue of applications, by moving all of them to destQueue.
java.lang.String	moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String newQueue)	Moves the given application to the given queue.
void	preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String newQueue)
void	removeQueue(java.lang.String queueName)	Remove an existing queue.
void	setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf)	Set the cluster max priority.
void	setEntitlement(java.lang.String queue, QueueEntitlement entitlement)	This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
org.apache.hadoop.yarn.api.records.Priority	updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority, org.apache.hadoop.yarn.api.records.ApplicationId applicationId, org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<java.lang.Object> future, org.apache.hadoop.security.UserGroupInformation user)	Change application priority of a submitted application at runtime

Methods inherited from interface org.apache.hadoop.yarn.event.EventHandler

handle

Method Detail

getQueueInfo

@Public
@Stable
org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(java.lang.String queueName,
                                                          boolean includeChildQueues,
                                                          boolean recursive)
                                                   throws java.io.IOException

Get queue information.

Parameters:

queueName - queue name

includeChildQueues - include child queues?

recursive - get children queues?

Returns:

queue information

Throws:

java.io.IOException - an I/O exception has occurred.

getQueueUserAclInfo

@Public
@Stable
java.util.List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo()

Get acls for queues for current user.

Returns:

acls for queues for current user

getClusterResource

@LimitedPrivate("yarn")
@Unstable
org.apache.hadoop.yarn.api.records.Resource getClusterResource()

Get the whole resource capacity of the cluster.

Returns:

the whole resource capacity of the cluster.

getMinimumResourceCapability

@Public
@Stable
org.apache.hadoop.yarn.api.records.Resource getMinimumResourceCapability()

Get minimum allocatable Resource.

Returns:

minimum allocatable resource

getMaximumResourceCapability

@Public
@Stable
org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability()

Get maximum allocatable Resource at the cluster level.

Returns:

maximum allocatable resource

getMaximumResourceCapability

@Public
@Stable
org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(java.lang.String queueName)

Get maximum allocatable Resource for the queue specified.

Parameters:

queueName - queue name

Returns:

maximum allocatable resource

getResourceCalculator

@LimitedPrivate("yarn")
@Evolving
org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

getNumClusterNodes

@Public
@Stable
int getNumClusterNodes()

Get the number of nodes available in the cluster.

Returns:

the number of available nodes.

allocate

@Public
@Stable
Allocation allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId,
                    java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask,
                    java.util.List<org.apache.hadoop.yarn.api.records.SchedulingRequest> schedulingRequests,
                    java.util.List<org.apache.hadoop.yarn.api.records.ContainerId> release,
                    java.util.List<java.lang.String> blacklistAdditions,
                    java.util.List<java.lang.String> blacklistRemovals,
                    ContainerUpdates updateRequests)

The main API between the ApplicationMaster and the Scheduler. The ApplicationMaster may request/update container resources, number of containers, node/rack preference for allocations etc. to the Scheduler.

Parameters:

appAttemptId - the id of the application attempt.

ask - the request made by an application to obtain various allocations like host/rack, resource, number of containers, relaxLocality etc., see ResourceRequest.

schedulingRequests - similar to ask, but with added ability to specify allocation tags etc., see SchedulingRequest.

release - the list of containers to be released.

blacklistAdditions - places (node/rack) to be added to the blacklist.

blacklistRemovals - places (node/rack) to be removed from the blacklist.

updateRequests - container promotion/demotion updates.

Returns:

the Allocation for the application.

getNodeReport

@LimitedPrivate("yarn")
@Stable
SchedulerNodeReport getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Get node resource usage report.

Parameters:

nodeId - nodeId.

Returns:

the SchedulerNodeReport for the node or null if nodeId does not point to a defined node.

getSchedulerAppInfo

@LimitedPrivate("yarn")
@Stable
SchedulerAppReport getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Get the Scheduler app for a given app attempt Id.

Parameters:

appAttemptId - the id of the application attempt

Returns:

SchedulerApp for this given attempt.

getAppResourceUsageReport

@LimitedPrivate("yarn")
@Evolving
org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Get a resource usage report from a given app attempt ID.

Parameters:

appAttemptId - the id of the application attempt

Returns:

resource usage report for this given attempt

getRootQueueMetrics

@LimitedPrivate("yarn")
@Evolving
QueueMetrics getRootQueueMetrics()

Get the root queue for the scheduler.

Returns:

the root queue for the scheduler.

checkAccess

boolean checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI,
                    org.apache.hadoop.yarn.api.records.QueueACL acl,
                    java.lang.String queueName)

Check if the user has permission to perform the operation. If the user has QueueACL.ADMINISTER_QUEUE permission, this user can view/modify the applications in this queue.

Parameters:

callerUGI - caller UserGroupInformation.

acl - queue ACL.

queueName - queue Name.

Returns:

true if the user has the permission, false otherwise

getAppsInQueue

@LimitedPrivate("yarn")
@Stable
java.util.List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> getAppsInQueue(java.lang.String queueName)

Gets the apps under a given queue

Parameters:

queueName - the name of the queue.

Returns:

a collection of app attempt ids in the given queue.

getRMContainer

@LimitedPrivate("yarn")
@Unstable
RMContainer getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Get the container for the given containerId.

Parameters:

containerId - the given containerId.

Returns:

the container for the given containerId.

moveApplication

@LimitedPrivate("yarn")
@Evolving
java.lang.String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                 java.lang.String newQueue)
                          throws org.apache.hadoop.yarn.exceptions.YarnException

Moves the given application to the given queue.

Parameters:

appId - application Id

newQueue - the given queue.

Returns:

the name of the queue the application was placed into

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

preValidateMoveApplication

@LimitedPrivate("yarn")
@Evolving
void preValidateMoveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                java.lang.String newQueue)
                         throws org.apache.hadoop.yarn.exceptions.YarnException

Parameters:

appId - Application ID

newQueue - Target QueueName

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the pre-validation for move cannot be carried out

moveAllApps

void moveAllApps(java.lang.String sourceQueue,
                 java.lang.String destQueue)
          throws org.apache.hadoop.yarn.exceptions.YarnException

Completely drain sourceQueue of applications, by moving all of them to destQueue.

Parameters:

sourceQueue - sourceQueue.

destQueue - destQueue.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

killAllAppsInQueue

void killAllAppsInQueue(java.lang.String queueName)
                 throws org.apache.hadoop.yarn.exceptions.YarnException

Terminate all applications in the specified queue.

Parameters:

queueName - the name of queue to be drained

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

removeQueue

void removeQueue(java.lang.String queueName)
          throws org.apache.hadoop.yarn.exceptions.YarnException

Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).

Parameters:

queueName - name of the queue to remove

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

addQueue

void addQueue(Queue newQueue)
       throws org.apache.hadoop.yarn.exceptions.YarnException,
              java.io.IOException

Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).

Parameters:

newQueue - the queue being added.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

java.io.IOException - when io exception occur.

setEntitlement

void setEntitlement(java.lang.String queue,
                    QueueEntitlement entitlement)
             throws org.apache.hadoop.yarn.exceptions.YarnException

This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.

Parameters:

queue - the queue for which we change entitlement

entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

getPlanQueues

java.util.Set<java.lang.String> getPlanQueues()
                                       throws org.apache.hadoop.yarn.exceptions.YarnException

Gets the list of names for queues managed by the Reservation System.

Returns:

the list of queues which support reservations

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

getSchedulingResourceTypes

java.util.EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()

Return a collection of the resource types that are considered when scheduling

Returns:

an EnumSet containing the resource types

checkAndGetApplicationPriority

org.apache.hadoop.yarn.api.records.Priority checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityRequestedByApp,
                                                                           org.apache.hadoop.security.UserGroupInformation user,
                                                                           java.lang.String queuePath,
                                                                           org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                    throws org.apache.hadoop.yarn.exceptions.YarnException

Verify whether a submitted application priority is valid as per configured Queue

Parameters:

priorityRequestedByApp - Submitted Application priority.

user - User who submitted the Application

queuePath - Name of the Queue

applicationId - Application ID

Returns:

Updated Priority from scheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

updateApplicationPriority

org.apache.hadoop.yarn.api.records.Priority updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority,
                                                                      org.apache.hadoop.yarn.api.records.ApplicationId applicationId,
                                                                      org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFuture<java.lang.Object> future,
                                                                      org.apache.hadoop.security.UserGroupInformation user)
                                                               throws org.apache.hadoop.yarn.exceptions.YarnException

Change application priority of a submitted application at runtime

Parameters:

newPriority - Submitted Application priority.

applicationId - Application ID

future - Sets any type of exception happened from StateStore

user - who submitted the application

Returns:

updated priority

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

getTransferredContainers

java.util.List<org.apache.hadoop.yarn.api.records.Container> getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Get previous attempts' live containers for work-preserving AM restart.

Parameters:

appAttemptId - the id of the application attempt

Returns:

list of live containers for the given attempt

setClusterMaxPriority

void setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf)
                    throws org.apache.hadoop.yarn.exceptions.YarnException

Set the cluster max priority.

Parameters:

conf - Configuration.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - when yarn exception occur.

getPendingResourceRequestsForAttempt

java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)

Get pending resource request for specified application attempt.

Parameters:

attemptId - the id of the application attempt

Returns:

pending resource requests.

getPendingSchedulingRequestsForAttempt

java.util.List<org.apache.hadoop.yarn.api.records.SchedulingRequest> getPendingSchedulingRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)

Get pending scheduling request for specified application attempt.

Parameters:

attemptId - the id of the application attempt

Returns:

pending scheduling requests

getMaxClusterLevelAppPriority

org.apache.hadoop.yarn.api.records.Priority getMaxClusterLevelAppPriority()

Get cluster max priority.

Returns:

maximum priority of cluster

getSchedulerNode

SchedulerNode getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Get SchedulerNode corresponds to nodeId.

Parameters:

nodeId - the node id of RMNode

Returns:

SchedulerNode corresponds to nodeId

getNormalizedResource

org.apache.hadoop.yarn.api.records.Resource getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource requestedResource,
                                                                  org.apache.hadoop.yarn.api.records.Resource maxResourceCapability)

Normalize a resource request using scheduler level maximum resource or queue based maximum resource.

Parameters:

requestedResource - the resource to be normalized

maxResourceCapability - Maximum container allocation value, if null or empty scheduler level maximum container allocation value will be used

Returns:

the normalized resource

checkAndGetApplicationLifetime

@Public
@Evolving
long checkAndGetApplicationLifetime(java.lang.String queueName,
                                    long lifetime)

Verify whether a submitted application lifetime is valid as per configured Queue lifetime.

Parameters:

queueName - Name of the Queue

lifetime - configured application lifetime

Returns:

valid lifetime as per queue

getMaximumApplicationLifetime

@Public
@Evolving
long getMaximumApplicationLifetime(java.lang.String queueName)

Get maximum lifetime for a queue.

Parameters:

queueName - to get lifetime

Returns:

maximum lifetime in seconds