org.apache.hadoop.yarn.server.resourcemanager.scheduler

Class AbstractYarnScheduler<T extends SchedulerApplicationAttempt,N extends SchedulerNode>

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler<T,N>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>, Recoverable, ResourceScheduler, YarnScheduler

Direct Known Subclasses:

CapacityScheduler, FairScheduler, FifoScheduler

public abstract class AbstractYarnScheduler<T extends SchedulerApplicationAttempt,N extends SchedulerNode>
extends org.apache.hadoop.service.AbstractService
implements ResourceScheduler

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>>	applications
protected org.apache.hadoop.yarn.api.records.Resource	clusterResource
protected static Allocation	EMPTY_ALLOCATION
protected static List<org.apache.hadoop.yarn.api.records.Container>	EMPTY_CONTAINER_LIST
protected org.apache.hadoop.yarn.api.records.Resource	minimumAllocation
protected int	nmExpireInterval
protected Map<org.apache.hadoop.yarn.api.records.NodeId,N>	nodes
protected RMContext	rmContext

Constructor Summary

Constructors

Constructor and Description
AbstractYarnScheduler(String name) Construct the service.

Method Summary

Methods

Modifier and Type	Method and Description
void	addQueue(Queue newQueue) Add to the scheduler a new Queue.
protected abstract void	completedContainer(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
protected void	containerLaunchedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId, SchedulerNode node)
protected void	createReleaseCache()
T	getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)
org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport	getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId) Get a resource usage report from a given app attempt ID.
org.apache.hadoop.yarn.api.records.Resource	getClusterResource() Get the whole resource capacity of the cluster.
T	getCurrentAttemptForContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability() Get maximum allocatable Resource at the cluster level.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability(String queueName) Get maximum allocatable Resource for the queue specified.
org.apache.hadoop.yarn.api.records.Resource	getMinimumResourceCapability() Get minimum allocatable Resource.
SchedulerNodeReport	getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId) Get node resource usage report.
List<org.apache.hadoop.yarn.api.records.ResourceRequest>	getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
Set<String>	getPlanQueues() Gets the list of names for queues managed by the Reservation System
RMContainer	getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Get the container for the given containerId.
SchedulerAppReport	getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId) Get the Scheduler app for a given app attempt Id.
Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>>	getSchedulerApplications()
SchedulerNode	getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>	getSchedulingResourceTypes() Return a collection of the resource types that are considered when scheduling
List<org.apache.hadoop.yarn.api.records.Container>	getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId currentAttempt)
protected void	initMaximumResourceCapability(org.apache.hadoop.yarn.api.records.Resource maximumAllocation)
void	killAllAppsInQueue(String queueName) Terminate all applications in the specified queue.
void	moveAllApps(String sourceQueue, String destQueue) Completely drain sourceQueue of applications, by moving all of them to destQueue.
String	moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue) Moves the given application to the given queue
void	recoverContainersOnNode(List<org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus> containerReports, RMNode nm)
protected void	recoverResourceRequestForContainer(RMContainer rmContainer) Recover resource request back from RMContainer when a container is preempted before AM pulled the same.
protected void	refreshMaximumAllocation(org.apache.hadoop.yarn.api.records.Resource newMaxAlloc)
protected void	releaseContainers(List<org.apache.hadoop.yarn.api.records.ContainerId> containers, SchedulerApplicationAttempt attempt)
void	removeQueue(String queueName) Remove an existing queue.
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	setEntitlement(String queue, QueueEntitlement entitlement) This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
protected void	updateMaximumAllocation(SchedulerNode node, boolean add)
void	updateNodeResource(RMNode nm, org.apache.hadoop.yarn.api.records.ResourceOption resourceOption) Process resource update on a node.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler

reinitialize, setRMContext

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

allocate, checkAccess, getAppsInQueue, getIncrementResourceCapability, getNumClusterNodes, getQueueInfo, getQueueUserAclInfo, getResourceCalculator, getRootQueueMetrics

Methods inherited from interface org.apache.hadoop.yarn.event.EventHandler

handle

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.recovery.Recoverable

recover

Field Detail

nodes

protected Map<org.apache.hadoop.yarn.api.records.NodeId,N extends SchedulerNode> nodes

clusterResource

protected org.apache.hadoop.yarn.api.records.Resource clusterResource

minimumAllocation

protected org.apache.hadoop.yarn.api.records.Resource minimumAllocation

rmContext

protected RMContext rmContext

applications

protected Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T extends SchedulerApplicationAttempt>> applications

nmExpireInterval

protected int nmExpireInterval

EMPTY_CONTAINER_LIST

protected static final List<org.apache.hadoop.yarn.api.records.Container> EMPTY_CONTAINER_LIST

EMPTY_ALLOCATION

protected static final Allocation EMPTY_ALLOCATION

Constructor Detail

AbstractYarnScheduler

public AbstractYarnScheduler(String name)

Construct the service.

Parameters:

name - service name

Method Detail

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

getTransferredContainers

public List<org.apache.hadoop.yarn.api.records.Container> getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId currentAttempt)

getSchedulerApplications

public Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>> getSchedulerApplications()

getClusterResource

public org.apache.hadoop.yarn.api.records.Resource getClusterResource()

Description copied from interface: YarnScheduler

Get the whole resource capacity of the cluster.

Specified by:

getClusterResource in interface YarnScheduler

Returns:

the whole resource capacity of the cluster.

getMinimumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMinimumResourceCapability()

Description copied from interface: YarnScheduler

Get minimum allocatable Resource.

Specified by:

getMinimumResourceCapability in interface YarnScheduler

Returns:

minimum allocatable resource

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability()

Description copied from interface: YarnScheduler

Get maximum allocatable Resource at the cluster level.

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Returns:

maximum allocatable resource

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(String queueName)

Description copied from interface: YarnScheduler

Get maximum allocatable Resource for the queue specified.

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Parameters:

queueName - queue name

Returns:

maximum allocatable resource

initMaximumResourceCapability

protected void initMaximumResourceCapability(org.apache.hadoop.yarn.api.records.Resource maximumAllocation)

containerLaunchedOnNode

protected void containerLaunchedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                           SchedulerNode node)

getApplicationAttempt

public T getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)

getSchedulerAppInfo

public SchedulerAppReport getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Description copied from interface: YarnScheduler

Get the Scheduler app for a given app attempt Id.

Specified by:

getSchedulerAppInfo in interface YarnScheduler

Parameters:

appAttemptId - the id of the application attempt

Returns:

SchedulerApp for this given attempt.

getAppResourceUsageReport

public org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Description copied from interface: YarnScheduler

Get a resource usage report from a given app attempt ID.

Specified by:

getAppResourceUsageReport in interface YarnScheduler

Parameters:

appAttemptId - the id of the application attempt

Returns:

resource usage report for this given attempt

getCurrentAttemptForContainer

public T getCurrentAttemptForContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

getRMContainer

public RMContainer getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from interface: YarnScheduler

Get the container for the given containerId.

Specified by:

getRMContainer in interface YarnScheduler

Returns:

the container for the given containerId.

getNodeReport

public SchedulerNodeReport getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Description copied from interface: YarnScheduler

Get node resource usage report.

Specified by:

getNodeReport in interface YarnScheduler

Returns:

the SchedulerNodeReport for the node or null if nodeId does not point to a defined node.

moveApplication

public String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                     String newQueue)
                       throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Moves the given application to the given queue

Specified by:

moveApplication in interface YarnScheduler

Returns:

the name of the queue the application was placed into

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

removeQueue

public void removeQueue(String queueName)
                 throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).

Specified by:

removeQueue in interface YarnScheduler

Parameters:

queueName - name of the queue to remove

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

addQueue

public void addQueue(Queue newQueue)
              throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).

Specified by:

addQueue in interface YarnScheduler

Parameters:

newQueue - the queue being added.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

setEntitlement

public void setEntitlement(String queue,
                  QueueEntitlement entitlement)
                    throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.

Specified by:

setEntitlement in interface YarnScheduler

Parameters:

queue - the queue for which we change entitlement

entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

recoverContainersOnNode

public void recoverContainersOnNode(List<org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus> containerReports,
                           RMNode nm)

recoverResourceRequestForContainer

protected void recoverResourceRequestForContainer(RMContainer rmContainer)

Recover resource request back from RMContainer when a container is preempted before AM pulled the same. If container is pulled by AM, then RMContainer will not have resource request to recover.

Parameters:

rmContainer -

createReleaseCache

protected void createReleaseCache()

completedContainer

protected abstract void completedContainer(RMContainer rmContainer,
                      org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                      RMContainerEventType event)

releaseContainers

protected void releaseContainers(List<org.apache.hadoop.yarn.api.records.ContainerId> containers,
                     SchedulerApplicationAttempt attempt)

getSchedulerNode

public SchedulerNode getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)

moveAllApps

public void moveAllApps(String sourceQueue,
               String destQueue)
                 throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Completely drain sourceQueue of applications, by moving all of them to destQueue.

Specified by:

moveAllApps in interface YarnScheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

killAllAppsInQueue

public void killAllAppsInQueue(String queueName)
                        throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Terminate all applications in the specified queue.

Specified by:

killAllAppsInQueue in interface YarnScheduler

Parameters:

queueName - the name of queue to be drained

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

updateNodeResource

public void updateNodeResource(RMNode nm,
                      org.apache.hadoop.yarn.api.records.ResourceOption resourceOption)

Process resource update on a node.

getSchedulingResourceTypes

public EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()

Return a collection of the resource types that are considered when scheduling

Specified by:

getSchedulingResourceTypes in interface YarnScheduler

Returns:

an EnumSet containing the resource types

getPlanQueues

public Set<String> getPlanQueues()
                          throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Gets the list of names for queues managed by the Reservation System

Specified by:

getPlanQueues in interface YarnScheduler

Returns:

the list of queues which support reservations

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

updateMaximumAllocation

protected void updateMaximumAllocation(SchedulerNode node,
                           boolean add)

refreshMaximumAllocation

protected void refreshMaximumAllocation(org.apache.hadoop.yarn.api.records.Resource newMaxAlloc)

getPendingResourceRequestsForAttempt

public List<org.apache.hadoop.yarn.api.records.ResourceRequest> getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)