org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class CapacityScheduler

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>, Recoverable, CapacitySchedulerContext, PreemptableResourceScheduler, ResourceScheduler, YarnScheduler

@InterfaceAudience.LimitedPrivate(value="yarn")
@InterfaceStability.Evolving
public class CapacityScheduler
extends AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>
implements PreemptableResourceScheduler, CapacitySchedulerContext, org.apache.hadoop.conf.Configurable

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static String	ROOT_QUEUE
protected long	THREAD_JOIN_TIMEOUT_MS

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler

applications, clusterResource, EMPTY_ALLOCATION, EMPTY_CONTAINER_LIST, minimumAllocation, nmExpireInterval, nodes, rmContext

Constructor Summary

Constructors

Constructor and Description
CapacityScheduler()

Method Summary

Methods

Modifier and Type	Method and Description
void	addQueue(Queue queue) Add to the scheduler a new Queue.
Allocation	allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask, List<org.apache.hadoop.yarn.api.records.ContainerId> release, List<String> blacklistAdditions, List<String> blacklistRemovals) The main api between the ApplicationMaster and the Scheduler.
boolean	checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI, org.apache.hadoop.yarn.api.records.QueueACL acl, String queueName) Check if the user has permission to perform the operation.
protected void	completedContainer(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
void	dropContainerReservation(RMContainer container) If the scheduler support container reservations, this method is used to ask the scheduler to drop the reservation for the given container.
FiCaSchedulerApp	getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)
Comparator<FiCaSchedulerApp>	getApplicationComparator()
List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId>	getAppsInQueue(String queueName) Gets the apps under a given queue
org.apache.hadoop.conf.Configuration	getConf() Get the yarn configuration.
CapacitySchedulerConfiguration	getConfiguration()
RMContainerTokenSecretManager	getContainerTokenSecretManager()
org.apache.hadoop.yarn.api.records.Resource	getIncrementResourceCapability() Get increment allocatable Resource.
String	getMappedQueueForTest(String user)
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability(String queueName) Get maximum allocatable Resource for the queue specified.
FiCaSchedulerNode	getNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
int	getNumClusterNodes() Get the number of nodes available in the cluster.
Set<String>	getPlanQueues() Gets the list of names for queues managed by the Reservation System
CSQueue	getQueue(String queueName)
Comparator<CSQueue>	getQueueComparator()
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(String queueName, boolean includeChildQueues, boolean recursive) Get queue information
List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>	getQueueUserAclInfo() Get acls for queues for current user.
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator()
RMContext	getRMContext()
CSQueue	getRootQueue()
QueueMetrics	getRootQueueMetrics() Get the root queue for the scheduler.
EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>	getSchedulingResourceTypes() Return a collection of the resource types that are considered when scheduling
void	handle(SchedulerEvent event)
void	killContainer(RMContainer cont) Ask the scheduler to forcibly interrupt the container given as input
String	moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String targetQueueName) Moves the given application to the given queue
void	preemptContainer(org.apache.hadoop.yarn.api.records.ApplicationAttemptId aid, RMContainer cont) Ask the scheduler to obtain back the container from a specific application by issuing a preemption request
void	recover(RMStateStore.RMState state)
void	reinitialize(org.apache.hadoop.conf.Configuration conf, RMContext rmContext) Re-initialize the ResourceScheduler.
void	removeQueue(String queueName) Remove an existing queue.
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	serviceStart()
void	serviceStop()
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setEntitlement(String inQueue, QueueEntitlement entitlement) This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
static void	setQueueAcls(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer, Map<String,CSQueue> queues)
void	setRMContext(RMContext rmContext) Set RMContext for ResourceScheduler.

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler

containerLaunchedOnNode, createReleaseCache, getAppResourceUsageReport, getClusterResource, getCurrentAttemptForContainer, getMaximumResourceCapability, getMinimumResourceCapability, getNodeReport, getPendingResourceRequestsForAttempt, getRMContainer, getSchedulerAppInfo, getSchedulerApplications, getSchedulerNode, getTransferredContainers, initMaximumResourceCapability, killAllAppsInQueue, moveAllApps, recoverContainersOnNode, recoverResourceRequestForContainer, refreshMaximumAllocation, releaseContainers, updateMaximumAllocation, updateNodeResource

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

getAppResourceUsageReport, getClusterResource, getMaximumResourceCapability, getMinimumResourceCapability, getNodeReport, getRMContainer, getSchedulerAppInfo, killAllAppsInQueue, moveAllApps

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext

getClusterResource, getMaximumResourceCapability, getMinimumResourceCapability

Field Detail

THREAD_JOIN_TIMEOUT_MS

protected final long THREAD_JOIN_TIMEOUT_MS

See Also:

Constant Field Values

ROOT_QUEUE

@InterfaceAudience.Private
public static final String ROOT_QUEUE

See Also:

Constant Field Values

Constructor Detail

CapacityScheduler

public CapacityScheduler()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Description copied from interface: CapacitySchedulerContext

Get the yarn configuration.

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

Specified by:

getConf in interface CapacitySchedulerContext

getMappedQueueForTest

public String getMappedQueueForTest(String user)
                             throws IOException

Throws:

IOException

getRootQueueMetrics

public QueueMetrics getRootQueueMetrics()

Description copied from interface: YarnScheduler

Get the root queue for the scheduler.

Specified by:

getRootQueueMetrics in interface YarnScheduler

Returns:

the root queue for the scheduler.

getRootQueue

public CSQueue getRootQueue()

getConfiguration

public CapacitySchedulerConfiguration getConfiguration()

Specified by:

getConfiguration in interface CapacitySchedulerContext

getContainerTokenSecretManager

public RMContainerTokenSecretManager getContainerTokenSecretManager()

Specified by:

getContainerTokenSecretManager in interface CapacitySchedulerContext

getApplicationComparator

public Comparator<FiCaSchedulerApp> getApplicationComparator()

Specified by:

getApplicationComparator in interface CapacitySchedulerContext

getResourceCalculator

public org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

Specified by:

getResourceCalculator in interface CapacitySchedulerContext

Specified by:

getResourceCalculator in interface YarnScheduler

getQueueComparator

public Comparator<CSQueue> getQueueComparator()

Specified by:

getQueueComparator in interface CapacitySchedulerContext

getNumClusterNodes

public int getNumClusterNodes()

Description copied from interface: YarnScheduler

Get the number of nodes available in the cluster.

Specified by:

getNumClusterNodes in interface CapacitySchedulerContext

Specified by:

getNumClusterNodes in interface YarnScheduler

Returns:

the number of available nodes.

getRMContext

public RMContext getRMContext()

Specified by:

getRMContext in interface CapacitySchedulerContext

setRMContext

public void setRMContext(RMContext rmContext)

Description copied from interface: ResourceScheduler

Set RMContext for ResourceScheduler. This method should be called immediately after instantiating a scheduler once.

Specified by:

setRMContext in interface ResourceScheduler

Parameters:

rmContext - created by ResourceManager

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Throws:

Exception

serviceStart

public void serviceStart()
                  throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

public void serviceStop()
                 throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

reinitialize

public void reinitialize(org.apache.hadoop.conf.Configuration conf,
                RMContext rmContext)
                  throws IOException

Description copied from interface: ResourceScheduler

Re-initialize the ResourceScheduler.

Specified by:

reinitialize in interface ResourceScheduler

Parameters:

conf - configuration

Throws:

IOException

setQueueAcls

public static void setQueueAcls(org.apache.hadoop.yarn.security.YarnAuthorizationProvider authorizer,
                Map<String,CSQueue> queues)
                         throws IOException

Throws:

IOException

getQueue

public CSQueue getQueue(String queueName)

allocate

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public Allocation allocate(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                                                                            List<org.apache.hadoop.yarn.api.records.ResourceRequest> ask,
                                                                            List<org.apache.hadoop.yarn.api.records.ContainerId> release,
                                                                            List<String> blacklistAdditions,
                                                                            List<String> blacklistRemovals)

Description copied from interface: YarnScheduler

The main api between the ApplicationMaster and the Scheduler. The ApplicationMaster is updating his future resource requirements and may release containers he doens't need.

Specified by:

allocate in interface YarnScheduler

Returns:

the Allocation for the application

getQueueInfo

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(String queueName,
                                                                                                                  boolean includeChildQueues,
                                                                                                                  boolean recursive)
                                                          throws IOException

Description copied from interface: YarnScheduler

Get queue information

Specified by:

getQueueInfo in interface YarnScheduler

Parameters:

queueName - queue name

includeChildQueues - include child queues?

recursive - get children queues?

Returns:

queue information

Throws:

IOException

getQueueUserAclInfo

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo()

Description copied from interface: YarnScheduler

Get acls for queues for current user.

Specified by:

getQueueUserAclInfo in interface YarnScheduler

Returns:

acls for queues for current user

getIncrementResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getIncrementResourceCapability()

Description copied from interface: YarnScheduler

Get increment allocatable Resource.

Specified by:

getIncrementResourceCapability in interface YarnScheduler

Returns:

incremental allocatable resource.

handle

public void handle(SchedulerEvent event)

Specified by:

handle in interface org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>

completedContainer

@Lock(value=CapacityScheduler.class)
protected void completedContainer(RMContainer rmContainer,
                                                  org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                                                  RMContainerEventType event)

Specified by:

completedContainer in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

getApplicationAttempt

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public FiCaSchedulerApp getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)

Overrides:

getApplicationAttempt in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

getNode

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public FiCaSchedulerNode getNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Specified by:

getNode in interface CapacitySchedulerContext

recover

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public void recover(RMStateStore.RMState state)
             throws Exception

Specified by:

recover in interface Recoverable

Throws:

Exception

dropContainerReservation

public void dropContainerReservation(RMContainer container)

Description copied from interface: PreemptableResourceScheduler

If the scheduler support container reservations, this method is used to ask the scheduler to drop the reservation for the given container.

Specified by:

dropContainerReservation in interface PreemptableResourceScheduler

Parameters:

container - Reference to reserved container allocation.

preemptContainer

public void preemptContainer(org.apache.hadoop.yarn.api.records.ApplicationAttemptId aid,
                    RMContainer cont)

Description copied from interface: PreemptableResourceScheduler

Ask the scheduler to obtain back the container from a specific application by issuing a preemption request

Specified by:

preemptContainer in interface PreemptableResourceScheduler

Parameters:

aid - the application from which we want to get a container back

cont - the container we want back

killContainer

public void killContainer(RMContainer cont)

Description copied from interface: PreemptableResourceScheduler

Ask the scheduler to forcibly interrupt the container given as input

Specified by:

killContainer in interface PreemptableResourceScheduler

checkAccess

public boolean checkAccess(org.apache.hadoop.security.UserGroupInformation callerUGI,
                  org.apache.hadoop.yarn.api.records.QueueACL acl,
                  String queueName)

Description copied from interface: YarnScheduler

Check if the user has permission to perform the operation. If the user has QueueACL.ADMINISTER_QUEUE permission, this user can view/modify the applications in this queue

Specified by:

checkAccess in interface YarnScheduler

Returns:

true if the user has the permission, false otherwise

getAppsInQueue

public List<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> getAppsInQueue(String queueName)

Description copied from interface: YarnScheduler

Gets the apps under a given queue

Specified by:

getAppsInQueue in interface YarnScheduler

Parameters:

queueName - the name of the queue.

Returns:

a collection of app attempt ids in the given queue.

removeQueue

public void removeQueue(String queueName)
                 throws SchedulerDynamicEditException

Description copied from interface: YarnScheduler

Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).

Specified by:

removeQueue in interface YarnScheduler

Overrides:

removeQueue in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - name of the queue to remove

Throws:

SchedulerDynamicEditException

addQueue

public void addQueue(Queue queue)
              throws SchedulerDynamicEditException

Description copied from interface: YarnScheduler

Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).

Specified by:

addQueue in interface YarnScheduler

Overrides:

addQueue in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queue - the queue being added.

Throws:

SchedulerDynamicEditException

setEntitlement

public void setEntitlement(String inQueue,
                  QueueEntitlement entitlement)
                    throws SchedulerDynamicEditException,
                           org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.

Specified by:

setEntitlement in interface YarnScheduler

Overrides:

setEntitlement in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

inQueue - the queue for which we change entitlement

entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

SchedulerDynamicEditException

moveApplication

public String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                     String targetQueueName)
                       throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Moves the given application to the given queue

Specified by:

moveApplication in interface YarnScheduler

Overrides:

moveApplication in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

the name of the queue the application was placed into

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

getSchedulingResourceTypes

public EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()

Return a collection of the resource types that are considered when scheduling

Specified by:

getSchedulingResourceTypes in interface YarnScheduler

Overrides:

getSchedulingResourceTypes in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

an EnumSet containing the resource types

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(String queueName)

Description copied from interface: YarnScheduler

Get maximum allocatable Resource for the queue specified.

Specified by:

getMaximumResourceCapability in interface CapacitySchedulerContext

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Overrides:

getMaximumResourceCapability in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Parameters:

queueName - queue name

Returns:

maximum allocatable resource

getPlanQueues

public Set<String> getPlanQueues()

Description copied from interface: YarnScheduler

Gets the list of names for queues managed by the Reservation System

Specified by:

getPlanQueues in interface YarnScheduler

Overrides:

getPlanQueues in class AbstractYarnScheduler<FiCaSchedulerApp,FiCaSchedulerNode>

Returns:

the list of queues which support reservations