Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement

Class LocalityAppPlacementAllocator<N extends SchedulerNode>

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.AppPlacementAllocator<N>

org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.LocalityAppPlacementAllocator<N>

public class LocalityAppPlacementAllocator<N extends SchedulerNode>
extends AppPlacementAllocator<N>

This is an implementation of the AppPlacementAllocator that takes into account locality preferences (node, rack, any) when allocating containers.

Field Summary

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.AppPlacementAllocator

appSchedulingInfo, rmContext, schedulerRequestKey

Constructor Summary

Constructors

Constructor	Description
LocalityAppPlacementAllocator()

Method Summary

Modifier and Type	Method	Description
ContainerRequest	allocate(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerKey, NodeType type, SchedulerNode node)	Notify container allocated.
boolean	canAllocate(NodeType type, SchedulerNode node)	We can still have pending requirement for a given NodeType and node
boolean	canDelayTo(java.lang.String resourceName)	Can delay to give locality? TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl.
org.apache.hadoop.yarn.api.records.ResourceRequest	cloneResourceRequest(org.apache.hadoop.yarn.api.records.ResourceRequest request)
int	getOutstandingAsksCount(java.lang.String resourceName)	Get #pending-allocations for given resourceName.
PendingAsk	getPendingAsk(java.lang.String resourceName)	Get pending ask for given resourceName.
java.lang.String	getPrimaryRequestedNodePartition()	It is possible that one request can accept multiple node partition, So this method returns primary node partition for pending resource / headroom calculation.
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.ResourceRequest>	getResourceRequests()	Get pending ResourceRequests by given schedulerRequestKey
org.apache.hadoop.yarn.api.records.SchedulingRequest	getSchedulingRequest()	Get pending SchedulingRequest.
int	getUniqueLocationAsks()
void	initialize(AppSchedulingInfo appSchedulingInfo, org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, RMContext rmContext)	Initialize this allocator, this will be called by Factory automatically.
boolean	precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode)
boolean	precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode, java.util.Optional<DiagnosticsCollector> dcOpt)	Does this AppPlacementAllocator accept resources on given node?
void	showRequests()	Print human-readable requests to LOG debug.
PendingAskUpdateResult	updatePendingAsk(java.util.Collection<org.apache.hadoop.yarn.api.records.ResourceRequest> requests, boolean recoverPreemptedRequestForAContainer)	Replace existing pending asks by the new requests
PendingAskUpdateResult	updatePendingAsk(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest, boolean recoverPreemptedRequestForAContainer)	Replace existing pending asks by the new SchedulingRequest

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.AppPlacementAllocator

getPlacementAttempt, getPreferredNodeIterator, incrementPlacementAttempt

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LocalityAppPlacementAllocator

public LocalityAppPlacementAllocator()

Method Detail

initialize

public void initialize(AppSchedulingInfo appSchedulingInfo,
                       org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey,
                       RMContext rmContext)

Description copied from class: AppPlacementAllocator

Initialize this allocator, this will be called by Factory automatically.

Overrides:

initialize in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

appSchedulingInfo - appSchedulingInfo

schedulerRequestKey - schedulerRequestKey

rmContext - rmContext

updatePendingAsk

public PendingAskUpdateResult updatePendingAsk(java.util.Collection<org.apache.hadoop.yarn.api.records.ResourceRequest> requests,
                                               boolean recoverPreemptedRequestForAContainer)

Description copied from class: AppPlacementAllocator

Replace existing pending asks by the new requests

Specified by:

updatePendingAsk in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

requests - new asks

recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container

Returns:

true if total pending resource changed

updatePendingAsk

public PendingAskUpdateResult updatePendingAsk(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey,
                                               org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest,
                                               boolean recoverPreemptedRequestForAContainer)
                                        throws org.apache.hadoop.yarn.exceptions.SchedulerInvalidResoureRequestException

Description copied from class: AppPlacementAllocator

Replace existing pending asks by the new SchedulingRequest

Specified by:

updatePendingAsk in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

schedulerRequestKey - scheduler request key

schedulingRequest - new asks

recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container

Returns:

true if total pending resource changed

Throws:

org.apache.hadoop.yarn.exceptions.SchedulerInvalidResoureRequestException

getResourceRequests

public java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.ResourceRequest> getResourceRequests()

Description copied from class: AppPlacementAllocator

Get pending ResourceRequests by given schedulerRequestKey

Specified by:

getResourceRequests in class AppPlacementAllocator<N extends SchedulerNode>

Returns:

Map of resourceName to ResourceRequest

getPendingAsk

public PendingAsk getPendingAsk(java.lang.String resourceName)

Description copied from class: AppPlacementAllocator

Get pending ask for given resourceName. If there's no such pendingAsk, returns PendingAsk.ZERO

Specified by:

getPendingAsk in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

resourceName - resourceName

Returns:

PendingAsk

getOutstandingAsksCount

public int getOutstandingAsksCount(java.lang.String resourceName)

Description copied from class: AppPlacementAllocator

Get #pending-allocations for given resourceName. If there's no such pendingAsk, returns 0

Specified by:

getOutstandingAsksCount in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

resourceName - resourceName

Returns:

#pending-allocations

cloneResourceRequest

public org.apache.hadoop.yarn.api.records.ResourceRequest cloneResourceRequest(org.apache.hadoop.yarn.api.records.ResourceRequest request)

canAllocate

public boolean canAllocate(NodeType type,
                           SchedulerNode node)

Description copied from class: AppPlacementAllocator

We can still have pending requirement for a given NodeType and node

Specified by:

canAllocate in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

type - Locality Type

node - which node we will allocate on

Returns:

true if we has pending requirement

canDelayTo

public boolean canDelayTo(java.lang.String resourceName)

Description copied from class: AppPlacementAllocator

Can delay to give locality? TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.

Specified by:

canDelayTo in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

resourceName - resourceName

Returns:

can/cannot

precheckNode

public boolean precheckNode(SchedulerNode schedulerNode,
                            SchedulingMode schedulingMode,
                            java.util.Optional<DiagnosticsCollector> dcOpt)

Description copied from class: AppPlacementAllocator

Does this AppPlacementAllocator accept resources on given node?

Specified by:

precheckNode in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

schedulerNode - schedulerNode

schedulingMode - schedulingMode

dcOpt - optional diagnostics collector

Returns:

accepted/not

precheckNode

public boolean precheckNode(SchedulerNode schedulerNode,
                            SchedulingMode schedulingMode)

Specified by:

precheckNode in class AppPlacementAllocator<N extends SchedulerNode>

getPrimaryRequestedNodePartition

public java.lang.String getPrimaryRequestedNodePartition()

Description copied from class: AppPlacementAllocator

It is possible that one request can accept multiple node partition, So this method returns primary node partition for pending resource / headroom calculation.

Specified by:

getPrimaryRequestedNodePartition in class AppPlacementAllocator<N extends SchedulerNode>

Returns:

primary requested node partition

getUniqueLocationAsks

public int getUniqueLocationAsks()

Specified by:

getUniqueLocationAsks in class AppPlacementAllocator<N extends SchedulerNode>

Returns:

number of unique location asks with #pending greater than 0, (like /rack1, host1, etc.). TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.

showRequests

public void showRequests()

Description copied from class: AppPlacementAllocator

Print human-readable requests to LOG debug.

Specified by:

showRequests in class AppPlacementAllocator<N extends SchedulerNode>

allocate

public ContainerRequest allocate(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerKey,
                                 NodeType type,
                                 SchedulerNode node)

Description copied from class: AppPlacementAllocator

Notify container allocated.

Specified by:

allocate in class AppPlacementAllocator<N extends SchedulerNode>

Parameters:

schedulerKey - SchedulerRequestKey for this ResourceRequest

type - Type of the allocation

node - Which node this container allocated on

Returns:

ContainerRequest which include resource requests associated with the container. This will be used by scheduler to recover requests. Please refer to ContainerRequest for more details.

getSchedulingRequest

public org.apache.hadoop.yarn.api.records.SchedulingRequest getSchedulingRequest()

Description copied from class: AppPlacementAllocator

Get pending SchedulingRequest.

Specified by:

getSchedulingRequest in class AppPlacementAllocator<N extends SchedulerNode>

Returns:

SchedulingRequest