Package org.apache.hadoop.yarn.server.resourcemanager.reservation

Class AbstractSchedulerPlanFollower

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.reservation.AbstractSchedulerPlanFollower

All Implemented Interfaces:

java.lang.Runnable, PlanFollower

Direct Known Subclasses:

CapacitySchedulerPlanFollower, FairSchedulerPlanFollower

public abstract class AbstractSchedulerPlanFollower
extends java.lang.Object
implements PlanFollower

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.yarn.util.Clock	clock
protected java.util.Collection<Plan>	plans
protected YarnScheduler	scheduler

Constructor Summary

Constructors

Constructor	Description
AbstractSchedulerPlanFollower()

Method Summary

Modifier and Type	Method	Description
protected abstract void	addReservationQueue(java.lang.String planQueueName, Queue queue, java.lang.String currResId)	Add a new reservation queue for reservation currResId for this planQueue.
protected void	cleanupExpiredQueues(java.lang.String planQueueName, boolean shouldMove, java.util.Set<java.lang.String> toRemove, java.lang.String defReservationQueue)	First sets entitlement of queues to zero to prevent new app submission.
protected abstract void	createDefaultReservationQueue(java.lang.String planQueueName, Queue queue, java.lang.String defReservationQueue)	Creates the default reservation queue for use when no reservation is used for applications submitted to this planQueue.
protected abstract java.util.List<? extends Queue>	getChildReservationQueues(Queue planQueue)	Get a list of reservation queues for this planQueue.
protected abstract Queue	getPlanQueue(java.lang.String planQueueName)	Get queue associated with reservable queue named.
protected abstract org.apache.hadoop.yarn.api.records.Resource	getPlanResources(Plan plan, Queue queue, org.apache.hadoop.yarn.api.records.Resource clusterResources)	Get plan resources for this planQueue.
protected java.lang.String	getReservationIdFromQueueName(java.lang.String resQueueName)
protected java.lang.String	getReservationQueueName(java.lang.String planQueueName, java.lang.String reservationId)
protected abstract org.apache.hadoop.yarn.api.records.Resource	getReservationQueueResourceIfExists(Plan plan, org.apache.hadoop.yarn.api.records.ReservationId reservationId)	Get reservation queue resources if it exists otherwise return null.
protected int	getReservedResources(long now, java.util.Set<ReservationAllocation> currentReservations, java.util.Set<java.lang.String> curReservationNames, org.apache.hadoop.yarn.api.records.Resource reservedResources)
void	init(org.apache.hadoop.yarn.util.Clock clock, ResourceScheduler sched, java.util.Collection<Plan> plans)	Init function that configures the PlanFollower, by providing:
void	run()
void	setPlans(java.util.Collection<Plan> plans)	Setter for the list of plans.
protected void	setQueueEntitlement(java.lang.String planQueueName, java.lang.String currResId, float targetCapacity, float maxCapacity)
protected java.util.List<ReservationAllocation>	sortByDelta(java.util.List<ReservationAllocation> currentReservations, long now, Plan plan)	Sort in the order from the least new amount of resources asked (likely negative) to the highest.
void	synchronizePlan(Plan plan, boolean shouldReplan)	The function performing the actual synchronization operation for a given Plan.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

plans

protected java.util.Collection<Plan> plans

scheduler

protected YarnScheduler scheduler

clock

protected org.apache.hadoop.yarn.util.Clock clock

Constructor Detail

AbstractSchedulerPlanFollower

public AbstractSchedulerPlanFollower()

Method Detail

init

public void init(org.apache.hadoop.yarn.util.Clock clock,
                 ResourceScheduler sched,
                 java.util.Collection<Plan> plans)

Description copied from interface: PlanFollower

Init function that configures the PlanFollower, by providing:

Specified by:

init in interface PlanFollower

Parameters:

clock - a reference to the system clock.

sched - a reference to the underlying scheduler

plans - references to the plans we should keep synchronized at every time tick.

run

public void run()

Specified by:

run in interface java.lang.Runnable

setPlans

public void setPlans(java.util.Collection<Plan> plans)

Description copied from interface: PlanFollower

Setter for the list of plans.

Specified by:

setPlans in interface PlanFollower

Parameters:

plans - the collection of Plans we operate on at every time tick.

synchronizePlan

public void synchronizePlan(Plan plan,
                            boolean shouldReplan)

Description copied from interface: PlanFollower

The function performing the actual synchronization operation for a given Plan. This is normally invoked by the run method, but it can be invoked synchronously to avoid race conditions when a user's reservation request start time is imminent.

Specified by:

synchronizePlan in interface PlanFollower

Parameters:

plan - the Plan to synchronize

shouldReplan - replan on reduction of plan capacity if true or proportionally scale down reservations if false

getReservationIdFromQueueName

protected java.lang.String getReservationIdFromQueueName(java.lang.String resQueueName)

setQueueEntitlement

protected void setQueueEntitlement(java.lang.String planQueueName,
                                   java.lang.String currResId,
                                   float targetCapacity,
                                   float maxCapacity)
                            throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

getReservationQueueName

protected java.lang.String getReservationQueueName(java.lang.String planQueueName,
                                                   java.lang.String reservationId)

cleanupExpiredQueues

protected void cleanupExpiredQueues(java.lang.String planQueueName,
                                    boolean shouldMove,
                                    java.util.Set<java.lang.String> toRemove,
                                    java.lang.String defReservationQueue)

First sets entitlement of queues to zero to prevent new app submission. Then move all apps in the set of queues to the parent plan queue's default reservation queue if move is enabled. Finally cleanups the queue by killing any apps (if move is disabled or move failed) and removing the queue

Parameters:

planQueueName - the name of PlanQueue

shouldMove - flag to indicate if any running apps should be moved or killed

toRemove - the remnant apps to clean up

defReservationQueue - the default ReservationQueue of the Plan

getReservedResources

protected int getReservedResources(long now,
                                   java.util.Set<ReservationAllocation> currentReservations,
                                   java.util.Set<java.lang.String> curReservationNames,
                                   org.apache.hadoop.yarn.api.records.Resource reservedResources)

sortByDelta

protected java.util.List<ReservationAllocation> sortByDelta(java.util.List<ReservationAllocation> currentReservations,
                                                            long now,
                                                            Plan plan)

Sort in the order from the least new amount of resources asked (likely negative) to the highest. This prevents "order-of-operation" errors related to exceeding 100% capacity temporarily.

Parameters:

currentReservations - the currently active reservations

now - the current time

plan - the Plan that is being considered

Returns:

the sorted list of ReservationAllocations

getPlanQueue

protected abstract Queue getPlanQueue(java.lang.String planQueueName)

Get queue associated with reservable queue named.

Parameters:

planQueueName - name of the reservable queue

Returns:

queue associated with the reservable queue

getChildReservationQueues

protected abstract java.util.List<? extends Queue> getChildReservationQueues(Queue planQueue)

Get a list of reservation queues for this planQueue.

Parameters:

planQueue - the queue for the current Plan

Returns:

the queues corresponding to the reservations

addReservationQueue

protected abstract void addReservationQueue(java.lang.String planQueueName,
                                            Queue queue,
                                            java.lang.String currResId)

Add a new reservation queue for reservation currResId for this planQueue.

Parameters:

planQueueName - name of the reservable queue.

queue - the queue for the current Plan.

currResId - curr reservationId.

createDefaultReservationQueue

protected abstract void createDefaultReservationQueue(java.lang.String planQueueName,
                                                      Queue queue,
                                                      java.lang.String defReservationQueue)

Creates the default reservation queue for use when no reservation is used for applications submitted to this planQueue.

Parameters:

planQueueName - name of the reservable queue

queue - the queue for the current Plan

defReservationQueue - name of the default ReservationQueue

getPlanResources

protected abstract org.apache.hadoop.yarn.api.records.Resource getPlanResources(Plan plan,
                                                                                Queue queue,
                                                                                org.apache.hadoop.yarn.api.records.Resource clusterResources)

Get plan resources for this planQueue.

Parameters:

plan - the current Plan being considered

queue - the queue for the current Plan

clusterResources - the resources available in the cluster

Returns:

the resources allocated to the specified Plan

getReservationQueueResourceIfExists

protected abstract org.apache.hadoop.yarn.api.records.Resource getReservationQueueResourceIfExists(Plan plan,
                                                                                                   org.apache.hadoop.yarn.api.records.ReservationId reservationId)

Get reservation queue resources if it exists otherwise return null.

Parameters:

plan - the current Plan being considered

reservationId - the identifier of the reservation

Returns:

the resources allocated to the specified reservation