Package org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity

Class ProportionalCapacityPreemptionPolicy

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy

All Implemented Interfaces:

CapacitySchedulerPreemptionContext, SchedulingEditPolicy

public class ProportionalCapacityPreemptionPolicy
extends java.lang.Object
implements SchedulingEditPolicy, CapacitySchedulerPreemptionContext

This class implement a SchedulingEditPolicy that is designed to be paired with the CapacityScheduler. At every invocation of editSchedule() it computes the ideal amount of resources assigned to each queue (for each queue in the hierarchy), and determines whether preemption is needed. Overcapacity is distributed among queues in a weighted fair manner, where the weight is the amount of guaranteed capacity for the queue. Based on this ideal assignment it determines whether preemption is required and select a set of containers from each application that would be killed if the corresponding amount of resources is not freed up by the application. If not in observeOnly mode, it triggers preemption requests via a ContainerPreemptEvent that the ResourceManager will ensure to deliver to the application (or to execute). If the deficit of resources is persistent over a long enough period of time this policy will trigger forced termination of containers (again by generating ContainerPreemptEvent).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ProportionalCapacityPreemptionPolicy.IntraQueuePreemptionOrderPolicy	IntraQueuePreemptionOrder will be used to define various priority orders which could be configured by admin.

Constructor Summary

Constructors

Constructor	Description
ProportionalCapacityPreemptionPolicy()
ProportionalCapacityPreemptionPolicy(RMContext context, CapacityScheduler scheduler, org.apache.hadoop.yarn.util.Clock clock)

Method Summary

Modifier and Type	Method	Description
void	addPartitionToUnderServedQueues(java.lang.String queueName, java.lang.String partition)
void	editSchedule()	This method is invoked at regular intervals.
java.util.Set<java.lang.String>	getAllPartitions()
int	getClusterMaxApplicationPriority()
boolean	getCrossQueuePreemptionConservativeDRF()
long	getDefaultMaximumKillWaitTimeout()
boolean	getInQueuePreemptionConservativeDRF()
ProportionalCapacityPreemptionPolicy.IntraQueuePreemptionOrderPolicy	getIntraQueuePreemptionOrderPolicy()
java.util.Set<org.apache.hadoop.yarn.api.records.ContainerId>	getKillableContainers()
java.util.Set<java.lang.String>	getLeafQueueNames()
float	getMaxAllowableLimitForIntraQueuePreemption()
double	getMaxIgnoreOverCapacity()
float	getMinimumThresholdForIntraQueuePreemption()
long	getMonitoringInterval()
double	getNaturalTerminationFactor()
org.apache.hadoop.yarn.api.records.Resource	getPartitionResource(java.lang.String partition)
java.lang.String	getPolicyName()
TempQueuePerPartition	getQueueByPartition(java.lang.String queueName, java.lang.String partition)	Get queue partition by given queueName and partitionName
java.util.Collection<TempQueuePerPartition>	getQueuePartitions(java.lang.String queueName)	Get all queue partitions by given queueName
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator()
RMContext	getRMContext()
CapacityScheduler	getScheduler()
java.util.Map<RMContainer,java.lang.Long>	getToPreemptContainers()
java.util.LinkedHashSet<java.lang.String>	getUnderServedQueuesPerPartition(java.lang.String partition)
void	init(org.apache.hadoop.conf.Configuration config, RMContext context, ResourceScheduler sched)
boolean	isObserveOnly()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ProportionalCapacityPreemptionPolicy

public ProportionalCapacityPreemptionPolicy()

ProportionalCapacityPreemptionPolicy

@VisibleForTesting
public ProportionalCapacityPreemptionPolicy(RMContext context,
                                            CapacityScheduler scheduler,
                                            org.apache.hadoop.yarn.util.Clock clock)

Method Detail

init

public void init(org.apache.hadoop.conf.Configuration config,
                 RMContext context,
                 ResourceScheduler sched)

Specified by:

init in interface SchedulingEditPolicy

getResourceCalculator

public org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

Specified by:

getResourceCalculator in interface CapacitySchedulerPreemptionContext

editSchedule

public void editSchedule()

Description copied from interface: SchedulingEditPolicy

This method is invoked at regular intervals. Internally the policy is allowed to track containers and affect the scheduler. The "actions" performed are passed back through an EventHandler.

Specified by:

editSchedule in interface SchedulingEditPolicy

getMonitoringInterval

public long getMonitoringInterval()

Specified by:

getMonitoringInterval in interface SchedulingEditPolicy

getPolicyName

public java.lang.String getPolicyName()

Specified by:

getPolicyName in interface SchedulingEditPolicy

getToPreemptContainers

@VisibleForTesting
public java.util.Map<RMContainer,java.lang.Long> getToPreemptContainers()

getQueueByPartition

public TempQueuePerPartition getQueueByPartition(java.lang.String queueName,
                                                 java.lang.String partition)

Get queue partition by given queueName and partitionName

Specified by:

getQueueByPartition in interface CapacitySchedulerPreemptionContext

getQueuePartitions

public java.util.Collection<TempQueuePerPartition> getQueuePartitions(java.lang.String queueName)

Get all queue partitions by given queueName

Specified by:

getQueuePartitions in interface CapacitySchedulerPreemptionContext

getScheduler

public CapacityScheduler getScheduler()

Specified by:

getScheduler in interface CapacitySchedulerPreemptionContext

getRMContext

public RMContext getRMContext()

Specified by:

getRMContext in interface CapacitySchedulerPreemptionContext

isObserveOnly

public boolean isObserveOnly()

Specified by:

isObserveOnly in interface CapacitySchedulerPreemptionContext

getKillableContainers

public java.util.Set<org.apache.hadoop.yarn.api.records.ContainerId> getKillableContainers()

Specified by:

getKillableContainers in interface CapacitySchedulerPreemptionContext

getMaxIgnoreOverCapacity

public double getMaxIgnoreOverCapacity()

Specified by:

getMaxIgnoreOverCapacity in interface CapacitySchedulerPreemptionContext

getNaturalTerminationFactor

public double getNaturalTerminationFactor()

Specified by:

getNaturalTerminationFactor in interface CapacitySchedulerPreemptionContext

getLeafQueueNames

public java.util.Set<java.lang.String> getLeafQueueNames()

Specified by:

getLeafQueueNames in interface CapacitySchedulerPreemptionContext

getAllPartitions

public java.util.Set<java.lang.String> getAllPartitions()

Specified by:

getAllPartitions in interface CapacitySchedulerPreemptionContext

getClusterMaxApplicationPriority

public int getClusterMaxApplicationPriority()

Specified by:

getClusterMaxApplicationPriority in interface CapacitySchedulerPreemptionContext

getMaxAllowableLimitForIntraQueuePreemption

public float getMaxAllowableLimitForIntraQueuePreemption()

Specified by:

getMaxAllowableLimitForIntraQueuePreemption in interface CapacitySchedulerPreemptionContext

getMinimumThresholdForIntraQueuePreemption

public float getMinimumThresholdForIntraQueuePreemption()

Specified by:

getMinimumThresholdForIntraQueuePreemption in interface CapacitySchedulerPreemptionContext

getPartitionResource

public org.apache.hadoop.yarn.api.records.Resource getPartitionResource(java.lang.String partition)

Specified by:

getPartitionResource in interface CapacitySchedulerPreemptionContext

getUnderServedQueuesPerPartition

public java.util.LinkedHashSet<java.lang.String> getUnderServedQueuesPerPartition(java.lang.String partition)

Specified by:

getUnderServedQueuesPerPartition in interface CapacitySchedulerPreemptionContext

addPartitionToUnderServedQueues

public void addPartitionToUnderServedQueues(java.lang.String queueName,
                                            java.lang.String partition)

Specified by:

addPartitionToUnderServedQueues in interface CapacitySchedulerPreemptionContext

getIntraQueuePreemptionOrderPolicy

public ProportionalCapacityPreemptionPolicy.IntraQueuePreemptionOrderPolicy getIntraQueuePreemptionOrderPolicy()

Specified by:

getIntraQueuePreemptionOrderPolicy in interface CapacitySchedulerPreemptionContext

getCrossQueuePreemptionConservativeDRF

public boolean getCrossQueuePreemptionConservativeDRF()

Specified by:

getCrossQueuePreemptionConservativeDRF in interface CapacitySchedulerPreemptionContext

getInQueuePreemptionConservativeDRF

public boolean getInQueuePreemptionConservativeDRF()

Specified by:

getInQueuePreemptionConservativeDRF in interface CapacitySchedulerPreemptionContext

getDefaultMaximumKillWaitTimeout

public long getDefaultMaximumKillWaitTimeout()

Specified by:

getDefaultMaximumKillWaitTimeout in interface CapacitySchedulerPreemptionContext