org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity
Class ProportionalCapacityPreemptionPolicy

java.lang.Object
  extended by org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
All Implemented Interfaces:
SchedulingEditPolicy

public class ProportionalCapacityPreemptionPolicy
extends Object
implements SchedulingEditPolicy

This class implement a SchedulingEditPolicy that is designed to be paired with the CapacityScheduler. At every invocation of editSchedule() it computes the ideal amount of resources assigned to each queue (for each queue in the hierarchy), and determines whether preemption is needed. Overcapacity is distributed among queues in a weighted fair manner, where the weight is the amount of guaranteed capacity for the queue. Based on this ideal assignment it determines whether preemption is required and select a set of containers from each application that would be killed if the corresponding amount of resources is not freed up by the application. If not in observeOnly mode, it triggers preemption requests via a ContainerPreemptEvent that the ResourceManager will ensure to deliver to the application (or to execute). If the deficit of resources is persistent over a long enough period of time this policy will trigger forced termination of containers (again by generating ContainerPreemptEvent).


Field Summary
 org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher
           
static String MAX_IGNORED_OVER_CAPACITY
          Maximum amount of resources above the target capacity ignored for preemption.
static String MONITORING_INTERVAL
          Time in milliseconds between invocations of this policy
static String NATURAL_TERMINATION_FACTOR
          Given a computed preemption target, account for containers naturally expiring and preempt only this percentage of the delta.
static String OBSERVE_ONLY
          If true, run the policy but do not affect the cluster with preemption and kill events.
static String TOTAL_PREEMPTION_PER_ROUND
          Maximum percentage of resources preempted in a single round.
static String WAIT_TIME_BEFORE_KILL
          Time in milliseconds between requesting a preemption from an application and killing the container.
 
Constructor Summary
ProportionalCapacityPreemptionPolicy()
           
ProportionalCapacityPreemptionPolicy(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher, CapacityScheduler scheduler)
           
ProportionalCapacityPreemptionPolicy(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher, CapacityScheduler scheduler, org.apache.hadoop.yarn.util.Clock clock)
           
 
Method Summary
 void editSchedule()
          This method is invoked at regular intervals.
 long getMonitoringInterval()
           
 String getPolicyName()
           
 void init(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> disp, PreemptableResourceScheduler sched)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

OBSERVE_ONLY

public static final String OBSERVE_ONLY
If true, run the policy but do not affect the cluster with preemption and kill events.

See Also:
Constant Field Values

MONITORING_INTERVAL

public static final String MONITORING_INTERVAL
Time in milliseconds between invocations of this policy

See Also:
Constant Field Values

WAIT_TIME_BEFORE_KILL

public static final String WAIT_TIME_BEFORE_KILL
Time in milliseconds between requesting a preemption from an application and killing the container.

See Also:
Constant Field Values

TOTAL_PREEMPTION_PER_ROUND

public static final String TOTAL_PREEMPTION_PER_ROUND
Maximum percentage of resources preempted in a single round. By controlling this value one can throttle the pace at which containers are reclaimed from the cluster. After computing the total desired preemption, the policy scales it back within this limit.

See Also:
Constant Field Values

MAX_IGNORED_OVER_CAPACITY

public static final String MAX_IGNORED_OVER_CAPACITY
Maximum amount of resources above the target capacity ignored for preemption. This defines a deadzone around the target capacity that helps prevent thrashing and oscillations around the computed target balance. High values would slow the time to capacity and (absent natural completions) it might prevent convergence to guaranteed capacity.

See Also:
Constant Field Values

NATURAL_TERMINATION_FACTOR

public static final String NATURAL_TERMINATION_FACTOR
Given a computed preemption target, account for containers naturally expiring and preempt only this percentage of the delta. This determines the rate of geometric convergence into the deadzone (MAX_IGNORED_OVER_CAPACITY). For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 * WAIT_TIME_BEFORE_KILL, even absent natural termination.

See Also:
Constant Field Values

dispatcher

public org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher
Constructor Detail

ProportionalCapacityPreemptionPolicy

public ProportionalCapacityPreemptionPolicy()

ProportionalCapacityPreemptionPolicy

public ProportionalCapacityPreemptionPolicy(org.apache.hadoop.conf.Configuration config,
                                            org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher,
                                            CapacityScheduler scheduler)

ProportionalCapacityPreemptionPolicy

public ProportionalCapacityPreemptionPolicy(org.apache.hadoop.conf.Configuration config,
                                            org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> dispatcher,
                                            CapacityScheduler scheduler,
                                            org.apache.hadoop.yarn.util.Clock clock)
Method Detail

init

public void init(org.apache.hadoop.conf.Configuration config,
                 org.apache.hadoop.yarn.event.EventHandler<ContainerPreemptEvent> disp,
                 PreemptableResourceScheduler sched)
Specified by:
init in interface SchedulingEditPolicy

editSchedule

public void editSchedule()
Description copied from interface: SchedulingEditPolicy
This method is invoked at regular intervals. Internally the policy is allowed to track containers and affect the scheduler. The "actions" performed are passed back through an EventHandler.

Specified by:
editSchedule in interface SchedulingEditPolicy

getMonitoringInterval

public long getMonitoringInterval()
Specified by:
getMonitoringInterval in interface SchedulingEditPolicy

getPolicyName

public String getPolicyName()
Specified by:
getPolicyName in interface SchedulingEditPolicy


Copyright © 2014 Apache Software Foundation. All Rights Reserved.