Package org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator

Class LoadBasedGlobalPolicy

java.lang.Object

org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.GlobalPolicy

org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.LoadBasedGlobalPolicy

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class LoadBasedGlobalPolicy
extends GlobalPolicy

Load based policy that generates weighted policies by scaling the cluster load (based on pending) to a weight from 0.0 to 1.0.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LoadBasedGlobalPolicy.Scaling

Constructor Summary

Constructors

Constructor	Description
LoadBasedGlobalPolicy()

Method Summary

Modifier and Type	Method	Description
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo>	getSubClustersMetricsInfo(java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> subClusterMetricsInfo)	Get the ClusterMetric information of the subCluster.
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo,java.lang.Float>	getTargetWeights(java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> clusterMetrics)	Get subCluster target weight.
protected float	getWeightByScaling(int maxPendingVal, int curPendingVal)	Get weight information.
protected org.apache.hadoop.yarn.server.federation.policies.manager.WeightedLocalityPolicyManager	getWeightedLocalityPolicyManager(java.lang.String queue, java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> subClusterMetricInfos)	GPG can help update the policy of the queue.
protected java.util.Map<java.lang.Class<?>,java.lang.String>	registerPaths()	Return a map of the object type and RM path to request it from - the framework will query these paths and provide the objects to the policy.
void	setConf(org.apache.hadoop.conf.Configuration conf)
protected org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager	updatePolicy(java.lang.String queueName, java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> clusterInfo, org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager currentManager)	Update the policy of the queue.

Methods inherited from class org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.GlobalPolicy

getConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LoadBasedGlobalPolicy

public LoadBasedGlobalPolicy()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class GlobalPolicy

registerPaths

protected java.util.Map<java.lang.Class<?>,java.lang.String> registerPaths()

Description copied from class: GlobalPolicy

Return a map of the object type and RM path to request it from - the framework will query these paths and provide the objects to the policy. Delegating this responsibility to the PolicyGenerator enables us to avoid duplicate calls to the same * endpoints as the GlobalPolicy is invoked once per queue.

Overrides:

registerPaths in class GlobalPolicy

Returns:

a map of the object type and RM path.

updatePolicy

protected org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager updatePolicy(java.lang.String queueName,
                                                                                                         java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> clusterInfo,
                                                                                                         org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager currentManager)

Update the policy of the queue.

Specified by:

updatePolicy in class GlobalPolicy

Parameters:

queueName - name of the queue

clusterInfo - subClusterId map to cluster information about the SubCluster used to make policy decisions

currentManager - the FederationPolicyManager for the queue's existing policy the manager may be null, in which case the policy will need to be created.

Returns:

FederationPolicyManager.

getWeightedLocalityPolicyManager

protected org.apache.hadoop.yarn.server.federation.policies.manager.WeightedLocalityPolicyManager getWeightedLocalityPolicyManager(java.lang.String queue,
                                                                                                                                   java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> subClusterMetricInfos)

GPG can help update the policy of the queue. We automatically generate the weight of the subCluster according to the clusterMetrics of the subCluster.

Parameters:

queue - queueName.

subClusterMetricInfos - Metric information of the subCluster.

Returns:

WeightedLocalityPolicyManager.

getSubClustersMetricsInfo

protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> getSubClustersMetricsInfo(java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.Map<java.lang.Class,java.lang.Object>> subClusterMetricsInfo)

Get the ClusterMetric information of the subCluster.

Parameters:

subClusterMetricsInfo - subCluster Metric Information.

Returns:

Mapping relationship between subCluster and Metric.

getTargetWeights

@VisibleForTesting
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo,java.lang.Float> getTargetWeights(java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> clusterMetrics)

Get subCluster target weight.

Parameters:

clusterMetrics - Metric of the subCluster.

Returns:

subCluster Weights.

getWeightByScaling

protected float getWeightByScaling(int maxPendingVal,
                                   int curPendingVal)

Get weight information. We will calculate the weight information according to different Scaling. NONE: No calculation is required, and the weight is 1 at this time. LINEAR: For linear computation, we will use (maxPendingVal - curPendingVal) / (maxPendingVal). QUADRATIC: Calculated using quadratic, We will calculate quadratic for maxPendingVal, curPendingVal, then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). LOG(LOGARITHM): Calculated using logarithm, We will calculate logarithm for maxPendingVal, curPendingVal, then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal).

Parameters:

maxPendingVal - maxPending - minPending

curPendingVal - pending - minPending

Returns:

Calculated weight information.