Package org.apache.hadoop.yarn.server.nodemanager.amrmproxy

Class FederationInterceptor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AbstractRequestInterceptor

org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.yarn.api.ApplicationMasterProtocol, org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol, RequestInterceptor

public class FederationInterceptor
extends AbstractRequestInterceptor

Extends the AbstractRequestInterceptor and provides an implementation for federation of YARN RM and scaling an application across multiple YARN sub-clusters. All the federation specific implementation is encapsulated in this class. This is always the last interceptor in the chain.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	NMSS_CLASS_PREFIX
static java.lang.String	NMSS_REG_REQUEST_KEY
static java.lang.String	NMSS_REG_RESPONSE_KEY
static java.lang.String	NMSS_SECONDARY_SC_PREFIX	When AMRMProxy HA is enabled, secondary AMRMTokens will be stored in Yarn Registry.
static java.lang.String	STRING_TO_BYTE_FORMAT

Constructor Summary

Constructors

Constructor	Description
FederationInterceptor()	Creates an instance of the FederationInterceptor class.

Method Summary

Modifier and Type	Method	Description
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse	allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request)	Sends the heart beats to the home RM and the secondary sub-cluster RMs that are being used by the application.
protected void	cacheAllocatedContainersForSubClusterId(java.util.List<org.apache.hadoop.yarn.api.records.Container> containers, org.apache.hadoop.yarn.server.federation.store.records.SubClusterId subClusterId)
protected void	cleanupRegistry()	Only for unit test cleanup.
protected org.apache.hadoop.yarn.server.AMHeartbeatRequestHandler	createHomeHeartbeatHandler(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.server.AMRMClientRelayer rmProxyRelayer)
protected <T> T	createHomeRMProxy(AMRMProxyApplicationContext appContext, java.lang.Class<T> protocol, org.apache.hadoop.security.UserGroupInformation user)	Create a proxy instance that is used to connect to the Home resource manager.
protected org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager	createUnmanagedAMPoolManager(java.util.concurrent.ExecutorService threadPool)	Create the UAM pool manager for secondary sub-clusters.
org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse	finishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)	Sends the finish application master request to all the resource managers used by the application.
protected org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse	generateBaseAllocationResponse()	Prepare the base allocation response.
java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.List<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse>>	getAsyncResponseSink()
protected org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptId()
protected java.util.Map<org.apache.hadoop.yarn.api.records.ContainerId,org.apache.hadoop.yarn.server.federation.store.records.SubClusterId>	getContainerIdToSubClusterIdMap()
protected org.apache.hadoop.yarn.server.AMHeartbeatRequestHandler	getHomeHeartbeatHandler()
protected org.apache.hadoop.yarn.server.federation.utils.FederationRegistryClient	getRegistryClient()
protected java.util.Set<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId>	getTimedOutSCs(boolean verbose)
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.concurrent.Future<?>>	getUamRegisterFutures()
protected org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager	getUnmanagedAMPool()
protected int	getUnmanagedAMPoolSize()
void	init(AMRMProxyApplicationContext appContext)	Initializes the instance using specified context.
static <T> boolean	isNullOrEmpty(java.util.Collection<T> c)	Utility method to check if the specified Collection is null or empty.
static <T1,T2> boolean	isNullOrEmpty(java.util.Map<T1,T2> c)	Utility method to check if the specified Collection is null or empty.
protected TokenAndRegisterResponse	launchUAMAndRegisterApplicationMaster(org.apache.hadoop.yarn.conf.YarnConfiguration config, java.lang.String subClusterId, org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
protected void	mergeAllocateResponse(org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse homeResponse, org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse otherResponse, org.apache.hadoop.yarn.server.federation.store.records.SubClusterId otherRMAddress)
protected void	reAttachUAMAndMergeRegisterResponse(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse homeResponse, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Try re-attach to all existing and running UAMs in secondary sub-clusters launched by previous application attempts if any.
void	recover(java.util.Map<java.lang.String,byte[]> recoveredDataMap)	Recover RequestInterceptor state from store.
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest request)	Sends the application master's registration request to the home RM.
void	setNextInterceptor(RequestInterceptor next)	Sets the RequestInterceptor in the chain.
void	shutdown()	This is called when the application pipeline is being destroyed.
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest>>	splitResourceRequests(java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> askList)	Splits the specified request to send it to different sub clusters.

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AbstractRequestInterceptor

allocateForDistributedScheduling, getApplicationContext, getConf, getNextInterceptor, getNMStateStore, registerApplicationMasterForDistributedScheduling, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NMSS_CLASS_PREFIX

public static final java.lang.String NMSS_CLASS_PREFIX

See Also:

Constant Field Values

NMSS_REG_REQUEST_KEY

public static final java.lang.String NMSS_REG_REQUEST_KEY

See Also:

Constant Field Values

NMSS_REG_RESPONSE_KEY

public static final java.lang.String NMSS_REG_RESPONSE_KEY

See Also:

Constant Field Values

NMSS_SECONDARY_SC_PREFIX

public static final java.lang.String NMSS_SECONDARY_SC_PREFIX

When AMRMProxy HA is enabled, secondary AMRMTokens will be stored in Yarn Registry. Otherwise, if NM recovery is enabled, the UAM token are stored in local NMSS instead under this directory name.

See Also:

Constant Field Values

STRING_TO_BYTE_FORMAT

public static final java.lang.String STRING_TO_BYTE_FORMAT

See Also:

Constant Field Values

Constructor Detail

FederationInterceptor

public FederationInterceptor()

Creates an instance of the FederationInterceptor class.

Method Detail

init

public void init(AMRMProxyApplicationContext appContext)

Initializes the instance using specified context.

Specified by:

init in interface RequestInterceptor

Overrides:

init in class AbstractRequestInterceptor

Parameters:

appContext - AMRMProxy application context

recover

public void recover(java.util.Map<java.lang.String,byte[]> recoveredDataMap)

Description copied from class: AbstractRequestInterceptor

Recover RequestInterceptor state from store.

Specified by:

recover in interface RequestInterceptor

Overrides:

recover in class AbstractRequestInterceptor

Parameters:

recoveredDataMap - states for all interceptors recovered from NMSS

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest request)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              java.io.IOException

Sends the application master's registration request to the home RM. Between AM and AMRMProxy, FederationInterceptor modifies the RM behavior, so that when AM registers more than once, it returns the same register success response instead of throwing InvalidApplicationMasterRequestException. Furthermore, we present to AM as if we are the RM that never fails over (except when AMRMProxy restarts). When actual RM fails over, we always re-register automatically. We did this because FederationInterceptor can receive concurrent register requests from AM because of timeout between AM and AMRMProxy, which is shorter than the timeout + failOver between FederationInterceptor (AMRMProxy) and RM. For the same reason, this method needs to be synchronized.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

java.io.IOException

allocate

public org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request)
                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                            java.io.IOException

Sends the heart beats to the home RM and the secondary sub-cluster RMs that are being used by the application.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

java.io.IOException

finishApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse finishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)
                                                                                                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                          java.io.IOException

Sends the finish application master request to all the resource managers used by the application.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

java.io.IOException

setNextInterceptor

public void setNextInterceptor(RequestInterceptor next)

Description copied from class: AbstractRequestInterceptor

Sets the RequestInterceptor in the chain.

Specified by:

setNextInterceptor in interface RequestInterceptor

Overrides:

setNextInterceptor in class AbstractRequestInterceptor

Parameters:

next - the next interceptor to set

shutdown

public void shutdown()

This is called when the application pipeline is being destroyed. We will release all the resources that we are holding in this call.

Specified by:

shutdown in interface RequestInterceptor

Overrides:

shutdown in class AbstractRequestInterceptor

cleanupRegistry

@VisibleForTesting
protected void cleanupRegistry()

Only for unit test cleanup.

getRegistryClient

@VisibleForTesting
protected org.apache.hadoop.yarn.server.federation.utils.FederationRegistryClient getRegistryClient()

getAttemptId

@VisibleForTesting
protected org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptId()

getHomeHeartbeatHandler

@VisibleForTesting
protected org.apache.hadoop.yarn.server.AMHeartbeatRequestHandler getHomeHeartbeatHandler()

createUnmanagedAMPoolManager

@VisibleForTesting
protected org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager createUnmanagedAMPoolManager(java.util.concurrent.ExecutorService threadPool)

Create the UAM pool manager for secondary sub-clusters. For unit test to override.

Parameters:

threadPool - the thread pool to use

Returns:

the UAM pool manager instance

createHomeHeartbeatHandler

@VisibleForTesting
protected org.apache.hadoop.yarn.server.AMHeartbeatRequestHandler createHomeHeartbeatHandler(org.apache.hadoop.conf.Configuration conf,
                                                                                             org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                                                             org.apache.hadoop.yarn.server.AMRMClientRelayer rmProxyRelayer)

createHomeRMProxy

protected <T> T createHomeRMProxy(AMRMProxyApplicationContext appContext,
                                  java.lang.Class<T> protocol,
                                  org.apache.hadoop.security.UserGroupInformation user)

Create a proxy instance that is used to connect to the Home resource manager.

Type Parameters:

T - the type of the proxy

Parameters:

appContext - AMRMProxyApplicationContext

protocol - the protocol class for the proxy

user - the ugi for the proxy

Returns:

the proxy created

reAttachUAMAndMergeRegisterResponse

protected void reAttachUAMAndMergeRegisterResponse(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse homeResponse,
                                                   org.apache.hadoop.yarn.api.records.ApplicationId appId)

Try re-attach to all existing and running UAMs in secondary sub-clusters launched by previous application attempts if any. All running containers in the UAMs will be combined into the registerResponse. For the first attempt, the registry will be empty for this application and thus no-op here.

launchUAMAndRegisterApplicationMaster

protected TokenAndRegisterResponse launchUAMAndRegisterApplicationMaster(org.apache.hadoop.yarn.conf.YarnConfiguration config,
                                                                         java.lang.String subClusterId,
                                                                         org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                  throws java.io.IOException,
                                                                         org.apache.hadoop.yarn.exceptions.YarnException

Throws:

java.io.IOException

org.apache.hadoop.yarn.exceptions.YarnException

generateBaseAllocationResponse

protected org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse generateBaseAllocationResponse()

Prepare the base allocation response. Use lastSCResponse and lastHeartbeatTimeStamp to assemble entries about cluster-wide info, e.g. AvailableResource, NumClusterNodes.

mergeAllocateResponse

@VisibleForTesting
protected void mergeAllocateResponse(org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse homeResponse,
                                     org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse otherResponse,
                                     org.apache.hadoop.yarn.server.federation.store.records.SubClusterId otherRMAddress)

getTimedOutSCs

protected java.util.Set<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId> getTimedOutSCs(boolean verbose)

splitResourceRequests

protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest>> splitResourceRequests(java.util.List<org.apache.hadoop.yarn.api.records.ResourceRequest> askList)
                                                                                                                                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException

Splits the specified request to send it to different sub clusters. The splitting algorithm is very simple. If the request does not have a node preference, the policy decides the sub cluster. If the request has a node preference and if locality is required, then it is sent to the sub cluster that contains the requested node. If node preference is specified and locality is not required, then the policy decides the sub cluster.

Parameters:

askList - the ask list to split

Returns:

the split asks

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if split fails

getUnmanagedAMPoolSize

@VisibleForTesting
protected int getUnmanagedAMPoolSize()

getUnmanagedAMPool

@VisibleForTesting
protected org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager getUnmanagedAMPool()

getUamRegisterFutures

@VisibleForTesting
protected java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.concurrent.Future<?>> getUamRegisterFutures()

getAsyncResponseSink

@VisibleForTesting
public java.util.Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,java.util.List<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse>> getAsyncResponseSink()

isNullOrEmpty

public static <T> boolean isNullOrEmpty(java.util.Collection<T> c)

Utility method to check if the specified Collection is null or empty.

Type Parameters:

T - element type of the collection

Parameters:

c - the collection object

Returns:

whether is it is null or empty

isNullOrEmpty

public static <T1,T2> boolean isNullOrEmpty(java.util.Map<T1,T2> c)

Utility method to check if the specified Collection is null or empty.

Type Parameters:

T1 - key type of the map

T2 - value type of the map

Parameters:

c - the map object

Returns:

whether is it is null or empty

cacheAllocatedContainersForSubClusterId

@VisibleForTesting
protected void cacheAllocatedContainersForSubClusterId(java.util.List<org.apache.hadoop.yarn.api.records.Container> containers,
                                                       org.apache.hadoop.yarn.server.federation.store.records.SubClusterId subClusterId)

getContainerIdToSubClusterIdMap

@VisibleForTesting
protected java.util.Map<org.apache.hadoop.yarn.api.records.ContainerId,org.apache.hadoop.yarn.server.federation.store.records.SubClusterId> getContainerIdToSubClusterIdMap()