Package org.apache.hadoop.yarn.server.uam

Class UnmanagedAMPoolManager

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.uam.UnmanagedAMPoolManager

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.service.Service

@Public
@Unstable
public class UnmanagedAMPoolManager
extends org.apache.hadoop.service.AbstractService

A service that manages a pool of UAM managers in UnmanagedApplicationManager.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
UnmanagedAMPoolManager(java.util.concurrent.ExecutorService threadpool)

Method Summary

Modifier and Type	Method	Description
void	allocateAsync(java.lang.String uamId, org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request, org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback)	AllocateAsync to an UAM.
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse>	batchFinishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request, java.lang.String appId)	Complete FinishApplicationMaster interface calls in batches.
java.lang.String	createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest, org.apache.hadoop.conf.Configuration conf, java.lang.String queueName, java.lang.String submitter, java.lang.String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, java.lang.String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)	Create a new UAM and register the application, without specifying uamId and appId.
protected UnmanagedApplicationManager	createUAM(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String queueName, java.lang.String submitter, java.lang.String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, java.lang.String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)	Creates the UAM instance.
void	drainUAMHeartbeats()
org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse	finishApplicationMaster(java.lang.String uamId, org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)	Finish an UAM/application.
java.util.Set<java.lang.String>	getAllUAMIds()	Get the id of all running UAMs.
AMRMClientRelayer	getAMRMClientRelayer(java.lang.String uamId)	Return the rmProxy relayer of an UAM.
protected java.lang.Thread	getFinishApplicationThread()
int	getRequestQueueSize(java.lang.String uamId)
protected java.util.Map<java.lang.String,UnmanagedApplicationManager>	getUnmanagedAppMasterMap()
boolean	hasUAMId(java.lang.String uamId)	Return whether an UAM exists.
org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier>	launchUAM(java.lang.String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String queueName, java.lang.String submitter, java.lang.String appNameSuffix, boolean keepContainersAcrossApplicationAttempts, java.lang.String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)	Launch a new UAM, using the provided uamId and appId.
void	reAttachUAM(java.lang.String uamId, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String queueName, java.lang.String submitter, java.lang.String appNameSuffix, org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken, java.lang.String rmName, org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)	Re-attach to an existing UAM, using the provided uamIdentifier.
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(java.lang.String uamId, org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest)	Register application master for the UAM.
protected void	serviceStart()
protected void	serviceStop()	Normally we should finish all applications before stop.
void	shutDownConnections()	Shutdown all UAM clients without killing them in YarnRM.
void	shutDownConnections(java.lang.String uamId)	Shutdown an UAM client without killing it in YarnRM.
void	unAttachUAM(java.lang.String uamId)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

UnmanagedAMPoolManager

public UnmanagedAMPoolManager(java.util.concurrent.ExecutorService threadpool)

Method Detail

serviceStart

protected void serviceStart()
                     throws java.lang.Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

java.lang.Exception

serviceStop

protected void serviceStop()
                    throws java.lang.Exception

Normally we should finish all applications before stop. If there are still UAMs running, force kill all of them. Do parallel kill because of performance reasons.

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

java.lang.Exception

createAndRegisterNewUAM

public java.lang.String createAndRegisterNewUAM(org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest,
                                                org.apache.hadoop.conf.Configuration conf,
                                                java.lang.String queueName,
                                                java.lang.String submitter,
                                                java.lang.String appNameSuffix,
                                                boolean keepContainersAcrossApplicationAttempts,
                                                java.lang.String rmName,
                                                org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                java.io.IOException

Create a new UAM and register the application, without specifying uamId and appId. We will ask for an appId from RM and use it as the uamId.

Parameters:

registerRequest - RegisterApplicationMasterRequest

conf - configuration for this UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.

rmName - name of the YarnRM

originalAppSubmissionContext - ApplicationSubmissionContext

Returns:

uamId for the UAM

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if registerApplicationMaster fails

java.io.IOException - if registerApplicationMaster fails

See Also:

#setKeepContainersAcrossApplicationAttempts(boolean)

launchUAM

public org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> launchUAM(java.lang.String uamId,
                                                                                                             org.apache.hadoop.conf.Configuration conf,
                                                                                                             org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                                                                             java.lang.String queueName,
                                                                                                             java.lang.String submitter,
                                                                                                             java.lang.String appNameSuffix,
                                                                                                             boolean keepContainersAcrossApplicationAttempts,
                                                                                                             java.lang.String rmName,
                                                                                                             org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
                                                                                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                             java.io.IOException

Launch a new UAM, using the provided uamId and appId.

Parameters:

uamId - uam Id

conf - configuration for this UAM

appId - application id for the UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

keepContainersAcrossApplicationAttempts - keep container flag for UAM recovery.

rmName - name of the YarnRM

originalAppSubmissionContext - AppSubmissionContext

Returns:

UAM token

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

java.io.IOException - if fails

See Also:

#setKeepContainersAcrossApplicationAttempts(boolean)

reAttachUAM

public void reAttachUAM(java.lang.String uamId,
                        org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.yarn.api.records.ApplicationId appId,
                        java.lang.String queueName,
                        java.lang.String submitter,
                        java.lang.String appNameSuffix,
                        org.apache.hadoop.security.token.Token<org.apache.hadoop.yarn.security.AMRMTokenIdentifier> uamToken,
                        java.lang.String rmName,
                        org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        java.io.IOException

Re-attach to an existing UAM, using the provided uamIdentifier.

Parameters:

uamId - uam Id

conf - configuration for this UAM

appId - application id for the UAM

queueName - queue of the application

submitter - submitter name of the UAM

appNameSuffix - application name suffix for the UAM

uamToken - UAM token

rmName - name of the YarnRM

originalAppSubmissionContext - AppSubmissionContext

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

java.io.IOException - if fails

createUAM

@VisibleForTesting
protected UnmanagedApplicationManager createUAM(org.apache.hadoop.conf.Configuration conf,
                                                org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                java.lang.String queueName,
                                                java.lang.String submitter,
                                                java.lang.String appNameSuffix,
                                                boolean keepContainersAcrossApplicationAttempts,
                                                java.lang.String rmName,
                                                org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext originalAppSubmissionContext)

Creates the UAM instance. Pull out to make unit test easy.

Parameters:

conf - Configuration

appId - application id

queueName - queue of the application

submitter - submitter name of the application

appNameSuffix - application name suffix

keepContainersAcrossApplicationAttempts - keep container flag for UAM

rmName - name of the YarnRM

originalAppSubmissionContext - ApplicationSubmissionContext

Returns:

the UAM instance

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(java.lang.String uamId,
                                                                                                              org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest registerRequest)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              java.io.IOException

Register application master for the UAM.

Parameters:

uamId - uam Id

registerRequest - RegisterApplicationMasterRequest

Returns:

register response

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if register fails

java.io.IOException - if register fails

allocateAsync

public void allocateAsync(java.lang.String uamId,
                          org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest request,
                          org.apache.hadoop.yarn.util.AsyncCallback<org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse> callback)
                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                          java.io.IOException

AllocateAsync to an UAM.

Parameters:

uamId - uam Id

request - AllocateRequest

callback - callback for response

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if allocate fails

java.io.IOException - if allocate fails

finishApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse finishApplicationMaster(java.lang.String uamId,
                                                                                                          org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request)
                                                                                                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                          java.io.IOException

Finish an UAM/application.

Parameters:

uamId - uam Id

request - FinishApplicationMasterRequest

Returns:

FinishApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if finishApplicationMaster call fails

java.io.IOException - if finishApplicationMaster call fails

shutDownConnections

public void shutDownConnections(java.lang.String uamId)
                         throws org.apache.hadoop.yarn.exceptions.YarnException

Shutdown an UAM client without killing it in YarnRM.

Parameters:

uamId - uam Id

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

shutDownConnections

public void shutDownConnections()
                         throws org.apache.hadoop.yarn.exceptions.YarnException

Shutdown all UAM clients without killing them in YarnRM.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

getAllUAMIds

public java.util.Set<java.lang.String> getAllUAMIds()

Get the id of all running UAMs.

Returns:

uamId set

hasUAMId

public boolean hasUAMId(java.lang.String uamId)

Return whether an UAM exists.

Parameters:

uamId - uam Id

Returns:

UAM exists or not

getAMRMClientRelayer

public AMRMClientRelayer getAMRMClientRelayer(java.lang.String uamId)
                                       throws org.apache.hadoop.yarn.exceptions.YarnException

Return the rmProxy relayer of an UAM.

Parameters:

uamId - uam Id

Returns:

the rmProxy relayer

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if fails

getRequestQueueSize

@VisibleForTesting
public int getRequestQueueSize(java.lang.String uamId)
                        throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

drainUAMHeartbeats

@VisibleForTesting
public void drainUAMHeartbeats()

batchFinishApplicationMaster

public java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse> batchFinishApplicationMaster(org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest request,
                                                                                                                                                     java.lang.String appId)

Complete FinishApplicationMaster interface calls in batches.

Parameters:

request - FinishApplicationMasterRequest

appId - application Id

Returns:

Returns the Map, the key is subClusterId, the value is FinishApplicationMasterResponse

unAttachUAM

public void unAttachUAM(java.lang.String uamId)

getUnmanagedAppMasterMap

@VisibleForTesting
protected java.util.Map<java.lang.String,UnmanagedApplicationManager> getUnmanagedAppMasterMap()

getFinishApplicationThread

@VisibleForTesting
protected java.lang.Thread getFinishApplicationThread()