Package org.apache.hadoop.yarn.applications.distributedshell

Class ApplicationMaster

java.lang.Object

org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster

@Public
@Unstable
public class ApplicationMaster
extends java.lang.Object

An ApplicationMaster for executing shell commands on a set of launched containers using the YARN framework.

This class is meant to act as an example on how to write yarn-based application masters.

The ApplicationMaster is started on a container by the ResourceManager's launcher. The first thing that the ApplicationMaster needs to do is to connect and register itself with the ResourceManager. The registration sets up information within the ResourceManager regarding what host:port the ApplicationMaster is listening on to provide any form of functionality to a client as well as a tracking url that a client can use to keep track of status/job history if needed. However, in the distributedshell, trackingurl and appMasterHost:appMasterRpcPort are not supported.

The ApplicationMaster needs to send a heartbeat to the ResourceManager at regular intervals to inform the ResourceManager that it is up and alive. The ApplicationMasterProtocol.allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest) to the ResourceManager from the ApplicationMaster acts as a heartbeat.

For the actual handling of the job, the ApplicationMaster has to request the ResourceManager via AllocateRequest for the required no. of containers using ResourceRequest with the necessary resource specifications such as node location, computational (memory/disk/cpu) resource requirements. The ResourceManager responds with an AllocateResponse that informs the ApplicationMaster of the set of newly allocated containers, completed containers as well as current state of available resources.

For each allocated container, the ApplicationMaster can then set up the necessary launch context via ContainerLaunchContext to specify the allocated container id, local resources required by the executable, the environment to be setup for the executable, commands to execute, etc. and submit a StartContainerRequest to the ContainerManagementProtocol to launch and execute the defined commands on the given allocated container.

The ApplicationMaster can monitor the launched container by either querying the ResourceManager using ApplicationMasterProtocol.allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest) to get updates on completed containers or via the ContainerManagementProtocol by querying for the status of the allocated container's ContainerId.

After the job has been completed, the ApplicationMaster has to send a FinishApplicationMasterRequest to the ResourceManager to inform it that the ApplicationMaster has been completed.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ApplicationMaster.DSEntity
static class	ApplicationMaster.DSEvent

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.yarn.api.records.ApplicationAttemptId	appAttemptID
protected java.util.Set<org.apache.hadoop.yarn.api.records.ContainerId>	launchedContainers
protected java.util.concurrent.atomic.AtomicInteger	numAllocatedContainers
protected java.util.concurrent.atomic.AtomicInteger	numIgnore
protected java.util.concurrent.atomic.AtomicInteger	numRequestedContainers
protected int	numTotalContainers
protected java.util.concurrent.atomic.AtomicInteger	totalRetries

Constructor Summary

Constructors

Constructor	Description
ApplicationMaster()

Method Summary

Modifier and Type	Method	Description
protected void	cleanup()
protected boolean	finish()
static java.lang.String	getRelativePath(java.lang.String appName, java.lang.String appId, java.lang.String fileDstPath)
boolean	init(java.lang.String[] args)	Parse command line options
static void	main(java.lang.String[] args)
void	run()	Main run function for the application master

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

appAttemptID

@VisibleForTesting
protected org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptID

numTotalContainers

@VisibleForTesting
protected int numTotalContainers

numAllocatedContainers

@VisibleForTesting
protected java.util.concurrent.atomic.AtomicInteger numAllocatedContainers

numRequestedContainers

@VisibleForTesting
protected java.util.concurrent.atomic.AtomicInteger numRequestedContainers

numIgnore

protected java.util.concurrent.atomic.AtomicInteger numIgnore

totalRetries

protected java.util.concurrent.atomic.AtomicInteger totalRetries

launchedContainers

@VisibleForTesting
protected final java.util.Set<org.apache.hadoop.yarn.api.records.ContainerId> launchedContainers

Constructor Detail

ApplicationMaster

public ApplicationMaster()

Method Detail

main

public static void main(java.lang.String[] args)

Parameters:

args - Command line args

init

public boolean init(java.lang.String[] args)
             throws org.apache.commons.cli.ParseException,
                    java.io.IOException

Parse command line options

Parameters:

args - Command line args

Returns:

Whether init successful and run should be invoked

Throws:

org.apache.commons.cli.ParseException

java.io.IOException

cleanup

protected void cleanup()

run

public void run()
         throws org.apache.hadoop.yarn.exceptions.YarnException,
                java.io.IOException,
                java.lang.InterruptedException

Main run function for the application master

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

java.io.IOException

java.lang.InterruptedException

finish

@VisibleForTesting
protected boolean finish()

getRelativePath

public static java.lang.String getRelativePath(java.lang.String appName,
                                               java.lang.String appId,
                                               java.lang.String fileDstPath)