org.apache.hadoop.mapreduce.v2.app
Class MRAppMaster

java.lang.Object
  org.apache.hadoop.service.AbstractService
      org.apache.hadoop.service.CompositeService
          org.apache.hadoop.mapreduce.v2.app.MRAppMaster

All Implemented Interfaces:

Closeable, org.apache.hadoop.service.Service

public class MRAppMaster
extends org.apache.hadoop.service.CompositeService

The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

Nested Class Summary

class

MRAppMaster.RunningAppContext

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

protected org.apache.hadoop.security.UserGroupInformation	currentUser
protected boolean	isLastAMRetry
protected MRAppMetrics	metrics
static int	SHUTDOWN_HOOK_PRIORITY Priority of the MRAppMaster shutdown hook.
protected AtomicBoolean	successfullyUnregistered

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Constructor Summary

MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime, int maxAppAttempts)

MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime, int maxAppAttempts)

Method Summary

void	cleanupStagingDir() clean up staging directories for the job.
protected ClientService	createClientService(AppContext context)
protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent>	createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)
protected ContainerAllocator	createContainerAllocator(ClientService clientService, AppContext context)
protected ContainerLauncher	createContainerLauncher(AppContext context)
protected org.apache.hadoop.yarn.event.Dispatcher	createDispatcher()
protected Job	createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic) Create and initialize (but don't start) a single job.
protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent>	createJobFinishEventHandler() create an event handler that handles the job finish event.
protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>	createJobHistoryHandler(AppContext context)
protected Speculator	createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)
protected org.apache.hadoop.service.AbstractService	createStagingDirCleaningService()
protected TaskAttemptListener	createTaskAttemptListener(AppContext context)
List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo>	getAllAMInfos()
org.apache.hadoop.yarn.api.records.ApplicationId	getAppID()
org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptID()
org.apache.hadoop.mapreduce.OutputCommitter	getCommitter()
Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo>	getCompletedTaskFromPreviousRun()
ContainerAllocator	getContainerAllocator()
ContainerLauncher	getContainerLauncher()
AppContext	getContext()
protected org.apache.hadoop.security.Credentials	getCredentials()
org.apache.hadoop.yarn.event.Dispatcher	getDispatcher()
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration conf) Create the default file System for this job.
org.apache.hadoop.mapreduce.v2.api.records.JobId	getJobId()
protected RMHeartbeatHandler	getRMHeartbeatHandler()
int	getStartCount()
TaskAttemptListener	getTaskAttemptListener()
protected static void	initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName)
protected void	initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf) Obtain the tokens needed by the job and put them in the UGI
Boolean	isLastAMRetry()
boolean	isNewApiCommitter()
protected boolean	keepJobFiles(org.apache.hadoop.mapred.JobConf conf)
static void	main(String[] args)
void	notifyIsLastAMRetry(boolean isLastAMRetry)
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
void	shutDownJob()
protected void	startJobs() This can be overridden to instantiate multiple jobs and create a workflow.
protected void	sysexit() Exit call.

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService, serviceStop

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the MRAppMaster shutdown hook.

See Also:

Constant Field Values

metrics

protected final MRAppMetrics metrics

currentUser

protected org.apache.hadoop.security.UserGroupInformation currentUser

isLastAMRetry

protected volatile boolean isLastAMRetry

successfullyUnregistered

protected AtomicBoolean successfullyUnregistered

Constructor Detail

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                   org.apache.hadoop.yarn.api.records.ContainerId containerId,
                   String nmHost,
                   int nmPort,
                   int nmHttpPort,
                   long appSubmitTime,
                   int maxAppAttempts)

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                   org.apache.hadoop.yarn.api.records.ContainerId containerId,
                   String nmHost,
                   int nmPort,
                   int nmHttpPort,
                   org.apache.hadoop.yarn.util.Clock clock,
                   long appSubmitTime,
                   int maxAppAttempts)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

createDispatcher

protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

keepJobFiles

protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf)

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Create the default file System for this job.

Parameters:

conf - the conf object

Returns:

the default filesystem for this job

Throws:

IOException

getCredentials

protected org.apache.hadoop.security.Credentials getCredentials()

cleanupStagingDir

public void cleanupStagingDir()
                       throws IOException

clean up staging directories for the job.

Throws:

IOException

sysexit

protected void sysexit()

Exit call. Just in a function call to enable testing.

shutDownJob

public void shutDownJob()

createJobFinishEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()

create an event handler that handles the job finish event.

Returns:

the job finish event handler.

createJob

protected Job createJob(org.apache.hadoop.conf.Configuration conf,
                        JobStateInternal forcedState,
                        String diagnostic)

Create and initialize (but don't start) a single job.

Parameters:

forcedState - a state to force the job into or null for normal operation.

diagnostic - a diagnostic message to include with the job.

initJobCredentialsAndUGI

protected void initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)

Obtain the tokens needed by the job and put them in the UGI

Parameters:

conf -

createJobHistoryHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

createStagingDirCleaningService

protected org.apache.hadoop.service.AbstractService createStagingDirCleaningService()

createSpeculator

protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf,
                                      AppContext context)

createTaskAttemptListener

protected TaskAttemptListener createTaskAttemptListener(AppContext context)

createCommitterEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context,
                                                                                                org.apache.hadoop.mapreduce.OutputCommitter committer)

createContainerAllocator

protected ContainerAllocator createContainerAllocator(ClientService clientService,
                                                      AppContext context)

getRMHeartbeatHandler

protected RMHeartbeatHandler getRMHeartbeatHandler()

createContainerLauncher

protected ContainerLauncher createContainerLauncher(AppContext context)

createClientService

protected ClientService createClientService(AppContext context)

getAppID

public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

getAttemptID

public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

getJobId

public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

getCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

isNewApiCommitter

public boolean isNewApiCommitter()

getStartCount

public int getStartCount()

getContext

public AppContext getContext()

getDispatcher

public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

getCompletedTaskFromPreviousRun

public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

getAllAMInfos

public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

getContainerAllocator

public ContainerAllocator getContainerAllocator()

getContainerLauncher

public ContainerLauncher getContainerLauncher()

getTaskAttemptListener

public TaskAttemptListener getTaskAttemptListener()

isLastAMRetry

public Boolean isLastAMRetry()

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

Exception

startJobs

protected void startJobs()

This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

main

public static void main(String[] args)

notifyIsLastAMRetry

public void notifyIsLastAMRetry(boolean isLastAMRetry)

initAndStartAppMaster

protected static void initAndStartAppMaster(MRAppMaster appMaster,
                                            org.apache.hadoop.mapred.JobConf conf,
                                            String jobUserName)
                                     throws IOException,
                                            InterruptedException

Throws:

IOException

InterruptedException