Package org.apache.hadoop.yarn.server.nodemanager

Class DefaultContainerExecutor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

WindowsSecureContainerExecutor

public class DefaultContainerExecutor
extends ContainerExecutor

The DefaultContainerExecuter class offers generic container execution services. Process execution is handled in a platform-independent way via ProcessBuilder.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	DefaultContainerExecutor.LocalWrapperScriptBuilder	This class is a utility to create a wrapper script that is platform appropriate.

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

ContainerExecutor.DelayedProcessKiller, ContainerExecutor.ExitCode, ContainerExecutor.Signal

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.fs.FileContext	lfs	A FileContext for the local file system.

Fields inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

DIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, TOKEN_FILE_NAME_FMT, WILDCARD

Constructor Summary

Constructors

Constructor	Description
DefaultContainerExecutor()	Default constructor for use in testing.

Method Summary

Modifier and Type	Method	Description
protected org.apache.hadoop.util.Shell.CommandExecutor	buildCommandExecutor(java.lang.String wrapperScriptPath, java.lang.String containerIdStr, java.lang.String user, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.yarn.api.records.Resource resource, java.io.File workDir, java.util.Map<java.lang.String,java.lang.String> environment, java.lang.String[] numaCommands)	Create a new Shell.ShellCommandExecutor using the parameters.
void	clearLogDirPermissions()	Clear the internal variable for repeatable testing.
static boolean	containerIsAlive(java.lang.String pid)	Returns true if the process with the specified pid is alive.
protected void	copyFile(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, java.lang.String owner)	Copy a file using the lfs FileContext.
protected ContainerLocalizer	createContainerLocalizer(java.lang.String user, java.lang.String appId, java.lang.String locId, java.lang.String tokenFileName, java.util.List<java.lang.String> localDirs, org.apache.hadoop.fs.FileContext localizerFc)	Create a new ContainerLocalizer instance.
protected void	createDir(org.apache.hadoop.fs.Path dirPath, org.apache.hadoop.fs.permission.FsPermission perms, boolean createParent, java.lang.String user)	Use the lfs FileContext to create the target directory.
void	deleteAsUser(DeletionAsUserContext ctx)	Delete specified directories as a given user.
org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair	execContainer(ContainerExecContext ctx)	Perform interactive docker command into running container.
protected DefaultContainerExecutor.LocalWrapperScriptBuilder	getLocalWrapperScriptBuilder(java.lang.String containerIdStr, org.apache.hadoop.fs.Path containerWorkDir)	Create a DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID and path that is appropriate to the current platform.
java.lang.String	getLogDirPermissions()	Return the default container log directory permissions.
protected org.apache.hadoop.fs.Path	getWorkingDir(java.util.List<java.lang.String> localDirs, java.lang.String user, java.lang.String appId)	Return a randomly chosen application directory from a list of local storage directories.
void	init(Context nmContext)	Run the executor initialization steps.
boolean	isContainerAlive(ContainerLivenessContext ctx)	Check if a container is alive.
protected void	killContainer(java.lang.String pid, ContainerExecutor.Signal signal)	Send a specified signal to the specified pid
int	launchContainer(ContainerStartContext ctx)	Launch the container on the node.
void	postComplete(org.apache.hadoop.yarn.api.records.ContainerId containerId)	clean up and release of resources.
int	reacquireContainer(ContainerReacquisitionContext ctx)	Recover an already existing container.
boolean	reapContainer(ContainerReapContext ctx)	No-op for reaping containers within the DefaultContainerExecutor.
int	relaunchContainer(ContainerStartContext ctx)	Relaunch the container on the node.
void	setNumactl(java.lang.String numactl)
void	setNumaResourceAllocator(NumaResourceAllocator numaResourceAllocator)
protected void	setScriptExecutable(org.apache.hadoop.fs.Path script, java.lang.String owner)	Make a file executable using the lfs FileContext.
boolean	signalContainer(ContainerSignalContext ctx)	Signal container with the specified signal.
void	startLocalizer(LocalizerStartContext ctx)	Prepare the environment for containers in this application to execute.
void	symLink(java.lang.String target, java.lang.String symlink)	Create a symlink file which points to the target.
void	updateYarnSysFS(Context ctx, java.lang.String user, java.lang.String appId, java.lang.String spec)	Update cluster information inside container.

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

activateContainer, cleanupBeforeRelaunch, deactivateContainer, getConf, getExposedPorts, getIpAndHost, getLocalIpAndHost, getLocalResources, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, localizeClasspathJar, logOutput, pauseContainer, prepareContainer, readDirAsUser, resumeContainer, setConf, start, stop, writeLaunchEnv, writeLaunchEnv

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

lfs

protected final org.apache.hadoop.fs.FileContext lfs

A FileContext for the local file system.

Constructor Detail

DefaultContainerExecutor

@VisibleForTesting
public DefaultContainerExecutor()

Default constructor for use in testing.

Method Detail

copyFile

protected void copyFile(org.apache.hadoop.fs.Path src,
                        org.apache.hadoop.fs.Path dst,
                        java.lang.String owner)
                 throws java.io.IOException

Copy a file using the lfs FileContext.

Parameters:

src - the file to copy

dst - where to copy the file

owner - the owner of the new copy. Used only in secure Windows clusters

Throws:

java.io.IOException - when the copy fails

See Also:

WindowsSecureContainerExecutor

setScriptExecutable

protected void setScriptExecutable(org.apache.hadoop.fs.Path script,
                                   java.lang.String owner)
                            throws java.io.IOException

Make a file executable using the lfs FileContext.

Parameters:

script - the path to make executable

owner - the new owner for the file. Used only in secure Windows clusters

Throws:

java.io.IOException - when the change mode operation fails

See Also:

WindowsSecureContainerExecutor

init

public void init(Context nmContext)
          throws java.io.IOException

Description copied from class: ContainerExecutor

Run the executor initialization steps. Verify that the necessary configs and permissions are in place.

Specified by:

init in class ContainerExecutor

Parameters:

nmContext - Context of NM

Throws:

java.io.IOException - if initialization fails

startLocalizer

public void startLocalizer(LocalizerStartContext ctx)
                    throws java.io.IOException,
                           java.lang.InterruptedException

Description copied from class: ContainerExecutor

Prepare the environment for containers in this application to execute.

 For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc -> $N/$user/$appId/filecache/idef
 

Specified by:

startLocalizer in class ContainerExecutor

Parameters:

ctx - LocalizerStartContext that encapsulates necessary information for starting a localizer.

Throws:

java.io.IOException - for most application init failures

java.lang.InterruptedException - if application init thread is halted by NM

createContainerLocalizer

@Private
@VisibleForTesting
protected ContainerLocalizer createContainerLocalizer(java.lang.String user,
                                                      java.lang.String appId,
                                                      java.lang.String locId,
                                                      java.lang.String tokenFileName,
                                                      java.util.List<java.lang.String> localDirs,
                                                      org.apache.hadoop.fs.FileContext localizerFc)
                                               throws java.io.IOException

Create a new ContainerLocalizer instance.

Parameters:

user - the user who owns the job for which the localization is being run

appId - the ID of the application for which the localization is being run

locId - the ID of the container for which the localization is being run

localDirs - a list of directories to use as destinations for the localization

localizerFc - the FileContext to use when localizing files

Returns:

the new ContainerLocalizer instance

Throws:

java.io.IOException - if user or locId is null or if the container localizer has an initialization failure

launchContainer

public int launchContainer(ContainerStartContext ctx)
                    throws java.io.IOException,
                           org.apache.hadoop.yarn.exceptions.ConfigurationException

Description copied from class: ContainerExecutor

Launch the container on the node. This is a blocking call and returns only when the container exits.

Specified by:

launchContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for launching containers.

Returns:

the return status of the launch

Throws:

java.io.IOException - if the container launch fails

org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

relaunchContainer

public int relaunchContainer(ContainerStartContext ctx)
                      throws java.io.IOException,
                             org.apache.hadoop.yarn.exceptions.ConfigurationException

Description copied from class: ContainerExecutor

Relaunch the container on the node. This is a blocking call and returns only when the container exits.

Specified by:

relaunchContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for relaunching containers.

Returns:

the return status of the relaunch

Throws:

java.io.IOException - if the container relaunch fails

org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

buildCommandExecutor

protected org.apache.hadoop.util.Shell.CommandExecutor buildCommandExecutor(java.lang.String wrapperScriptPath,
                                                                            java.lang.String containerIdStr,
                                                                            java.lang.String user,
                                                                            org.apache.hadoop.fs.Path pidFile,
                                                                            org.apache.hadoop.yarn.api.records.Resource resource,
                                                                            java.io.File workDir,
                                                                            java.util.Map<java.lang.String,java.lang.String> environment,
                                                                            java.lang.String[] numaCommands)

Create a new Shell.ShellCommandExecutor using the parameters.

Parameters:

wrapperScriptPath - the path to the script to execute

containerIdStr - the container ID

user - the application owner's username

pidFile - the path to the container's PID file

resource - this parameter controls memory and CPU limits.

workDir - If not-null, specifies the directory which should be set as the current working directory for the command. If null, the current working directory is not modified.

environment - the container environment

numaCommands - list of prefix numa commands

Returns:

the new Shell.ShellCommandExecutor

See Also:

Shell.ShellCommandExecutor

getLocalWrapperScriptBuilder

protected DefaultContainerExecutor.LocalWrapperScriptBuilder getLocalWrapperScriptBuilder(java.lang.String containerIdStr,
                                                                                          org.apache.hadoop.fs.Path containerWorkDir)

Create a DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID and path that is appropriate to the current platform.

Parameters:

containerIdStr - the container ID

containerWorkDir - the container's working directory

Returns:

a new DefaultContainerExecutor.LocalWrapperScriptBuilder

signalContainer

public boolean signalContainer(ContainerSignalContext ctx)
                        throws java.io.IOException

Description copied from class: ContainerExecutor

Signal container with the specified signal.

Specified by:

signalContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for signaling containers.

Returns:

returns true if the operation succeeded

Throws:

java.io.IOException - if signaling the container fails

reapContainer

public boolean reapContainer(ContainerReapContext ctx)

No-op for reaping containers within the DefaultContainerExecutor.

Specified by:

reapContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for reaping containers.

Returns:

true given no operations are needed.

isContainerAlive

public boolean isContainerAlive(ContainerLivenessContext ctx)
                         throws java.io.IOException

Description copied from class: ContainerExecutor

Check if a container is alive.

Specified by:

isContainerAlive in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for container liveness check.

Returns:

true if container is still alive

Throws:

java.io.IOException - if there is a failure while checking the container status

containerIsAlive

@VisibleForTesting
public static boolean containerIsAlive(java.lang.String pid)
                                throws java.io.IOException

Returns true if the process with the specified pid is alive.

Parameters:

pid - String pid

Returns:

boolean true if the process is alive

Throws:

java.io.IOException - if the command to test process liveliness fails

killContainer

protected void killContainer(java.lang.String pid,
                             ContainerExecutor.Signal signal)
                      throws java.io.IOException

Send a specified signal to the specified pid

Parameters:

pid - the pid of the process [group] to signal.

signal - signal to send

Throws:

java.io.IOException - if the command to kill the process fails

deleteAsUser

public void deleteAsUser(DeletionAsUserContext ctx)
                  throws java.io.IOException,
                         java.lang.InterruptedException

Description copied from class: ContainerExecutor

Delete specified directories as a given user.

Specified by:

deleteAsUser in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for deletion.

Throws:

java.io.IOException - if delete fails

java.lang.InterruptedException - if interrupted while waiting for the deletion operation to complete

symLink

public void symLink(java.lang.String target,
                    java.lang.String symlink)
             throws java.io.IOException

Description copied from class: ContainerExecutor

Create a symlink file which points to the target.

Specified by:

symLink in class ContainerExecutor

Parameters:

target - The target for symlink

symlink - the symlink file

Throws:

java.io.IOException - Error when creating symlinks

getWorkingDir

protected org.apache.hadoop.fs.Path getWorkingDir(java.util.List<java.lang.String> localDirs,
                                                  java.lang.String user,
                                                  java.lang.String appId)
                                           throws java.io.IOException

Return a randomly chosen application directory from a list of local storage directories. The probability of selecting a directory is proportional to its size.

Parameters:

localDirs - the target directories from which to select

user - the user who owns the application

appId - the application ID

Returns:

the selected directory

Throws:

java.io.IOException - if no application directories for the user can be found

createDir

protected void createDir(org.apache.hadoop.fs.Path dirPath,
                         org.apache.hadoop.fs.permission.FsPermission perms,
                         boolean createParent,
                         java.lang.String user)
                  throws java.io.IOException

Use the lfs FileContext to create the target directory.

Parameters:

dirPath - the target directory

perms - the target permissions for the target directory

createParent - whether the parent directories should also be created

user - the user as whom the target directory should be created. Used only on secure Windows hosts.

Throws:

java.io.IOException - if there's a failure performing a file operation

See Also:

WindowsSecureContainerExecutor

getLogDirPermissions

@VisibleForTesting
public java.lang.String getLogDirPermissions()

Return the default container log directory permissions.

Returns:

the default container log directory permissions

clearLogDirPermissions

@VisibleForTesting
public void clearLogDirPermissions()

Clear the internal variable for repeatable testing.

execContainer

public org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx)
                                                                        throws ContainerExecutionException

Description copied from class: ContainerExecutor

Perform interactive docker command into running container.

Specified by:

execContainer in class ContainerExecutor

Parameters:

ctx - Encapsulates information necessary for exec containers.

Returns:

the input/output stream of interactive docker shell.

Throws:

ContainerExecutionException

updateYarnSysFS

public void updateYarnSysFS(Context ctx,
                            java.lang.String user,
                            java.lang.String appId,
                            java.lang.String spec)
                     throws java.io.IOException

Description copied from class: ContainerExecutor

Update cluster information inside container.

Specified by:

updateYarnSysFS in class ContainerExecutor

Parameters:

ctx - ContainerRuntimeContext

user - Owner of application

appId - YARN application ID

spec - Service Specification

Throws:

java.io.IOException - if there is a failure while writing spec to disk

reacquireContainer

public int reacquireContainer(ContainerReacquisitionContext ctx)
                       throws java.io.IOException,
                              java.lang.InterruptedException

Description copied from class: ContainerExecutor

Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.

Overrides:

reacquireContainer in class ContainerExecutor

Parameters:

ctx - encapsulates information necessary to reacquire container

Returns:

The exit code of the pre-existing container

Throws:

java.io.IOException - if there is a failure while reacquiring the container

java.lang.InterruptedException - if interrupted while waiting to reacquire the container

postComplete

public void postComplete(org.apache.hadoop.yarn.api.records.ContainerId containerId)

clean up and release of resources.

Parameters:

containerId - containerId of running container

setNumaResourceAllocator

@VisibleForTesting
public void setNumaResourceAllocator(NumaResourceAllocator numaResourceAllocator)

setNumactl

@VisibleForTesting
public void setNumactl(java.lang.String numactl)