Package org.apache.hadoop.yarn.server.nodemanager

Class WindowsSecureContainerExecutor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor

org.apache.hadoop.yarn.server.nodemanager.WindowsSecureContainerExecutor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class WindowsSecureContainerExecutor
extends DefaultContainerExecutor

Windows secure container executor (WSCE). This class offers a secure container executor on Windows, similar to the LinuxContainerExecutor. As the NM does not run on a high privileged context, this class delegates elevated operations to the helper hadoopwintuilsvc, implemented by the winutils.exe running as a service. JNI and LRPC is used to communicate with the privileged service.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor

DefaultContainerExecutor.LocalWrapperScriptBuilder

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

ContainerExecutor.DelayedProcessKiller, ContainerExecutor.ExitCode, ContainerExecutor.Signal

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	LOCALIZER_PID_FORMAT

Fields inherited from class org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor

lfs

Fields inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

DIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, TOKEN_FILE_NAME_FMT, WILDCARD

Constructor Summary

Constructors

Constructor	Description
WindowsSecureContainerExecutor()

Method Summary

Modifier and Type	Method	Description
protected org.apache.hadoop.util.Shell.CommandExecutor	buildCommandExecutor(java.lang.String wrapperScriptPath, java.lang.String containerIdStr, java.lang.String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.yarn.api.records.Resource resource, java.io.File wordDir, java.util.Map<java.lang.String,java.lang.String> environment, java.lang.String[] numaCommands)	Create a new Shell.ShellCommandExecutor using the parameters.
protected void	copyFile(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, java.lang.String owner)	Copy a file using the DefaultContainerExecutor.lfs FileContext.
protected void	createDir(org.apache.hadoop.fs.Path dirPath, org.apache.hadoop.fs.permission.FsPermission perms, boolean createParent, java.lang.String owner)	Use the DefaultContainerExecutor.lfs FileContext to create the target directory.
protected DefaultContainerExecutor.LocalWrapperScriptBuilder	getLocalWrapperScriptBuilder(java.lang.String containerIdStr, org.apache.hadoop.fs.Path containerWorkDir)	Create a DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID and path that is appropriate to the current platform.
protected java.lang.String[]	getRunCommand(java.lang.String command, java.lang.String groupId, java.lang.String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration conf)	Return a command line to execute the given command in the OS shell.
protected void	killContainer(java.lang.String pid, ContainerExecutor.Signal signal)	Send a specified signal to the specified pid
org.apache.hadoop.fs.Path	localizeClasspathJar(org.apache.hadoop.fs.Path jarPath, org.apache.hadoop.fs.Path target, java.lang.String owner)	This function localizes the JAR file on-demand.
void	setConf(org.apache.hadoop.conf.Configuration conf)
protected void	setScriptExecutable(org.apache.hadoop.fs.Path script, java.lang.String owner)	Make a file executable using the DefaultContainerExecutor.lfs FileContext.
void	startLocalizer(LocalizerStartContext ctx)	Prepare the environment for containers in this application to execute.

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor

clearLogDirPermissions, containerIsAlive, createContainerLocalizer, deleteAsUser, execContainer, getLogDirPermissions, getWorkingDir, init, isContainerAlive, launchContainer, postComplete, reacquireContainer, reapContainer, relaunchContainer, setNumactl, setNumaResourceAllocator, signalContainer, symLink, updateYarnSysFS

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

activateContainer, cleanupBeforeRelaunch, deactivateContainer, getConf, getExposedPorts, getIpAndHost, getLocalIpAndHost, getLocalResources, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, logOutput, pauseContainer, prepareContainer, readDirAsUser, resumeContainer, start, stop, writeLaunchEnv, writeLaunchEnv

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOCALIZER_PID_FORMAT

public static final java.lang.String LOCALIZER_PID_FORMAT

See Also:

Constant Field Values

Constructor Detail

WindowsSecureContainerExecutor

public WindowsSecureContainerExecutor()
                               throws java.io.IOException,
                                      java.net.URISyntaxException

Throws:

java.io.IOException

java.net.URISyntaxException

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class ContainerExecutor

getRunCommand

protected java.lang.String[] getRunCommand(java.lang.String command,
                                           java.lang.String groupId,
                                           java.lang.String userName,
                                           org.apache.hadoop.fs.Path pidFile,
                                           org.apache.hadoop.conf.Configuration conf)

Description copied from class: ContainerExecutor

Return a command line to execute the given command in the OS shell. On Windows, the {code}groupId{code} parameter can be used to launch and associate the given GID with a process group. On non-Windows hosts, the {code}groupId{code} parameter is ignored.

Overrides:

getRunCommand in class ContainerExecutor

Parameters:

command - the command to execute

groupId - the job owner's GID

userName - the job owner's username

pidFile - the path to the container's PID file

conf - the configuration

Returns:

the command line to execute

getLocalWrapperScriptBuilder

protected DefaultContainerExecutor.LocalWrapperScriptBuilder getLocalWrapperScriptBuilder(java.lang.String containerIdStr,
                                                                                          org.apache.hadoop.fs.Path containerWorkDir)

Description copied from class: DefaultContainerExecutor

Create a DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID and path that is appropriate to the current platform.

Overrides:

getLocalWrapperScriptBuilder in class DefaultContainerExecutor

Parameters:

containerIdStr - the container ID

containerWorkDir - the container's working directory

Returns:

a new DefaultContainerExecutor.LocalWrapperScriptBuilder

copyFile

protected void copyFile(org.apache.hadoop.fs.Path src,
                        org.apache.hadoop.fs.Path dst,
                        java.lang.String owner)
                 throws java.io.IOException

Description copied from class: DefaultContainerExecutor

Copy a file using the DefaultContainerExecutor.lfs FileContext.

Overrides:

copyFile in class DefaultContainerExecutor

Parameters:

src - the file to copy

dst - where to copy the file

owner - the owner of the new copy. Used only in secure Windows clusters

Throws:

java.io.IOException - when the copy fails

See Also:

WindowsSecureContainerExecutor

createDir

protected void createDir(org.apache.hadoop.fs.Path dirPath,
                         org.apache.hadoop.fs.permission.FsPermission perms,
                         boolean createParent,
                         java.lang.String owner)
                  throws java.io.IOException

Description copied from class: DefaultContainerExecutor

Use the DefaultContainerExecutor.lfs FileContext to create the target directory.

Overrides:

createDir in class DefaultContainerExecutor

Parameters:

dirPath - the target directory

perms - the target permissions for the target directory

createParent - whether the parent directories should also be created

owner - the user as whom the target directory should be created. Used only on secure Windows hosts.

Throws:

java.io.IOException - if there's a failure performing a file operation

See Also:

WindowsSecureContainerExecutor

setScriptExecutable

protected void setScriptExecutable(org.apache.hadoop.fs.Path script,
                                   java.lang.String owner)
                            throws java.io.IOException

Description copied from class: DefaultContainerExecutor

Make a file executable using the DefaultContainerExecutor.lfs FileContext.

Overrides:

setScriptExecutable in class DefaultContainerExecutor

Parameters:

script - the path to make executable

owner - the new owner for the file. Used only in secure Windows clusters

Throws:

java.io.IOException - when the change mode operation fails

See Also:

WindowsSecureContainerExecutor

localizeClasspathJar

public org.apache.hadoop.fs.Path localizeClasspathJar(org.apache.hadoop.fs.Path jarPath,
                                                      org.apache.hadoop.fs.Path target,
                                                      java.lang.String owner)
                                               throws java.io.IOException

Description copied from class: ContainerExecutor

This function localizes the JAR file on-demand. On Windows the ContainerLaunch creates a temporary special JAR manifest of other JARs to workaround the CLASSPATH length. In a secure cluster this JAR must be localized so that the container has access to it. The default implementation returns the classpath passed to it, which is expected to have been created in the node manager's fprivate folder, which will not work with secure Windows clusters.

Overrides:

localizeClasspathJar in class ContainerExecutor

Parameters:

jarPath - the path to the JAR to localize

target - the directory where the JAR file should be localized

owner - the name of the user who should own the localized file

Returns:

the path to the localized JAR file

Throws:

java.io.IOException - if localization fails

startLocalizer

public void startLocalizer(LocalizerStartContext ctx)
                    throws java.io.IOException,
                           java.lang.InterruptedException

Description copied from class: ContainerExecutor

Prepare the environment for containers in this application to execute.

 For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc -> $N/$user/$appId/filecache/idef
 

Overrides:

startLocalizer in class DefaultContainerExecutor

Parameters:

ctx - LocalizerStartContext that encapsulates necessary information for starting a localizer.

Throws:

java.io.IOException - for most application init failures

java.lang.InterruptedException - if application init thread is halted by NM

buildCommandExecutor

protected org.apache.hadoop.util.Shell.CommandExecutor buildCommandExecutor(java.lang.String wrapperScriptPath,
                                                                            java.lang.String containerIdStr,
                                                                            java.lang.String userName,
                                                                            org.apache.hadoop.fs.Path pidFile,
                                                                            org.apache.hadoop.yarn.api.records.Resource resource,
                                                                            java.io.File wordDir,
                                                                            java.util.Map<java.lang.String,java.lang.String> environment,
                                                                            java.lang.String[] numaCommands)

Description copied from class: DefaultContainerExecutor

Create a new Shell.ShellCommandExecutor using the parameters.

Overrides:

buildCommandExecutor in class DefaultContainerExecutor

Parameters:

wrapperScriptPath - the path to the script to execute

containerIdStr - the container ID

userName - the application owner's username

pidFile - the path to the container's PID file

resource - this parameter controls memory and CPU limits.

wordDir - If not-null, specifies the directory which should be set as the current working directory for the command. If null, the current working directory is not modified.

environment - the container environment

numaCommands - list of prefix numa commands

Returns:

the new Shell.ShellCommandExecutor

See Also:

Shell.ShellCommandExecutor

killContainer

protected void killContainer(java.lang.String pid,
                             ContainerExecutor.Signal signal)
                      throws java.io.IOException

Description copied from class: DefaultContainerExecutor

Send a specified signal to the specified pid

Overrides:

killContainer in class DefaultContainerExecutor

Parameters:

pid - the pid of the process [group] to signal.

signal - signal to send

Throws:

java.io.IOException - if the command to kill the process fails