Package org.apache.hadoop.mapred

Class YARNRunner

java.lang.Object

org.apache.hadoop.mapred.YARNRunner

All Implemented Interfaces:

org.apache.hadoop.ipc.VersionedProtocol, org.apache.hadoop.mapreduce.protocol.ClientProtocol

public class YARNRunner
extends java.lang.Object
implements org.apache.hadoop.mapreduce.protocol.ClientProtocol

This class enables the current JobClient (0.22 hadoop) to run on YARN.

Field Summary

Fields

Modifier and Type	Field	Description
static org.apache.hadoop.yarn.api.records.Priority	AM_CONTAINER_PRIORITY

Fields inherited from interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

versionID

Constructor Summary

Constructors

Constructor	Description
YARNRunner(org.apache.hadoop.conf.Configuration conf)	Yarn runner incapsulates the client interface of yarn
YARNRunner(org.apache.hadoop.conf.Configuration conf, ResourceMgrDelegate resMgrDelegate)	Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate.
YARNRunner(org.apache.hadoop.conf.Configuration conf, ResourceMgrDelegate resMgrDelegate, ClientCache clientCache)	Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache.

Method Summary

Modifier and Type	Method	Description
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
void	close()
org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext	createApplicationSubmissionContext(org.apache.hadoop.conf.Configuration jobConf, java.lang.String jobSubmitDir, org.apache.hadoop.security.Credentials ts)	Constructs all the necessary information to start the MR AM.
org.apache.hadoop.mapreduce.TaskTrackerInfo[]	getActiveTrackers()
org.apache.hadoop.mapreduce.JobStatus[]	getAllJobs()
org.apache.hadoop.mapreduce.TaskTrackerInfo[]	getBlacklistedTrackers()
org.apache.hadoop.mapreduce.QueueInfo[]	getChildQueues(java.lang.String parent)
org.apache.hadoop.mapreduce.ClusterMetrics	getClusterMetrics()
org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier>	getDelegationToken(org.apache.hadoop.io.Text renewer)
java.lang.String	getFilesystemName()
org.apache.hadoop.mapreduce.Counters	getJobCounters(org.apache.hadoop.mapreduce.JobID arg0)
java.lang.String	getJobHistoryDir()
org.apache.hadoop.mapreduce.JobStatus	getJobStatus(org.apache.hadoop.mapreduce.JobID jobID)
org.apache.hadoop.mapreduce.Cluster.JobTrackerStatus	getJobTrackerStatus()
org.apache.hadoop.mapreduce.v2.LogParams	getLogFileParams(org.apache.hadoop.mapreduce.JobID jobID, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID)
org.apache.hadoop.mapreduce.JobID	getNewJobID()
org.apache.hadoop.ipc.ProtocolSignature	getProtocolSignature(java.lang.String protocol, long clientVersion, int clientMethodsHash)
long	getProtocolVersion(java.lang.String arg0, long arg1)
org.apache.hadoop.mapreduce.QueueInfo	getQueue(java.lang.String queueName)
org.apache.hadoop.mapreduce.QueueAclsInfo[]	getQueueAclsForCurrentUser()
org.apache.hadoop.security.authorize.AccessControlList	getQueueAdmins(java.lang.String arg0)
org.apache.hadoop.mapreduce.QueueInfo[]	getQueues()
org.apache.hadoop.mapreduce.QueueInfo[]	getRootQueues()
java.lang.String	getStagingAreaDir()
java.lang.String	getSystemDir()
org.apache.hadoop.mapreduce.TaskCompletionEvent[]	getTaskCompletionEvents(org.apache.hadoop.mapreduce.JobID arg0, int arg1, int arg2)
java.lang.String[]	getTaskDiagnostics(org.apache.hadoop.mapreduce.TaskAttemptID arg0)
org.apache.hadoop.mapreduce.TaskReport[]	getTaskReports(org.apache.hadoop.mapreduce.JobID jobID, org.apache.hadoop.mapreduce.TaskType taskType)
long	getTaskTrackerExpiryInterval()
void	killJob(org.apache.hadoop.mapreduce.JobID arg0)
boolean	killTask(org.apache.hadoop.mapreduce.TaskAttemptID arg0, boolean arg1)
long	renewDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
void	setJobPriority(org.apache.hadoop.mapreduce.JobID arg0, java.lang.String arg1)
void	setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)
org.apache.hadoop.mapreduce.JobStatus	submitJob(org.apache.hadoop.mapreduce.JobID jobId, java.lang.String jobSubmitDir, org.apache.hadoop.security.Credentials ts)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AM_CONTAINER_PRIORITY

public static final org.apache.hadoop.yarn.api.records.Priority AM_CONTAINER_PRIORITY

Constructor Detail

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf)

Yarn runner incapsulates the client interface of yarn

Parameters:

conf - the configuration object for the client

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf,
                  ResourceMgrDelegate resMgrDelegate)

Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate. Enables mocking and testing.

Parameters:

conf - the configuration object for the client

resMgrDelegate - the resourcemanager client handle.

YARNRunner

public YARNRunner(org.apache.hadoop.conf.Configuration conf,
                  ResourceMgrDelegate resMgrDelegate,
                  ClientCache clientCache)

Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache. Enable mocking and testing.

Parameters:

conf - the configuration object

resMgrDelegate - the resource manager delegate

clientCache - the client cache object.

Method Detail

setResourceMgrDelegate

@Private
public void setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)

cancelDelegationToken

public void cancelDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
                           throws java.io.IOException,
                                  java.lang.InterruptedException

Specified by:

cancelDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getActiveTrackers

public org.apache.hadoop.mapreduce.TaskTrackerInfo[] getActiveTrackers()
                                                                throws java.io.IOException,
                                                                       java.lang.InterruptedException

Specified by:

getActiveTrackers in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getAllJobs

public org.apache.hadoop.mapreduce.JobStatus[] getAllJobs()
                                                   throws java.io.IOException,
                                                          java.lang.InterruptedException

Specified by:

getAllJobs in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getBlacklistedTrackers

public org.apache.hadoop.mapreduce.TaskTrackerInfo[] getBlacklistedTrackers()
                                                                     throws java.io.IOException,
                                                                            java.lang.InterruptedException

Specified by:

getBlacklistedTrackers in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getClusterMetrics

public org.apache.hadoop.mapreduce.ClusterMetrics getClusterMetrics()
                                                             throws java.io.IOException,
                                                                    java.lang.InterruptedException

Specified by:

getClusterMetrics in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getDelegationToken

public org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                                                                           throws java.io.IOException,
                                                                                                                                                  java.lang.InterruptedException

Specified by:

getDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getFilesystemName

public java.lang.String getFilesystemName()
                                   throws java.io.IOException,
                                          java.lang.InterruptedException

Specified by:

getFilesystemName in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getNewJobID

public org.apache.hadoop.mapreduce.JobID getNewJobID()
                                              throws java.io.IOException,
                                                     java.lang.InterruptedException

Specified by:

getNewJobID in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getQueue

public org.apache.hadoop.mapreduce.QueueInfo getQueue(java.lang.String queueName)
                                               throws java.io.IOException,
                                                      java.lang.InterruptedException

Specified by:

getQueue in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getQueueAclsForCurrentUser

public org.apache.hadoop.mapreduce.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                       throws java.io.IOException,
                                                                              java.lang.InterruptedException

Specified by:

getQueueAclsForCurrentUser in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getQueues()
                                                  throws java.io.IOException,
                                                         java.lang.InterruptedException

Specified by:

getQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getRootQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getRootQueues()
                                                      throws java.io.IOException,
                                                             java.lang.InterruptedException

Specified by:

getRootQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getChildQueues

public org.apache.hadoop.mapreduce.QueueInfo[] getChildQueues(java.lang.String parent)
                                                       throws java.io.IOException,
                                                              java.lang.InterruptedException

Specified by:

getChildQueues in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getStagingAreaDir

public java.lang.String getStagingAreaDir()
                                   throws java.io.IOException,
                                          java.lang.InterruptedException

Specified by:

getStagingAreaDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getSystemDir

public java.lang.String getSystemDir()
                              throws java.io.IOException,
                                     java.lang.InterruptedException

Specified by:

getSystemDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getTaskTrackerExpiryInterval

public long getTaskTrackerExpiryInterval()
                                  throws java.io.IOException,
                                         java.lang.InterruptedException

Specified by:

getTaskTrackerExpiryInterval in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

submitJob

public org.apache.hadoop.mapreduce.JobStatus submitJob(org.apache.hadoop.mapreduce.JobID jobId,
                                                       java.lang.String jobSubmitDir,
                                                       org.apache.hadoop.security.Credentials ts)
                                                throws java.io.IOException,
                                                       java.lang.InterruptedException

Specified by:

submitJob in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

createApplicationSubmissionContext

public org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext createApplicationSubmissionContext(org.apache.hadoop.conf.Configuration jobConf,
                                                                                                          java.lang.String jobSubmitDir,
                                                                                                          org.apache.hadoop.security.Credentials ts)
                                                                                                   throws java.io.IOException

Constructs all the necessary information to start the MR AM.

Parameters:

jobConf - the configuration for the MR job

jobSubmitDir - the directory path for the job

ts - the security credentials for the job

Returns:

ApplicationSubmissionContext

Throws:

java.io.IOException - on IO error (e.g. path resolution)

setJobPriority

public void setJobPriority(org.apache.hadoop.mapreduce.JobID arg0,
                           java.lang.String arg1)
                    throws java.io.IOException,
                           java.lang.InterruptedException

Specified by:

setJobPriority in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getProtocolVersion

public long getProtocolVersion(java.lang.String arg0,
                               long arg1)
                        throws java.io.IOException

Specified by:

getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

java.io.IOException

renewDelegationToken

public long renewDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> arg0)
                          throws java.io.IOException,
                                 java.lang.InterruptedException

Specified by:

renewDelegationToken in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getJobCounters

public org.apache.hadoop.mapreduce.Counters getJobCounters(org.apache.hadoop.mapreduce.JobID arg0)
                                                    throws java.io.IOException,
                                                           java.lang.InterruptedException

Specified by:

getJobCounters in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getJobHistoryDir

public java.lang.String getJobHistoryDir()
                                  throws java.io.IOException,
                                         java.lang.InterruptedException

Specified by:

getJobHistoryDir in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getJobStatus

public org.apache.hadoop.mapreduce.JobStatus getJobStatus(org.apache.hadoop.mapreduce.JobID jobID)
                                                   throws java.io.IOException,
                                                          java.lang.InterruptedException

Specified by:

getJobStatus in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getTaskCompletionEvents

public org.apache.hadoop.mapreduce.TaskCompletionEvent[] getTaskCompletionEvents(org.apache.hadoop.mapreduce.JobID arg0,
                                                                                 int arg1,
                                                                                 int arg2)
                                                                          throws java.io.IOException,
                                                                                 java.lang.InterruptedException

Specified by:

getTaskCompletionEvents in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getTaskDiagnostics

public java.lang.String[] getTaskDiagnostics(org.apache.hadoop.mapreduce.TaskAttemptID arg0)
                                      throws java.io.IOException,
                                             java.lang.InterruptedException

Specified by:

getTaskDiagnostics in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getTaskReports

public org.apache.hadoop.mapreduce.TaskReport[] getTaskReports(org.apache.hadoop.mapreduce.JobID jobID,
                                                               org.apache.hadoop.mapreduce.TaskType taskType)
                                                        throws java.io.IOException,
                                                               java.lang.InterruptedException

Specified by:

getTaskReports in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

killJob

public void killJob(org.apache.hadoop.mapreduce.JobID arg0)
             throws java.io.IOException,
                    java.lang.InterruptedException

Specified by:

killJob in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

killTask

public boolean killTask(org.apache.hadoop.mapreduce.TaskAttemptID arg0,
                        boolean arg1)
                 throws java.io.IOException,
                        java.lang.InterruptedException

Specified by:

killTask in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getQueueAdmins

public org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(java.lang.String arg0)
                                                                      throws java.io.IOException

Specified by:

getQueueAdmins in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

getJobTrackerStatus

public org.apache.hadoop.mapreduce.Cluster.JobTrackerStatus getJobTrackerStatus()
                                                                         throws java.io.IOException,
                                                                                java.lang.InterruptedException

Specified by:

getJobTrackerStatus in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

java.lang.InterruptedException

getProtocolSignature

public org.apache.hadoop.ipc.ProtocolSignature getProtocolSignature(java.lang.String protocol,
                                                                    long clientVersion,
                                                                    int clientMethodsHash)
                                                             throws java.io.IOException

Specified by:

getProtocolSignature in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

java.io.IOException

getLogFileParams

public org.apache.hadoop.mapreduce.v2.LogParams getLogFileParams(org.apache.hadoop.mapreduce.JobID jobID,
                                                                 org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID)
                                                          throws java.io.IOException

Specified by:

getLogFileParams in interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException