Package org.apache.hadoop.yarn.logaggregation.filecontroller.nodelocal

Class LogAggregationNodeLocalTFileController

java.lang.Object

org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

org.apache.hadoop.yarn.logaggregation.filecontroller.nodelocal.LogAggregationNodeLocalTFileController

public class LogAggregationNodeLocalTFileController
extends LogAggregationFileController

Field Summary

Fields inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

APP_DIR_PERMISSIONS, APP_LOG_FILE_UMASK, conf, fileControllerName, fsSupportsChmod, maxRetry, remoteOlderRootLogDirSuffix, remoteRootLogDir, remoteRootLogDirSuffix, retentionSize, retryTimeout, TLDIR_PERMISSIONS, usersAclsManager

Constructor Summary

Constructors

Constructor	Description
LogAggregationNodeLocalTFileController()

Method Summary

Modifier and Type	Method	Description
protected void	cleanOldLogs(org.apache.hadoop.fs.Path remoteNodeLogFileForApp, org.apache.hadoop.yarn.api.records.NodeId nodeId, org.apache.hadoop.security.UserGroupInformation userUgi)
void	closeWriter()	Close the writer.
void	createAppDir(java.lang.String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.security.UserGroupInformation userUgi)	Create remote Application directory for log aggregation.
protected void	extractRemoteRootLogDir()	Sets the remoteRootLogDir class variable extracting YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT from the configuration or YarnConfiguration.NM_REMOTE_APP_LOG_DIR, if the former is not set.
protected void	extractRemoteRootLogDirSuffix()	Sets the remoteRootLogDirSuffix class variable extracting YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_SUFFIX_FMT from the configuration, or YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX appended by the FileController's name, if the former is not set.
java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,java.lang.String>	getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLog, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Returns ACLs for the application.
java.lang.String	getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLog, org.apache.hadoop.yarn.api.records.ApplicationId appId)	Returns the owner of the application.
org.apache.hadoop.fs.Path	getNodeLocalMetadataDir()
java.lang.String	getNodeLocalMetadataDirPrefix()
NodeLocalMetadataReader	getNodeLocalMetadataReader()
org.apache.hadoop.fs.Path	getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String appOwner)	Get the older remote application directory for log aggregation.
org.apache.hadoop.fs.Path	getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String appOwner)	Get the remote application directory for log aggregation.
org.apache.hadoop.fs.Path	getRemoteNodeLogFileForApp(org.apache.hadoop.yarn.api.records.ApplicationId appId, java.lang.String user, org.apache.hadoop.yarn.api.records.NodeId nodeId)	Get the remote aggregated log path.
void	initializeWriter(LogAggregationFileControllerContext context)	Initialize the writer.
void	initInternal(org.apache.hadoop.conf.Configuration conf)	Derived classes initialize themselves using this method.
void	postWrite(LogAggregationFileControllerContext record)	Operations needed after write the log content.
boolean	readAggregatedLogs(ContainerLogsRequest logRequest, java.io.OutputStream os)	Output container log.
java.util.List<ContainerLogMeta>	readAggregatedLogsMeta(ContainerLogsRequest logRequest)	Return a list of ContainerLogMeta for an application from Remote FileSystem.
void	renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context)	Render Aggregated Logs block.
void	write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue)	Write the log content.

Methods inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

aggregatedLogSuffix, belongsToAppAttempt, checkExists, closePrintStream, createDir, extractRemoteOlderRootLogDirSuffix, getApplicationDirectoriesOfUser, getFileControllerName, getFileSystem, getLogMetaFilesOfNode, getNodeFilesOfApplicationDirectory, getRemoteOlderRootLogDirSuffix, getRemoteRootLogDir, getRemoteRootLogDirSuffix, initialize, isFsSupportsChmod, verifyAndCreateRemoteLogDir

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LogAggregationNodeLocalTFileController

public LogAggregationNodeLocalTFileController()

Method Detail

extractRemoteRootLogDir

protected void extractRemoteRootLogDir()

Description copied from class: LogAggregationFileController

Sets the remoteRootLogDir class variable extracting YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT from the configuration or YarnConfiguration.NM_REMOTE_APP_LOG_DIR, if the former is not set.

Overrides:

extractRemoteRootLogDir in class LogAggregationFileController

extractRemoteRootLogDirSuffix

protected void extractRemoteRootLogDirSuffix()

Description copied from class: LogAggregationFileController

Sets the remoteRootLogDirSuffix class variable extracting YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_SUFFIX_FMT from the configuration, or YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX appended by the FileController's name, if the former is not set.

Overrides:

extractRemoteRootLogDirSuffix in class LogAggregationFileController

initInternal

public void initInternal(org.apache.hadoop.conf.Configuration conf)

Description copied from class: LogAggregationFileController

Derived classes initialize themselves using this method.

Specified by:

initInternal in class LogAggregationFileController

Parameters:

conf - the Configuration

createAppDir

public void createAppDir(java.lang.String user,
                         org.apache.hadoop.yarn.api.records.ApplicationId appId,
                         org.apache.hadoop.security.UserGroupInformation userUgi)

Description copied from class: LogAggregationFileController

Create remote Application directory for log aggregation.

Overrides:

createAppDir in class LogAggregationFileController

Parameters:

user - the user

appId - the application ID

userUgi - the UGI

initializeWriter

public void initializeWriter(LogAggregationFileControllerContext context)
                      throws java.io.IOException

Description copied from class: LogAggregationFileController

Initialize the writer.

Specified by:

initializeWriter in class LogAggregationFileController

Parameters:

context - the LogAggregationFileControllerContext

Throws:

java.io.IOException - if fails to initialize the writer

closeWriter

public void closeWriter()
                 throws LogAggregationDFSException

Description copied from class: LogAggregationFileController

Close the writer.

Specified by:

closeWriter in class LogAggregationFileController

Throws:

LogAggregationDFSException - if the closing of the writer fails (for example due to HDFS quota being exceeded)

write

public void write(AggregatedLogFormat.LogKey logKey,
                  AggregatedLogFormat.LogValue logValue)
           throws java.io.IOException

Description copied from class: LogAggregationFileController

Write the log content.

Specified by:

write in class LogAggregationFileController

Parameters:

logKey - the log key

logValue - the log content

Throws:

java.io.IOException - if fails to write the logs

postWrite

public void postWrite(LogAggregationFileControllerContext record)
               throws java.lang.Exception

Description copied from class: LogAggregationFileController

Operations needed after write the log content.

Specified by:

postWrite in class LogAggregationFileController

Parameters:

record - the LogAggregationFileControllerContext

Throws:

java.lang.Exception - if anything fails

readAggregatedLogs

public boolean readAggregatedLogs(ContainerLogsRequest logRequest,
                                  java.io.OutputStream os)
                           throws java.io.IOException

Description copied from class: LogAggregationFileController

Output container log.

Specified by:

readAggregatedLogs in class LogAggregationFileController

Parameters:

logRequest - ContainerLogsRequest

os - the output stream

Returns:

true if we can read the aggregated logs successfully

Throws:

java.io.IOException - if we can not access the log file.

readAggregatedLogsMeta

public java.util.List<ContainerLogMeta> readAggregatedLogsMeta(ContainerLogsRequest logRequest)
                                                        throws java.io.IOException

Description copied from class: LogAggregationFileController

Return a list of ContainerLogMeta for an application from Remote FileSystem.

Specified by:

readAggregatedLogsMeta in class LogAggregationFileController

Parameters:

logRequest - ContainerLogsRequest

Returns:

a list of ContainerLogMeta

Throws:

java.io.IOException - if there is no available log file

renderAggregatedLogsBlock

public void renderAggregatedLogsBlock(HtmlBlock.Block html,
                                      View.ViewContext context)

Description copied from class: LogAggregationFileController

Render Aggregated Logs block.

Specified by:

renderAggregatedLogsBlock in class LogAggregationFileController

Parameters:

html - the html

context - the ViewContext

getRemoteNodeLogFileForApp

public org.apache.hadoop.fs.Path getRemoteNodeLogFileForApp(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                            java.lang.String user,
                                                            org.apache.hadoop.yarn.api.records.NodeId nodeId)

Description copied from class: LogAggregationFileController

Get the remote aggregated log path.

Overrides:

getRemoteNodeLogFileForApp in class LogAggregationFileController

Parameters:

appId - the ApplicationId

user - the Application Owner

nodeId - the NodeManager Id

Returns:

the remote aggregated log path

getRemoteAppLogDir

public org.apache.hadoop.fs.Path getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                    java.lang.String appOwner)
                                             throws java.io.IOException

Description copied from class: LogAggregationFileController

Get the remote application directory for log aggregation.

Overrides:

getRemoteAppLogDir in class LogAggregationFileController

Parameters:

appId - the Application ID

appOwner - the Application Owner

Returns:

the remote application directory

Throws:

java.io.IOException - if can not find the remote application directory

getOlderRemoteAppLogDir

public org.apache.hadoop.fs.Path getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                         java.lang.String appOwner)
                                                  throws java.io.IOException

Description copied from class: LogAggregationFileController

Get the older remote application directory for log aggregation.

Overrides:

getOlderRemoteAppLogDir in class LogAggregationFileController

Parameters:

appId - the Application ID

appOwner - the Application Owner

Returns:

the older remote application directory

Throws:

java.io.IOException - if can not find the remote application directory

cleanOldLogs

protected void cleanOldLogs(org.apache.hadoop.fs.Path remoteNodeLogFileForApp,
                            org.apache.hadoop.yarn.api.records.NodeId nodeId,
                            org.apache.hadoop.security.UserGroupInformation userUgi)

Overrides:

cleanOldLogs in class LogAggregationFileController

getNodeLocalMetadataReader

public NodeLocalMetadataReader getNodeLocalMetadataReader()

getNodeLocalMetadataDir

public org.apache.hadoop.fs.Path getNodeLocalMetadataDir()

getNodeLocalMetadataDirPrefix

public java.lang.String getNodeLocalMetadataDirPrefix()

getApplicationOwner

public java.lang.String getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLog,
                                            org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                     throws java.io.IOException

Description copied from class: LogAggregationFileController

Returns the owner of the application.

Specified by:

getApplicationOwner in class LogAggregationFileController

Parameters:

aggregatedLog - the aggregatedLog path

appId - the ApplicationId

Returns:

the application owner

Throws:

java.io.IOException - if we can not get the application owner

getApplicationAcls

public java.util.Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,java.lang.String> getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLog,
                                                                                                                         org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                                                                  throws java.io.IOException

Description copied from class: LogAggregationFileController

Returns ACLs for the application. An empty map is returned if no ACLs are found.

Specified by:

getApplicationAcls in class LogAggregationFileController

Parameters:

aggregatedLog - the aggregatedLog path.

appId - the ApplicationId

Returns:

a map of the Application ACLs.

Throws:

java.io.IOException - if we can not get the application acls