Package org.apache.hadoop.yarn.server.timelineservice.collector

Class TimelineCollector

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.service.Service

Direct Known Subclasses:

AppLevelTimelineCollector

@Private
@Unstable
public abstract class TimelineCollector
extends org.apache.hadoop.service.CompositeService

Service that handles writes to the timeline service and writes them to the backing storage. Classes that extend this can add their own lifecycle management or customization of request handling.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	TimelineCollector.AggregationStatusTable

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	SEPARATOR

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Constructor Summary

Constructors

Constructor	Description
TimelineCollector(java.lang.String name)

Method Summary

Modifier and Type	Method	Description
static org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity	aggregateEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, java.lang.String resultEntityId, java.lang.String resultEntityType, boolean needsGroupIdInResult)	Aggregate all metrics in given timeline entities with no predefined states.
protected java.util.Map<java.lang.String,TimelineCollector.AggregationStatusTable>	getAggregationGroups()
protected java.util.Set<java.lang.String>	getEntityTypesSkipAggregation()	Method to decide the set of timeline entity types the collector should skip on aggregations.
abstract TimelineCollectorContext	getTimelineEntityContext()
protected boolean	isReadyToAggregate()
org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse	putDomain(org.apache.hadoop.yarn.api.records.timelineservice.TimelineDomain domain, org.apache.hadoop.security.UserGroupInformation callerUgi)	Add or update an domain.
org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse	putEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi)	Handles entity writes.
void	putEntitiesAsync(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi)	Handles entity writes in an asynchronous manner.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
protected void	setReadyToAggregate()
protected void	setWriter(TimelineWriter w)

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SEPARATOR

public static final java.lang.String SEPARATOR

See Also:

Constant Field Values

Constructor Detail

TimelineCollector

public TimelineCollector(java.lang.String name)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws java.lang.Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

java.lang.Exception

serviceStart

protected void serviceStart()
                     throws java.lang.Exception

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

java.lang.Exception

serviceStop

protected void serviceStop()
                    throws java.lang.Exception

Overrides:

serviceStop in class org.apache.hadoop.service.CompositeService

Throws:

java.lang.Exception

setWriter

protected void setWriter(TimelineWriter w)

getAggregationGroups

protected java.util.Map<java.lang.String,TimelineCollector.AggregationStatusTable> getAggregationGroups()

setReadyToAggregate

protected void setReadyToAggregate()

isReadyToAggregate

protected boolean isReadyToAggregate()

getEntityTypesSkipAggregation

protected java.util.Set<java.lang.String> getEntityTypesSkipAggregation()

Method to decide the set of timeline entity types the collector should skip on aggregations. Subclasses may want to override this method to customize their own behaviors.

Returns:

A set of strings consists of all types the collector should skip.

getTimelineEntityContext

public abstract TimelineCollectorContext getTimelineEntityContext()

putEntities

public org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse putEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities,
                                                                                            org.apache.hadoop.security.UserGroupInformation callerUgi)
                                                                                     throws java.io.IOException

Handles entity writes. These writes are synchronous and are written to the backing storage without buffering/batching. If any entity already exists, it results in an update of the entity. This method should be reserved for selected critical entities and events. For normal voluminous writes one should use the async method putEntitiesAsync(TimelineEntities, UserGroupInformation).

Parameters:

entities - entities to post

callerUgi - the caller UGI

Returns:

the response that contains the result of the post.

Throws:

java.io.IOException - if there is any exception encountered while putting entities.

putDomain

public org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse putDomain(org.apache.hadoop.yarn.api.records.timelineservice.TimelineDomain domain,
                                                                                          org.apache.hadoop.security.UserGroupInformation callerUgi)
                                                                                   throws java.io.IOException

Add or update an domain. If the domain already exists, only the owner and the admin can update it.

Parameters:

domain - domain to post

callerUgi - the caller UGI

Returns:

the response that contains the result of the post.

Throws:

java.io.IOException - if there is any exception encountered while putting domain.

putEntitiesAsync

public void putEntitiesAsync(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities,
                             org.apache.hadoop.security.UserGroupInformation callerUgi)
                      throws java.io.IOException

Handles entity writes in an asynchronous manner. The method returns as soon as validation is done. No promises are made on how quickly it will be written to the backing storage or if it will always be written to the backing storage. Multiple writes to the same entities may be batched and appropriate values updated and result in fewer writes to the backing storage.

Parameters:

entities - entities to post

callerUgi - the caller UGI

Throws:

java.io.IOException - if there is any exception encounted while putting entities.

aggregateEntities

public static org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity aggregateEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities,
                                                                                                  java.lang.String resultEntityId,
                                                                                                  java.lang.String resultEntityType,
                                                                                                  boolean needsGroupIdInResult)

Aggregate all metrics in given timeline entities with no predefined states.

Parameters:

entities - Entities to aggregate

resultEntityId - Id of the result entity

resultEntityType - Type of the result entity

needsGroupIdInResult - Marks if we want the aggregation group id in each aggregated metrics.

Returns:

A timeline entity that contains all aggregated TimelineMetric.