org.apache.hadoop.yarn.server.applicationhistoryservice.timeline

Class LeveldbTimelineStore

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.LeveldbTimelineStore

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, TimelineReader, TimelineStore, TimelineWriter

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class LeveldbTimelineStore
extends org.apache.hadoop.service.AbstractService
implements TimelineStore

An implementation of an application timeline store backed by leveldb.

There are three sections of the db, the start time section, the entity section, and the indexed entity section.

The start time section is used to retrieve the unique start time for a given entity. Its values each contain a start time while its keys are of the form:

   START_TIME_LOOKUP_PREFIX + entity type + entity id

The entity section is ordered by entity type, then entity start time descending, then entity ID. There are four sub-sections of the entity section: events, primary filters, related entities, and other info. The event entries have event info serialized into their values. The other info entries have values corresponding to the values of the other info name/value map for the entry (note the names are contained in the key). All other entries have empty values. The key structure is as follows:

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     EVENTS_COLUMN + reveventtimestamp + eventtype

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     PRIMARY_FILTERS_COLUMN + name + value

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     OTHER_INFO_COLUMN + name

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     RELATED_ENTITIES_COLUMN + relatedentity type + relatedentity id

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     INVISIBLE_REVERSE_RELATED_ENTITIES_COLUMN + relatedentity type +
     relatedentity id

The indexed entity section contains a primary filter name and primary filter value as the prefix. Within a given name/value, entire entity entries are stored in the same format as described in the entity section above (below, "key" represents any one of the possible entity entry keys described above).

   INDEXED_ENTRY_PREFIX + primaryfilter name + primaryfilter value +
     key

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Nested classes/interfaces inherited from interface org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.TimelineReader

TimelineReader.Field

Field Summary

Fields inherited from interface org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.TimelineReader

DEFAULT_LIMIT

Constructor Summary

Constructors

Constructor and Description
LeveldbTimelineStore()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.yarn.api.records.timeline.TimelineEntities	getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection<NameValuePair> secondaryFilters, EnumSet<TimelineReader.Field> fields) This method retrieves a list of entity information, TimelineEntity, sorted by the starting timestamp for the entity, descending.
org.apache.hadoop.yarn.api.records.timeline.TimelineEntity	getEntity(String entityId, String entityType, EnumSet<TimelineReader.Field> fields) This method retrieves the entity information for a given entity.
org.apache.hadoop.yarn.api.records.timeline.TimelineEvents	getEntityTimelines(String entityType, SortedSet<String> entityIds, Long limit, Long windowStart, Long windowEnd, Set<String> eventType) This method retrieves the events for a list of entities all of the same entity type.
org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse	put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities entities) Stores entity information to the timeline store.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStop()

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Constructor Detail

LeveldbTimelineStore

public LeveldbTimelineStore()

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

getEntity

public org.apache.hadoop.yarn.api.records.timeline.TimelineEntity getEntity(String entityId,
                                                                   String entityType,
                                                                   EnumSet<TimelineReader.Field> fields)
                                                                     throws IOException

Description copied from interface: TimelineReader

This method retrieves the entity information for a given entity.

Specified by:

getEntity in interface TimelineReader

Parameters:

entityId - The entity whose information will be retrieved.

entityType - The type of the entity.

fields - Specifies which fields of the entity object to retrieve (see TimelineReader.Field). If the set of fields contains TimelineReader.Field.LAST_EVENT_ONLY and not TimelineReader.Field.EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.

Returns:

An TimelineEntity object.

Throws:

IOException

getEntityTimelines

public org.apache.hadoop.yarn.api.records.timeline.TimelineEvents getEntityTimelines(String entityType,
                                                                            SortedSet<String> entityIds,
                                                                            Long limit,
                                                                            Long windowStart,
                                                                            Long windowEnd,
                                                                            Set<String> eventType)
                                                                              throws IOException

Description copied from interface: TimelineReader

This method retrieves the events for a list of entities all of the same entity type. The events for each entity are sorted in order of their timestamps, descending.

Specified by:

getEntityTimelines in interface TimelineReader

Parameters:

entityType - The type of entities to retrieve events for.

entityIds - The entity IDs to retrieve events for.

limit - A limit on the number of events to return for each entity. If null, defaults to TimelineReader.DEFAULT_LIMIT events per entity.

windowStart - If not null, retrieves only events later than the given time (exclusive)

windowEnd - If not null, retrieves only events earlier than the given time (inclusive)

eventType - Restricts the events returned to the given types. If null, events of all types will be returned.

Returns:

An TimelineEvents object.

Throws:

IOException

getEntities

public org.apache.hadoop.yarn.api.records.timeline.TimelineEntities getEntities(String entityType,
                                                                       Long limit,
                                                                       Long windowStart,
                                                                       Long windowEnd,
                                                                       String fromId,
                                                                       Long fromTs,
                                                                       NameValuePair primaryFilter,
                                                                       Collection<NameValuePair> secondaryFilters,
                                                                       EnumSet<TimelineReader.Field> fields)
                                                                         throws IOException

Description copied from interface: TimelineReader

This method retrieves a list of entity information, TimelineEntity, sorted by the starting timestamp for the entity, descending. The starting timestamp of an entity is a timestamp specified by the client. If it is not explicitly specified, it will be chosen by the store to be the earliest timestamp of the events received in the first put for the entity.

Specified by:

getEntities in interface TimelineReader

Parameters:

entityType - The type of entities to return (required).

limit - A limit on the number of entities to return. If null, defaults to TimelineReader.DEFAULT_LIMIT.

windowStart - The earliest start timestamp to retrieve (exclusive). If null, defaults to retrieving all entities until the limit is reached.

windowEnd - The latest start timestamp to retrieve (inclusive). If null, defaults to Long.MAX_VALUE

fromId - If fromId is not null, retrieve entities earlier than and including the specified ID. If no start time is found for the specified ID, an empty list of entities will be returned. The windowEnd parameter will take precedence if the start time of this entity falls later than windowEnd.

fromTs - If fromTs is not null, ignore entities that were inserted into the store after the given timestamp. The entity's insert timestamp used for this comparison is the store's system time when the first put for the entity was received (not the entity's start time).

primaryFilter - Retrieves only entities that have the specified primary filter. If null, retrieves all entities. This is an indexed retrieval, and no entities that do not match the filter are scanned.

secondaryFilters - Retrieves only entities that have exact matches for all the specified filters in their primary filters or other info. This is not an indexed retrieval, so all entities are scanned but only those matching the filters are returned.

fields - Specifies which fields of the entity object to retrieve (see TimelineReader.Field). If the set of fields contains TimelineReader.Field.LAST_EVENT_ONLY and not TimelineReader.Field.EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.

Returns:

An TimelineEntities object.

Throws:

IOException

put

public org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities entities)

Description copied from interface: TimelineWriter

Stores entity information to the timeline store. Any errors occurring for individual put request objects will be reported in the response.

Specified by:

put in interface TimelineWriter

Parameters:

entities - An TimelineEntities object.

Returns:

An TimelinePutResponse object.