Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf

Class YarnConfigurationStore

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.YarnConfigurationStore

All Implemented Interfaces:

java.lang.AutoCloseable

Direct Known Subclasses:

FSSchedulerConfigurationStore, InMemoryConfigurationStore, LeveldbConfigurationStore, ZKConfigurationStore

public abstract class YarnConfigurationStore
extends java.lang.Object
implements java.lang.AutoCloseable

YarnConfigurationStore exposes the methods needed for retrieving and persisting CapacityScheduler configuration via key-value using write-ahead logging. When configuration mutation is requested, caller should first log it with logMutation, which persists this pending mutation. This mutation is merged to the persisted configuration only after confirmMutation is called. On startup/recovery, caller should call retrieve to get all confirmed mutations, then get pending mutations which were not confirmed via getPendingMutations, and replay/confirm them via confirmMutation as in the normal case.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	YarnConfigurationStore.LogMutation	LogMutation encapsulates the fields needed for configuration mutation audit logging and recovery.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
YarnConfigurationStore()

Method Summary

Modifier and Type	Method	Description
void	checkVersion()
abstract void	close()	Closes the configuration store, releasing any required resources.
abstract void	confirmMutation(YarnConfigurationStore.LogMutation pendingMutation, boolean isValid)	Should be called after logMutation.
abstract void	format()	Format the persisted configuration.
abstract long	getConfigVersion()	Get the last updated config version.
abstract java.util.List<YarnConfigurationStore.LogMutation>	getConfirmedConfHistory(long fromId)	Get a list of confirmed configuration mutations starting from a given id.
protected abstract org.apache.hadoop.yarn.server.records.Version	getConfStoreVersion()	Get schema version of persisted conf store, for detecting compatibility issues when changing conf store schema.
protected abstract org.apache.hadoop.yarn.server.records.Version	getCurrentVersion()	Get the hard-coded schema version, for comparison against the schema version currently persisted.
protected abstract java.util.LinkedList<YarnConfigurationStore.LogMutation>	getLogs()	Get a list of configuration mutations.
abstract void	initialize(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.conf.Configuration schedConf, RMContext rmContext)	Initialize the configuration store, with schedConf as the initial scheduler configuration.
abstract void	logMutation(YarnConfigurationStore.LogMutation logMutation)	Logs the configuration change to backing store.
abstract org.apache.hadoop.conf.Configuration	retrieve()	Retrieve the persisted configuration.
protected abstract void	storeVersion()	Persist the hard-coded schema version to the conf store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

YarnConfigurationStore

public YarnConfigurationStore()

Method Detail

initialize

public abstract void initialize(org.apache.hadoop.conf.Configuration conf,
                                org.apache.hadoop.conf.Configuration schedConf,
                                RMContext rmContext)
                         throws java.lang.Exception

Initialize the configuration store, with schedConf as the initial scheduler configuration. If a persisted store already exists, use the scheduler configuration stored there, and ignore schedConf.

Parameters:

conf - configuration to initialize store with

schedConf - Initial key-value scheduler configuration to persist.

rmContext - RMContext for this configuration store

Throws:

java.io.IOException - if initialization fails

java.lang.Exception

close

public abstract void close()
                    throws java.io.IOException

Closes the configuration store, releasing any required resources.

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException - on failure to close

logMutation

public abstract void logMutation(YarnConfigurationStore.LogMutation logMutation)
                          throws java.lang.Exception

Logs the configuration change to backing store.

Parameters:

logMutation - configuration change to be persisted in write ahead log

Throws:

java.io.IOException - if logging fails

java.lang.Exception

confirmMutation

public abstract void confirmMutation(YarnConfigurationStore.LogMutation pendingMutation,
                                     boolean isValid)
                              throws java.lang.Exception

Should be called after logMutation. Gets the pending mutation last logged by logMutation and marks the mutation as persisted (no longer pending). If isValid is true, merge the mutation with the persisted configuration.

Parameters:

pendingMutation - the log mutation to apply

isValid - if true, update persisted configuration with pending mutation.

Throws:

java.lang.Exception - if mutation confirmation fails

retrieve

public abstract org.apache.hadoop.conf.Configuration retrieve()
                                                       throws java.io.IOException

Retrieve the persisted configuration.

Returns:

configuration as key-value

Throws:

java.io.IOException - an I/O exception has occurred.

format

public abstract void format()
                     throws java.lang.Exception

Format the persisted configuration.

Throws:

java.io.IOException - on failure to format

java.lang.Exception

getConfigVersion

public abstract long getConfigVersion()
                               throws java.lang.Exception

Get the last updated config version.

Returns:

Last updated config version.

Throws:

java.lang.Exception - On version fetch failure.

getConfirmedConfHistory

public abstract java.util.List<YarnConfigurationStore.LogMutation> getConfirmedConfHistory(long fromId)

Get a list of confirmed configuration mutations starting from a given id.

Parameters:

fromId - id from which to start getting mutations, inclusive

Returns:

list of configuration mutations

getConfStoreVersion

protected abstract org.apache.hadoop.yarn.server.records.Version getConfStoreVersion()
                                                                              throws java.lang.Exception

Get schema version of persisted conf store, for detecting compatibility issues when changing conf store schema.

Returns:

Schema version currently used by the persisted configuration store.

Throws:

java.lang.Exception - On version fetch failure

getLogs

protected abstract java.util.LinkedList<YarnConfigurationStore.LogMutation> getLogs()
                                                                             throws java.lang.Exception

Get a list of configuration mutations.

Returns:

list of configuration mutations.

Throws:

java.lang.Exception - On mutation fetch failure

storeVersion

protected abstract void storeVersion()
                              throws java.lang.Exception

Persist the hard-coded schema version to the conf store.

Throws:

java.lang.Exception - On storage failure

getCurrentVersion

protected abstract org.apache.hadoop.yarn.server.records.Version getCurrentVersion()

Get the hard-coded schema version, for comparison against the schema version currently persisted.

Returns:

Current hard-coded schema version

checkVersion

public void checkVersion()
                  throws java.lang.Exception

Throws:

java.lang.Exception