Package org.apache.hadoop.hdfs.server.namenode

Class FSImage

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.FSImage

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

BackupImage

@Private
@Evolving
public class FSImage
extends java.lang.Object
implements java.io.Closeable

FSImage handles checkpointing and logging of the namespace edits.

Field Summary

Fields

Modifier and Type	Field	Description
protected NNStorageRetentionManager	archivalManager
protected FSEditLog	editLog
protected long	lastAppliedTxId	The last transaction ID that was either loaded from an image or loaded by loading edits files.
static org.slf4j.Logger	LOG
static int	SHUTDOWN_HOOK_PRIORITY	Priority of the FSImageSaver shutdown hook: 10.
protected NNStorage	storage

Constructor Summary

Constructors

Modifier	Constructor	Description
	FSImage(org.apache.hadoop.conf.Configuration conf)	Construct an FSImage
protected	FSImage(org.apache.hadoop.conf.Configuration conf, java.util.Collection<java.net.URI> imageDirs, java.util.List<java.net.URI> editsDirs)	Construct the FSImage.

Method Summary

Modifier and Type	Method	Description
boolean	addToCheckpointing(long txid)
static void	checkUpgrade(NNStorage storage)	Check if upgrade is in progress.
void	close()
java.lang.String	getBlockPoolID()
java.lang.String	getClusterID()
long	getCorrectLastAppliedOrWrittenTxId()	This method holds a lock of FSEditLog to get the correct value.
FSEditLog	getEditLog()
long	getLastAppliedOrWrittenTxId()
long	getLastAppliedTxId()
int	getLayoutVersion()
long	getMostRecentCheckpointTxId()
int	getNamespaceID()
NNStorage	getStorage()
boolean	hasRollbackFSImage()
void	initEditLog(HdfsServerConstants.StartupOption startOpt)
long	loadEdits(java.lang.Iterable<EditLogInputStream> editStreams, FSNamesystem target)	Load the specified list of edit files into the image.
long	loadEdits(java.lang.Iterable<EditLogInputStream> editStreams, FSNamesystem target, long maxTxnsToRead, HdfsServerConstants.StartupOption startOpt, MetaRecoveryContext recovery)
static boolean	recoverStorageDirs(HdfsServerConstants.StartupOption startOpt, NNStorage storage, java.util.Map<Storage.StorageDirectory,Storage.StorageState> dataDirStates)	For each storage directory, performs recovery of incomplete transitions (eg.
void	removeFromCheckpointing(long txid)
void	saveDigestAndRenameCheckpointImage(NNStorage.NameNodeFile nnf, long txid, org.apache.hadoop.io.MD5Hash digest)	This is called by the 2NN after having downloaded an image, and by the NN after having received a new image from the 2NN.
protected void	saveFSImageInAllDirs(FSNamesystem source, long txid)
void	saveLegacyOIVImage(FSNamesystem source, java.lang.String targetDir, Canceler canceler)	Save FSimage in the legacy format.
boolean	saveNamespace(long timeWindow, long txGap, FSNamesystem source)
void	saveNamespace(FSNamesystem source)
void	saveNamespace(FSNamesystem source, NNStorage.NameNodeFile nnf, Canceler canceler)	Save the contents of the FS image to a new image file in each of the current storage directories.
void	updateLastAppliedTxIdFromWritten()
void	updateStorageVersion()	Update version of all storage directories.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the FSImageSaver shutdown hook: 10.

See Also:

Constant Field Values

editLog

protected FSEditLog editLog

storage

protected NNStorage storage

lastAppliedTxId

protected long lastAppliedTxId

The last transaction ID that was either loaded from an image or loaded by loading edits files.

archivalManager

protected NNStorageRetentionManager archivalManager

Constructor Detail

FSImage

public FSImage(org.apache.hadoop.conf.Configuration conf)
        throws java.io.IOException

Construct an FSImage

Parameters:

conf - Configuration

Throws:

java.io.IOException - if default directories are invalid.

FSImage

protected FSImage(org.apache.hadoop.conf.Configuration conf,
                  java.util.Collection<java.net.URI> imageDirs,
                  java.util.List<java.net.URI> editsDirs)
           throws java.io.IOException

Construct the FSImage. Set the default checkpoint directories. Setup storage and initialize the edit log.

Parameters:

conf - Configuration

imageDirs - Directories the image can be stored in.

editsDirs - Directories the editlog can be stored in.

Throws:

java.io.IOException - if directories are invalid.

Method Detail

recoverStorageDirs

public static boolean recoverStorageDirs(HdfsServerConstants.StartupOption startOpt,
                                         NNStorage storage,
                                         java.util.Map<Storage.StorageDirectory,Storage.StorageState> dataDirStates)
                                  throws java.io.IOException

For each storage directory, performs recovery of incomplete transitions (eg. upgrade, rollback, checkpoint) and inserts the directory's storage state into the dataDirStates map.

Parameters:

dataDirStates - output of storage directory states

Returns:

true if there is at least one valid formatted storage directory

Throws:

java.io.IOException

checkUpgrade

public static void checkUpgrade(NNStorage storage)
                         throws java.io.IOException

Check if upgrade is in progress.

Throws:

java.io.IOException

hasRollbackFSImage

public boolean hasRollbackFSImage()
                           throws java.io.IOException

Returns:

true if there is rollback fsimage (for rolling upgrade) in NameNode directory.

Throws:

java.io.IOException

getEditLog

public FSEditLog getEditLog()

initEditLog

public void initEditLog(HdfsServerConstants.StartupOption startOpt)
                 throws java.io.IOException

Throws:

java.io.IOException

loadEdits

public long loadEdits(java.lang.Iterable<EditLogInputStream> editStreams,
                      FSNamesystem target)
               throws java.io.IOException

Load the specified list of edit files into the image.

Throws:

java.io.IOException

loadEdits

public long loadEdits(java.lang.Iterable<EditLogInputStream> editStreams,
                      FSNamesystem target,
                      long maxTxnsToRead,
                      HdfsServerConstants.StartupOption startOpt,
                      MetaRecoveryContext recovery)
               throws java.io.IOException

Throws:

java.io.IOException

saveLegacyOIVImage

public void saveLegacyOIVImage(FSNamesystem source,
                               java.lang.String targetDir,
                               Canceler canceler)
                        throws java.io.IOException

Save FSimage in the legacy format. This is not for NN consumption, but for tools like OIV.

Throws:

java.io.IOException

updateStorageVersion

public void updateStorageVersion()
                          throws java.io.IOException

Update version of all storage directories.

Throws:

java.io.IOException

saveNamespace

public boolean saveNamespace(long timeWindow,
                             long txGap,
                             FSNamesystem source)
                      throws java.io.IOException

Parameters:

timeWindow - a checkpoint is done if the latest checkpoint was done more than this number of seconds ago.

txGap - a checkpoint is done also if the gap between the latest tx id and the latest checkpoint is greater than this number.

Returns:

true if a checkpoint has been made

Throws:

java.io.IOException

See Also:

saveNamespace(FSNamesystem, NameNodeFile, Canceler)

saveNamespace

public void saveNamespace(FSNamesystem source)
                   throws java.io.IOException

Throws:

java.io.IOException

saveNamespace

public void saveNamespace(FSNamesystem source,
                          NNStorage.NameNodeFile nnf,
                          Canceler canceler)
                   throws java.io.IOException

Save the contents of the FS image to a new image file in each of the current storage directories.

Throws:

java.io.IOException

saveFSImageInAllDirs

protected void saveFSImageInAllDirs(FSNamesystem source,
                                    long txid)
                             throws java.io.IOException

Throws:

java.io.IOException

See Also:

saveFSImageInAllDirs(FSNamesystem, NameNodeFile, long, Canceler)

addToCheckpointing

public boolean addToCheckpointing(long txid)

removeFromCheckpointing

public void removeFromCheckpointing(long txid)

saveDigestAndRenameCheckpointImage

public void saveDigestAndRenameCheckpointImage(NNStorage.NameNodeFile nnf,
                                               long txid,
                                               org.apache.hadoop.io.MD5Hash digest)
                                        throws java.io.IOException

This is called by the 2NN after having downloaded an image, and by the NN after having received a new image from the 2NN. It renames the image from fsimage_N.ckpt to fsimage_N and also saves the related .md5 file into place.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

getStorage

public NNStorage getStorage()

getLayoutVersion

public int getLayoutVersion()

getNamespaceID

public int getNamespaceID()

getClusterID

public java.lang.String getClusterID()

getBlockPoolID

public java.lang.String getBlockPoolID()

getLastAppliedTxId

public long getLastAppliedTxId()

getLastAppliedOrWrittenTxId

public long getLastAppliedOrWrittenTxId()

getCorrectLastAppliedOrWrittenTxId

public long getCorrectLastAppliedOrWrittenTxId()

This method holds a lock of FSEditLog to get the correct value. This method must not be used for metrics.

updateLastAppliedTxIdFromWritten

public void updateLastAppliedTxIdFromWritten()

getMostRecentCheckpointTxId

public long getMostRecentCheckpointTxId()