Package org.apache.hadoop.hdfs.server.namenode

Class NNStorage

java.lang.Object

org.apache.hadoop.hdfs.server.common.StorageInfo

org.apache.hadoop.hdfs.server.common.Storage

org.apache.hadoop.hdfs.server.namenode.NNStorage

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, StorageErrorReporter

@Private
public class NNStorage
extends Storage
implements java.io.Closeable, StorageErrorReporter

NNStorage is responsible for management of the StorageDirectories used by the NameNode.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	NNStorage.NameNodeDirType	Implementation of StorageDirType specific to namenode storage A Storage directory could be of type IMAGE which stores only fsimage, or of type EDITS which stores edits or of type IMAGE_AND_EDITS which stores both fsimage and edits.
static class	NNStorage.NameNodeFile	The filenames used for storing the images.

Nested classes/interfaces inherited from class org.apache.hadoop.hdfs.server.common.Storage

Storage.FormatConfirmable, Storage.StorageDirectory, Storage.StorageDirType, Storage.StorageState

Field Summary

Fields

Modifier and Type	Field	Description
protected java.lang.String	blockpoolID
protected long	mostRecentCheckpointTxId	TxId of the last transaction that was included in the most recent fsimage file.
protected java.util.List<Storage.StorageDirectory>	removedStorageDirs	List of failed (and thus removed) storages.

Fields inherited from class org.apache.hadoop.hdfs.server.common.Storage

LAST_PRE_UPGRADE_LAYOUT_VERSION, LAST_UPGRADABLE_HADOOP_VERSION, LAST_UPGRADABLE_LAYOUT_VERSION, LAYOUT_VERSIONS_203, LOG, STORAGE_1_BBW, STORAGE_DIR_CURRENT, STORAGE_DIR_PREVIOUS, STORAGE_FILE_LOCK, STORAGE_PREVIOUS_CKPT, STORAGE_TMP_FINALIZED, STORAGE_TMP_LAST_CKPT, STORAGE_TMP_PREVIOUS, STORAGE_TMP_REMOVED

Fields inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

clusterID, cTime, layoutVersion, namespaceID, STORAGE_FILE_VERSION, storageType

Constructor Summary

Constructors

Constructor	Description
NNStorage(org.apache.hadoop.conf.Configuration conf, java.util.Collection<java.net.URI> imageDirs, java.util.Collection<java.net.URI> editsDirs)	Construct the NNStorage.

Method Summary

Modifier and Type	Method	Description
void	close()
java.lang.String	determineClusterId()	Try to find current cluster id in the VERSION files.
void	format()
void	format(NamespaceInfo nsInfo)	Format all available storage directories.
void	format(NamespaceInfo nsInfo, boolean isRollingUpgrade)	Format all available storage directories.
java.lang.String	getBlockPoolID()
static java.lang.String	getCheckpointImageFileName(long txid)
static java.io.File	getFinalizedEditsFile(Storage.StorageDirectory sd, long startTxId, long endTxId)
static java.lang.String	getFinalizedEditsFileName(long startTxId, long endTxId)
java.io.File	getFsImage(long txid, java.util.EnumSet<NNStorage.NameNodeFile> nnfs)
java.io.File	getFsImageName(long txid)
java.io.File	getFsImageName(long txid, NNStorage.NameNodeFile nnf)
java.io.File[]	getFsImageNameCheckpoint(long txid)	Return the name of the image file that is uploaded by periodic checkpointing.
java.io.File	getHighestFsImageName()
static java.lang.String	getImageFileName(long txid)
static java.lang.String	getInProgressEditsFileName(long startTxId)
static java.lang.String	getLegacyOIVImageFileName(long txid)
long	getMostRecentCheckpointTxId()
NamespaceInfo	getNamespaceInfo()
java.lang.String	getNNDirectorySize()
static java.lang.String	getRollbackImageFileName(long txid)
Storage.StorageDirectory	getStorageDirectory(java.net.URI uri)	Return the storage directory corresponding to the passed URI.
static java.io.File	getTemporaryEditsFile(Storage.StorageDirectory sd, long startTxId, long endTxId, long timestamp)
static java.lang.String	getTemporaryEditsFileName(long startTxId, long endTxId, long timestamp)
boolean	isPreUpgradableLayout(Storage.StorageDirectory sd)	Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
static java.lang.String	newClusterID()	Generate new clusterID.
static NamespaceInfo	newNamespaceInfo()
void	reportErrorOnFile(java.io.File f)	Report that an IOE has occurred on some file which may or may not be within one of the NN image storage directories.
void	setBlockPoolID(java.lang.String bpid)	Validate and set block pool ID.
protected void	setFieldsFromProperties(java.util.Properties props, Storage.StorageDirectory sd)	Get common storage fields.
protected void	setPropertiesFromFields(java.util.Properties props, Storage.StorageDirectory sd)	Write version file into the storage directory.
void	updateNameDirSize()
void	writeAll()	Write all data storage files.
void	writeTransactionIdFileToStorage(long txid)	Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID.
void	writeTransactionIdFileToStorage(long txid, NNStorage.NameNodeDirType type)	Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID.

Methods inherited from class org.apache.hadoop.hdfs.server.common.Storage

addStorageDir, checkVersionUpgradable, confirmFormat, containsStorageDir, containsStorageDir, containsStorageDir, deleteDir, dirIterable, dirIterator, dirIterator, dirIterator, dirIterator, getBuildVersion, getFiles, getNumStorageDirs, getRegistrationID, getSingularStorageDir, getStorageDir, getStorageDirs, is203LayoutVersion, listStorageDirectories, nativeCopyFileUnbuffered, rename, unlockAll, writeProperties, writeProperties, writeProperties

Methods inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

checkStorageType, getClusterID, getClusterIdFromColonSeparatedString, getCTime, getLayoutVersion, getNamespaceID, getNsIdFromColonSeparatedString, getProperty, getServiceLayoutFeatureMap, getServiceLayoutVersion, readPreviousVersionProperties, readProperties, readPropertiesFile, setClusterId, setcTime, setLayoutVersion, setNamespaceID, setServiceLayoutVersion, setStorageInfo, toColonSeparatedString, toMapString, toString, versionSupportsFederation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

blockpoolID

protected java.lang.String blockpoolID

mostRecentCheckpointTxId

protected volatile long mostRecentCheckpointTxId

TxId of the last transaction that was included in the most recent fsimage file. This does not include any transactions that have since been written to the edit log.

removedStorageDirs

protected final java.util.List<Storage.StorageDirectory> removedStorageDirs

List of failed (and thus removed) storages.

Constructor Detail

NNStorage

public NNStorage(org.apache.hadoop.conf.Configuration conf,
                 java.util.Collection<java.net.URI> imageDirs,
                 java.util.Collection<java.net.URI> editsDirs)
          throws java.io.IOException

Construct the NNStorage.

Parameters:

conf - Namenode configuration.

imageDirs - Directories the image can be stored in.

editsDirs - Directories the editlog can be stored in.

Throws:

java.io.IOException - if any directories are inaccessible.

Method Detail

isPreUpgradableLayout

public boolean isPreUpgradableLayout(Storage.StorageDirectory sd)
                              throws java.io.IOException

Description copied from class: Storage

Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.

Specified by:

isPreUpgradableLayout in class Storage

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

getStorageDirectory

public Storage.StorageDirectory getStorageDirectory(java.net.URI uri)

Return the storage directory corresponding to the passed URI.

Parameters:

uri - URI of a storage directory

Returns:

The matching storage directory or null if none found

getMostRecentCheckpointTxId

public long getMostRecentCheckpointTxId()

Returns:

the transaction ID of the last checkpoint.

writeTransactionIdFileToStorage

public void writeTransactionIdFileToStorage(long txid)

Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID. This is used when the image is loaded to avoid accidental rollbacks in the case where an edit log is fully deleted but there is no checkpoint. See TestNameEditsConfigs.testNameEditsConfigsFailure()

Parameters:

txid - the txid that has been reached

writeTransactionIdFileToStorage

public void writeTransactionIdFileToStorage(long txid,
                                            NNStorage.NameNodeDirType type)

Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID. This is used when the image is loaded to avoid accidental rollbacks in the case where an edit log is fully deleted but there is no checkpoint. See TestNameEditsConfigs.testNameEditsConfigsFailure()

Parameters:

txid - the txid that has been reached

type - the type of directory

getFsImageNameCheckpoint

public java.io.File[] getFsImageNameCheckpoint(long txid)

Return the name of the image file that is uploaded by periodic checkpointing.

Returns:

List of filenames to save checkpoints to.

getFsImageName

public java.io.File getFsImageName(long txid,
                                   NNStorage.NameNodeFile nnf)

Returns:

The first image file with the given txid and image type.

getFsImage

public java.io.File getFsImage(long txid,
                               java.util.EnumSet<NNStorage.NameNodeFile> nnfs)

Returns:

The first image file whose txid is the same with the given txid and image type is one of the given types.

getFsImageName

public java.io.File getFsImageName(long txid)

getHighestFsImageName

public java.io.File getHighestFsImageName()

format

public void format(NamespaceInfo nsInfo)
            throws java.io.IOException

Format all available storage directories.

Throws:

java.io.IOException

format

public void format(NamespaceInfo nsInfo,
                   boolean isRollingUpgrade)
            throws java.io.IOException

Format all available storage directories.

Throws:

java.io.IOException

newNamespaceInfo

public static NamespaceInfo newNamespaceInfo()
                                      throws java.net.UnknownHostException

Throws:

java.net.UnknownHostException

format

public void format()
            throws java.io.IOException

Throws:

java.io.IOException

setFieldsFromProperties

protected void setFieldsFromProperties(java.util.Properties props,
                                       Storage.StorageDirectory sd)
                                throws java.io.IOException

Description copied from class: StorageInfo

Get common storage fields. Should be overloaded if additional fields need to be get.

Overrides:

setFieldsFromProperties in class StorageInfo

Parameters:

props - properties

Throws:

java.io.IOException - on error

setPropertiesFromFields

protected void setPropertiesFromFields(java.util.Properties props,
                                       Storage.StorageDirectory sd)
                                throws java.io.IOException

Write version file into the storage directory. The version file should always be written last. Missing or corrupted version file indicates that the checkpoint is not valid.

Overrides:

setPropertiesFromFields in class Storage

Parameters:

sd - storage directory

props - the Properties object to write into

Throws:

java.io.IOException

getCheckpointImageFileName

@VisibleForTesting
public static java.lang.String getCheckpointImageFileName(long txid)

getImageFileName

@VisibleForTesting
public static java.lang.String getImageFileName(long txid)

getRollbackImageFileName

@VisibleForTesting
public static java.lang.String getRollbackImageFileName(long txid)

getLegacyOIVImageFileName

public static java.lang.String getLegacyOIVImageFileName(long txid)

getInProgressEditsFileName

@VisibleForTesting
public static java.lang.String getInProgressEditsFileName(long startTxId)

getFinalizedEditsFile

public static java.io.File getFinalizedEditsFile(Storage.StorageDirectory sd,
                                                 long startTxId,
                                                 long endTxId)

getTemporaryEditsFile

public static java.io.File getTemporaryEditsFile(Storage.StorageDirectory sd,
                                                 long startTxId,
                                                 long endTxId,
                                                 long timestamp)

getFinalizedEditsFileName

@VisibleForTesting
public static java.lang.String getFinalizedEditsFileName(long startTxId,
                                                         long endTxId)

getTemporaryEditsFileName

public static java.lang.String getTemporaryEditsFileName(long startTxId,
                                                         long endTxId,
                                                         long timestamp)

reportErrorOnFile

public void reportErrorOnFile(java.io.File f)

Report that an IOE has occurred on some file which may or may not be within one of the NN image storage directories.

Specified by:

reportErrorOnFile in interface StorageErrorReporter

Parameters:

f - the file which had an error.

newClusterID

public static java.lang.String newClusterID()

Generate new clusterID. clusterID is a persistent attribute of the cluster. It is generated when the cluster is created and remains the same during the life cycle of the cluster. When a new name node is formated, if this is a new cluster, a new clusterID is geneated and stored. Subsequent name node must be given the same ClusterID during its format to be in the same cluster. When a datanode register it receive the clusterID and stick with it. If at any point, name node or data node tries to join another cluster, it will be rejected.

Returns:

new clusterID

determineClusterId

public java.lang.String determineClusterId()

Try to find current cluster id in the VERSION files. returns first cluster id found in any VERSION file null in case none found

Returns:

clusterId or null in case no cluster id found

setBlockPoolID

public void setBlockPoolID(java.lang.String bpid)

Validate and set block pool ID.

getBlockPoolID

public java.lang.String getBlockPoolID()

getNamespaceInfo

public NamespaceInfo getNamespaceInfo()

Overrides:

getNamespaceInfo in class Storage

getNNDirectorySize

public java.lang.String getNNDirectorySize()

updateNameDirSize

public void updateNameDirSize()

writeAll

public void writeAll()
              throws java.io.IOException

Write all data storage files.

Overrides:

writeAll in class Storage

Throws:

java.io.IOException - When all the storage directory fails to write VERSION file