Storage (Apache Hadoop HDFS 3.4.1.0-eep-940 API)

java.lang.Object
- org.apache.hadoop.hdfs.server.common.StorageInfo
- - org.apache.hadoop.hdfs.server.common.Storage

Direct Known Subclasses:

BlockPoolSliceStorage, DataStorage, NNStorage
```
@Private
public abstract class Storage
extends StorageInfo
```
Storage information file.
Local storage information is stored in a separate file VERSION. It contains type of the node, the storage layout version, the namespace id, and the fs state creation time.
Local storage can reside in multiple directories. Each directory should contain the same VERSION file as the others. During startup Hadoop servers (name-node and data-nodes) read their local storage information from them.
The servers hold a lock for each storage directory while they run so that other nodes were not able to startup sharing the same storage. The locks are released when the servers stop (normally or abnormally).

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`static interface`	`Storage.FormatConfirmable`	Interface for classes which need to have the user confirm their formatting during NameNode -format and other similar operations.
`static class`	`Storage.StorageDirectory`	One of the storage directories.
`static interface`	`Storage.StorageDirType`	An interface to denote storage directory type Implementations can define a type for storage directory by implementing this interface.
`static class`	`Storage.StorageState`

Field Summary

Fields
Modifier and Type	Field	Description
`static int`	`LAST_PRE_UPGRADE_LAYOUT_VERSION`
`protected static java.lang.String`	`LAST_UPGRADABLE_HADOOP_VERSION`
`static int`	`LAST_UPGRADABLE_LAYOUT_VERSION`
`static int[]`	`LAYOUT_VERSIONS_203`	Layout versions of 0.20.203 release
`static org.slf4j.Logger`	`LOG`
`static java.lang.String`	`STORAGE_1_BBW`	The blocksBeingWritten directory which was used in some 1.x and earlier releases.
`static java.lang.String`	`STORAGE_DIR_CURRENT`
`static java.lang.String`	`STORAGE_DIR_PREVIOUS`
`static java.lang.String`	`STORAGE_FILE_LOCK`
`static java.lang.String`	`STORAGE_PREVIOUS_CKPT`
`static java.lang.String`	`STORAGE_TMP_FINALIZED`
`static java.lang.String`	`STORAGE_TMP_LAST_CKPT`
`static java.lang.String`	`STORAGE_TMP_PREVIOUS`
`static java.lang.String`	`STORAGE_TMP_REMOVED`

Fields inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo
clusterID, cTime, layoutVersion, namespaceID, STORAGE_FILE_VERSION, storageType

Constructor Summary

Constructors
Modifier	Constructor	Description
`protected`	`Storage(HdfsServerConstants.NodeType type)`	Create empty storage info of the specified type
`protected`	`Storage(StorageInfo storageInfo)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method	Description
`protected void`	`addStorageDir(Storage.StorageDirectory sd)`
`static void`	`checkVersionUpgradable(int oldVersion)`	Checks if the upgrade from `oldVersion` is supported.
`static boolean`	`confirmFormat(java.lang.Iterable<? extends Storage.FormatConfirmable> items, boolean force, boolean interactive)`	Iterate over each of the `Storage.FormatConfirmable` objects, potentially checking with the user whether it should be formatted.
`protected boolean`	`containsStorageDir(java.io.File root)`	Returns true if the storage directory on the given directory is already loaded.
`protected boolean`	`containsStorageDir(StorageLocation location)`	Returns true if the storage directory on the given directory is already loaded.
`protected boolean`	`containsStorageDir(StorageLocation location, java.lang.String bpid)`	Returns true if the storage directory on the given location is already loaded.
`static void`	`deleteDir(java.io.File dir)`	Recursively delete all the content of the directory first and then the directory itself from the local filesystem.
`java.lang.Iterable<Storage.StorageDirectory>`	`dirIterable(Storage.StorageDirType dirType)`
`java.util.Iterator<Storage.StorageDirectory>`	`dirIterator()`	Return default iterator This iterator returns all entries in storageDirs
`java.util.Iterator<Storage.StorageDirectory>`	`dirIterator(boolean includeShared)`	Return all entries in storageDirs, potentially excluding shared dirs.
`java.util.Iterator<Storage.StorageDirectory>`	`dirIterator(Storage.StorageDirType dirType)`	Return iterator based on Storage Directory Type This iterator selects entries in storageDirs of type dirType and returns them via the Iterator
`java.util.Iterator<Storage.StorageDirectory>`	`dirIterator(Storage.StorageDirType dirType, boolean includeShared)`
`static java.lang.String`	`getBuildVersion()`
`java.util.List<java.io.File>`	`getFiles(Storage.StorageDirType dirType, java.lang.String fileName)`
`NamespaceInfo`	`getNamespaceInfo()`
`int`	`getNumStorageDirs()`
`static java.lang.String`	`getRegistrationID(StorageInfo storage)`
`Storage.StorageDirectory`	`getSingularStorageDir()`
`Storage.StorageDirectory`	`getStorageDir(int idx)`
`java.util.List<Storage.StorageDirectory>`	`getStorageDirs()`
`static boolean`	`is203LayoutVersion(int layoutVersion)`
`abstract boolean`	`isPreUpgradableLayout(Storage.StorageDirectory sd)`	Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
`java.lang.String`	`listStorageDirectories()`	generate storage list (debug line)
`static void`	`nativeCopyFileUnbuffered(java.io.File srcFile, java.io.File destFile, boolean preserveFileDate)`	Copies a file (usually large) to a new location using native unbuffered IO.
`static void`	`rename(java.io.File from, java.io.File to)`
`protected void`	`setPropertiesFromFields(java.util.Properties props, Storage.StorageDirectory sd)`	Set common storage fields into the given properties object.
`void`	`unlockAll()`	Unlock all storage directories.
`void`	`writeAll()`	Write all data storage files.
`static void`	`writeProperties(java.io.File to, java.util.Properties props)`
`void`	`writeProperties(java.io.File to, Storage.StorageDirectory sd)`
`void`	`writeProperties(Storage.StorageDirectory sd)`	Write properties to the VERSION file in the given storage directory.

Methods inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo
checkStorageType, getClusterID, getClusterIdFromColonSeparatedString, getCTime, getLayoutVersion, getNamespaceID, getNsIdFromColonSeparatedString, getProperty, getServiceLayoutFeatureMap, getServiceLayoutVersion, readPreviousVersionProperties, readProperties, readPropertiesFile, setClusterId, setcTime, setFieldsFromProperties, setLayoutVersion, setNamespaceID, setServiceLayoutVersion, setStorageInfo, toColonSeparatedString, toMapString, toString, versionSupportsFederation

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - LOG
```
public static final org.slf4j.Logger LOG
```
  - LAST_PRE_UPGRADE_LAYOUT_VERSION
```
public static final int LAST_PRE_UPGRADE_LAYOUT_VERSION
```
    See Also:
    
    Constant Field Values
  - LAST_UPGRADABLE_LAYOUT_VERSION
```
public static final int LAST_UPGRADABLE_LAYOUT_VERSION
```
    See Also:
    
    Constant Field Values
  - LAST_UPGRADABLE_HADOOP_VERSION
```
protected static final java.lang.String LAST_UPGRADABLE_HADOOP_VERSION
```
    See Also:
    
    Constant Field Values
  - LAYOUT_VERSIONS_203
```
public static final int[] LAYOUT_VERSIONS_203
```
    Layout versions of 0.20.203 release
  - STORAGE_FILE_LOCK
```
public static final java.lang.String STORAGE_FILE_LOCK
```
    See Also:
    
    Constant Field Values
  - STORAGE_DIR_CURRENT
```
public static final java.lang.String STORAGE_DIR_CURRENT
```
    See Also:
    
    Constant Field Values
  - STORAGE_DIR_PREVIOUS
```
public static final java.lang.String STORAGE_DIR_PREVIOUS
```
    See Also:
    
    Constant Field Values
  - STORAGE_TMP_REMOVED
```
public static final java.lang.String STORAGE_TMP_REMOVED
```
    See Also:
    
    Constant Field Values
  - STORAGE_TMP_PREVIOUS
```
public static final java.lang.String STORAGE_TMP_PREVIOUS
```
    See Also:
    
    Constant Field Values
  - STORAGE_TMP_FINALIZED
```
public static final java.lang.String STORAGE_TMP_FINALIZED
```
    See Also:
    
    Constant Field Values
  - STORAGE_TMP_LAST_CKPT
```
public static final java.lang.String STORAGE_TMP_LAST_CKPT
```
    See Also:
    
    Constant Field Values
  - STORAGE_PREVIOUS_CKPT
```
public static final java.lang.String STORAGE_PREVIOUS_CKPT
```
    See Also:
    
    Constant Field Values
  - STORAGE_1_BBW
```
public static final java.lang.String STORAGE_1_BBW
```
    The blocksBeingWritten directory which was used in some 1.x and earlier releases.
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - Storage
```
protected Storage(HdfsServerConstants.NodeType type)
```
    Create empty storage info of the specified type
  - Storage
```
protected Storage(StorageInfo storageInfo)
```
- Method Detail
  - getFiles
```
public java.util.List<java.io.File> getFiles(Storage.StorageDirType dirType,
                                             java.lang.String fileName)
```
    Returns:
    
    A list of the given File in every available storage directory, regardless of whether it might exist.
  - dirIterator
```
public java.util.Iterator<Storage.StorageDirectory> dirIterator()
```
    Return default iterator This iterator returns all entries in storageDirs
  - dirIterator
```
public java.util.Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType)
```
    Return iterator based on Storage Directory Type This iterator selects entries in storageDirs of type dirType and returns them via the Iterator
  - dirIterator
```
public java.util.Iterator<Storage.StorageDirectory> dirIterator(boolean includeShared)
```
    Return all entries in storageDirs, potentially excluding shared dirs.
    
    Parameters:
    
    includeShared - whether or not to include shared dirs.
    
    Returns:
    
    an iterator over the configured storage dirs.
  - dirIterator
```
public java.util.Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType,
                                                                boolean includeShared)
```
    Parameters:
    
    dirType - all entries will be of this type of dir
    
    includeShared - true to include any shared directories, false otherwise
    
    Returns:
    
    an iterator over the configured storage dirs.
  - dirIterable
```
public java.lang.Iterable<Storage.StorageDirectory> dirIterable(Storage.StorageDirType dirType)
```
  - listStorageDirectories
```
public java.lang.String listStorageDirectories()
```
    generate storage list (debug line)
  - getNumStorageDirs
```
public int getNumStorageDirs()
```
  - getStorageDirs
```
public java.util.List<Storage.StorageDirectory> getStorageDirs()
```
  - getStorageDir
```
public Storage.StorageDirectory getStorageDir(int idx)
```
  - getSingularStorageDir
```
public Storage.StorageDirectory getSingularStorageDir()
```
    Returns:
    
    the storage directory, with the precondition that this storage has exactly one storage directory
  - addStorageDir
```
protected void addStorageDir(Storage.StorageDirectory sd)
```
  - containsStorageDir
```
protected boolean containsStorageDir(java.io.File root)
                              throws java.io.IOException
```
    Returns true if the storage directory on the given directory is already loaded.
    
    Parameters:
    
    root - the root directory of a Storage.StorageDirectory
    
    Throws:
    
    java.io.IOException - if failed to get canonical path.
  - containsStorageDir
```
protected boolean containsStorageDir(StorageLocation location)
                              throws java.io.IOException
```
    Returns true if the storage directory on the given directory is already loaded.
    
    Parameters:
    
    location - the StorageLocation
    
    Throws:
    
    java.io.IOException - if failed to get canonical path.
  - containsStorageDir
```
protected boolean containsStorageDir(StorageLocation location,
                                     java.lang.String bpid)
                              throws java.io.IOException
```
    Returns true if the storage directory on the given location is already loaded.
    
    Parameters:
    
    location - the StorageLocation
    
    bpid - the block pool id
    
    Returns:
    
    true if the location matches to any existing storage directories
    
    Throws:
    
    java.io.IOException - IOException if failed to read location or storage directory path
  - getNamespaceInfo
```
public NamespaceInfo getNamespaceInfo()
```
  - isPreUpgradableLayout
```
public abstract boolean isPreUpgradableLayout(Storage.StorageDirectory sd)
                                       throws java.io.IOException
```
    Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
    
    Throws:
    
    java.io.IOException
  - checkVersionUpgradable
```
public static void checkVersionUpgradable(int oldVersion)
                                   throws java.io.IOException
```
    Checks if the upgrade from oldVersion is supported.
    
    Parameters:
    
    oldVersion - the version of the metadata to check with the current version
    
    Throws:
    
    java.io.IOException - if upgrade is not supported
  - confirmFormat
```
public static boolean confirmFormat(java.lang.Iterable<? extends Storage.FormatConfirmable> items,
                                    boolean force,
                                    boolean interactive)
                             throws java.io.IOException
```
    Iterate over each of the Storage.FormatConfirmable objects, potentially checking with the user whether it should be formatted. If running in interactive mode, will prompt the user for each directory to allow them to format anyway. Otherwise, returns false, unless 'force' is specified.
    
    Parameters:
    
    force - format regardless of whether dirs exist
    
    interactive - prompt the user when a dir exists
    
    Returns:
    
    true if formatting should proceed
    
    Throws:
    
    java.io.IOException - if some storage cannot be accessed
  - setPropertiesFromFields
```
protected void setPropertiesFromFields(java.util.Properties props,
                                       Storage.StorageDirectory sd)
                                throws java.io.IOException
```
    Set common storage fields into the given properties object. Should be overloaded if additional fields need to be set.
    
    Parameters:
    
    props - the Properties object to write into
    
    Throws:
    
    java.io.IOException
  - writeProperties
```
public void writeProperties(Storage.StorageDirectory sd)
                     throws java.io.IOException
```
    Write properties to the VERSION file in the given storage directory.
    
    Throws:
    
    java.io.IOException
  - writeProperties
```
public void writeProperties(java.io.File to,
                            Storage.StorageDirectory sd)
                     throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - writeProperties
```
public static void writeProperties(java.io.File to,
                                   java.util.Properties props)
                            throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - rename
```
public static void rename(java.io.File from,
                          java.io.File to)
                   throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - nativeCopyFileUnbuffered
```
public static void nativeCopyFileUnbuffered(java.io.File srcFile,
                                            java.io.File destFile,
                                            boolean preserveFileDate)
                                     throws java.io.IOException
```
    Copies a file (usually large) to a new location using native unbuffered IO.
    This method copies the contents of the specified source file to the specified destination file using OS specific unbuffered IO. The goal is to avoid churning the file system buffer cache when copying large files. We can't use FileUtils#copyFile from apache-commons-io because it is a buffered IO based on FileChannel#transferFrom, which uses MmapByteBuffer internally. The directory holding the destination file is created if it does not exist. If the destination file exists, then this method will delete it first.
    Note: Setting preserveFileDate to true tries to preserve the file's last modified date/times using File.setLastModified(long), however it is not guaranteed that the operation will succeed. If the modification operation fails, no indication is provided.
    
    Parameters:
    
    srcFile - an existing file to copy, must not be null
    
    destFile - the new file, must not be null
    
    preserveFileDate - true if the file date of the copy should be the same as the original
    
    Throws:
    
    java.lang.NullPointerException - if source or destination is null
    
    java.io.IOException - if source or destination is invalid
    
    java.io.IOException - if an IO error occurs during copying
  - deleteDir
```
public static void deleteDir(java.io.File dir)
                      throws java.io.IOException
```
    Recursively delete all the content of the directory first and then the directory itself from the local filesystem.
    
    Parameters:
    
    dir - The directory to delete
    
    Throws:
    
    java.io.IOException
  - writeAll
```
public void writeAll()
              throws java.io.IOException
```
    Write all data storage files.
    
    Throws:
    
    java.io.IOException
  - unlockAll
```
public void unlockAll()
               throws java.io.IOException
```
    Unlock all storage directories.
    
    Throws:
    
    java.io.IOException
  - getBuildVersion
```
public static java.lang.String getBuildVersion()
```
  - getRegistrationID
```
public static java.lang.String getRegistrationID(StorageInfo storage)
```
  - is203LayoutVersion
```
public static boolean is203LayoutVersion(int layoutVersion)
```

Class Storage

Nested Class Summary

Field Summary

Fields inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

Methods inherited from class java.lang.Object

Field Detail

LOG

LAST_PRE_UPGRADE_LAYOUT_VERSION

LAST_UPGRADABLE_LAYOUT_VERSION

LAST_UPGRADABLE_HADOOP_VERSION

LAYOUT_VERSIONS_203

STORAGE_FILE_LOCK

STORAGE_DIR_CURRENT

STORAGE_DIR_PREVIOUS

STORAGE_TMP_REMOVED

STORAGE_TMP_PREVIOUS

STORAGE_TMP_FINALIZED

STORAGE_TMP_LAST_CKPT

STORAGE_PREVIOUS_CKPT

STORAGE_1_BBW

Constructor Detail

Storage

Storage

Method Detail

getFiles

dirIterator

dirIterator

dirIterator

dirIterator

dirIterable

listStorageDirectories

getNumStorageDirs

getStorageDirs

getStorageDir

getSingularStorageDir

addStorageDir

containsStorageDir

containsStorageDir

containsStorageDir

getNamespaceInfo

isPreUpgradableLayout

checkVersionUpgradable

confirmFormat

setPropertiesFromFields

writeProperties

writeProperties

writeProperties

rename

nativeCopyFileUnbuffered

deleteDir

writeAll

unlockAll

getBuildVersion

getRegistrationID

is203LayoutVersion