LocalDirAllocator (Apache Hadoop Common 3.4.1.100-eep-941 API)

java.lang.Object
- org.apache.hadoop.fs.LocalDirAllocator

```
@LimitedPrivate({"HDFS","MapReduce"})
@Unstable
public class LocalDirAllocator
extends java.lang.Object
```
An implementation of a round-robin scheme for disk allocation for creating files. The way it works is that it is kept track what disk was last allocated for a file write. For the current request, the next disk from the set of disks would be allocated if the free space on the disk is sufficient enough to accommodate the file that is being considered for creation. If the space requirements cannot be met, the next disk in order would be tried and so on till a disk is found with sufficient capacity. Once a disk with sufficient space is identified, a check is done to make sure that the disk is writable. Also, there is an API provided that doesn't take the space requirements into consideration but just checks whether the disk under consideration is writable (this should be used for cases where the file size is not known apriori). An API is provided to read a path that was created earlier. That API works by doing a scan of all the disks for the input pathname. This implementation also provides the functionality of having multiple allocators per JVM (one for each unique functionality or context, like mapred, dfs-client, etc.). It ensures that there is only one instance of an allocator per context per JVM. Note: 1. The contexts referred above are actually the configuration items defined in the Configuration class like "mapred.local.dir" (for which we want to control the dir allocations). The context-strings are exactly those configuration items. 2. This implementation does not take into consideration cases where a disk becomes read-only or goes out of space while a file is being written to (disks are shared between multiple processes, and so the latter situation is probable). 3. In the class implementation, "Disk" is referred to as "Dir", which actually points to the configured directory on the Disk which will be the parent for all file write/read allocations.

Field Summary

Fields
Modifier and Type Field Description

static int SIZE_UNKNOWN
Used when size of file to be allocated is unknown.

Fields
Modifier and Type	Field	Description
`static int`	`SIZE_UNKNOWN`	Used when size of file to be allocated is unknown.

Constructor Summary

Constructors
Constructor	Description
`LocalDirAllocator(java.lang.String contextCfgItemName)`	Create an allocator object.
`LocalDirAllocator(java.lang.String contextCfgItemName, DiskValidator diskValidator)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method	Description
`java.io.File`	`createTmpFileForWrite(java.lang.String pathStr, long size, Configuration conf)`	Creates a temporary file in the local FS.
`java.lang.Iterable<Path>`	`getAllLocalPathsToRead(java.lang.String pathStr, Configuration conf)`	Get all of the paths that currently exist in the working directories.
`Path`	`getLocalPathForWrite(java.lang.String pathStr, long size, Configuration conf)`	Get a path from the local FS.
`Path`	`getLocalPathForWrite(java.lang.String pathStr, long size, Configuration conf, boolean checkWrite)`	Get a path from the local FS.
`Path`	`getLocalPathForWrite(java.lang.String pathStr, Configuration conf)`	Get a path from the local FS.
`Path`	`getLocalPathToRead(java.lang.String pathStr, Configuration conf)`	Get a path from the local FS for reading.
`boolean`	`ifExists(java.lang.String pathStr, Configuration conf)`	We search through all the configured dirs for the file's existence and return true when we find.
`static boolean`	`isContextValid(java.lang.String contextCfgItemName)`	Method to check whether a context is valid.
`static void`	`removeContext(java.lang.String contextCfgItemName)`	Deprecated.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - SIZE_UNKNOWN
```
public static final int SIZE_UNKNOWN
```
    Used when size of file to be allocated is unknown.
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - LocalDirAllocator
```
public LocalDirAllocator(java.lang.String contextCfgItemName)
```
    Create an allocator object.
    
    Parameters:
    
    contextCfgItemName - contextCfgItemName.
  - LocalDirAllocator
```
public LocalDirAllocator(java.lang.String contextCfgItemName,
                         DiskValidator diskValidator)
```
- Method Detail
  - getLocalPathForWrite
```
public Path getLocalPathForWrite(java.lang.String pathStr,
                                 Configuration conf)
                          throws java.io.IOException
```
    Get a path from the local FS. This method should be used if the size of the file is not known apriori. We go round-robin over the set of disks (via the configured dirs) and return the first complete path where we could create the parent directory of the passed path.
    
    Parameters:
    
    pathStr - the requested path (this will be created on the first available disk)
    
    conf - the Configuration object
    
    Returns:
    
    the complete path to the file on a local disk
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - getLocalPathForWrite
```
public Path getLocalPathForWrite(java.lang.String pathStr,
                                 long size,
                                 Configuration conf)
                          throws java.io.IOException
```
    Get a path from the local FS. Pass size as SIZE_UNKNOWN if not known apriori. We round-robin over the set of disks (via the configured dirs) and return the first complete path which has enough space
    
    Parameters:
    
    pathStr - the requested path (this will be created on the first available disk)
    
    size - the size of the file that is going to be written
    
    conf - the Configuration object
    
    Returns:
    
    the complete path to the file on a local disk
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - getLocalPathForWrite
```
public Path getLocalPathForWrite(java.lang.String pathStr,
                                 long size,
                                 Configuration conf,
                                 boolean checkWrite)
                          throws java.io.IOException
```
    Get a path from the local FS. Pass size as SIZE_UNKNOWN if not known apriori. We round-robin over the set of disks (via the configured dirs) and return the first complete path which has enough space
    
    Parameters:
    
    pathStr - the requested path (this will be created on the first available disk)
    
    size - the size of the file that is going to be written
    
    conf - the Configuration object
    
    checkWrite - ensure that the path is writable
    
    Returns:
    
    the complete path to the file on a local disk
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - getLocalPathToRead
```
public Path getLocalPathToRead(java.lang.String pathStr,
                               Configuration conf)
                        throws java.io.IOException
```
    Get a path from the local FS for reading. We search through all the configured dirs for the file's existence and return the complete path to the file when we find one
    
    Parameters:
    
    pathStr - the requested file (this will be searched)
    
    conf - the Configuration object
    
    Returns:
    
    the complete path to the file on a local disk
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - getAllLocalPathsToRead
```
public java.lang.Iterable<Path> getAllLocalPathsToRead(java.lang.String pathStr,
                                                       Configuration conf)
                                                throws java.io.IOException
```
    Get all of the paths that currently exist in the working directories.
    
    Parameters:
    
    pathStr - the path underneath the roots
    
    conf - the configuration to look up the roots in
    
    Returns:
    
    all of the paths that exist under any of the roots
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - createTmpFileForWrite
```
public java.io.File createTmpFileForWrite(java.lang.String pathStr,
                                          long size,
                                          Configuration conf)
                                   throws java.io.IOException
```
    Creates a temporary file in the local FS. Pass size as -1 if not known apriori. We round-robin over the set of disks (via the configured dirs) and select the first complete path which has enough space. A file is created on this directory. The file is guaranteed to go away when the JVM exits.
    
    Parameters:
    
    pathStr - prefix for the temporary file
    
    size - the size of the file that is going to be written
    
    conf - the Configuration object
    
    Returns:
    
    a unique temporary file
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - isContextValid
```
public static boolean isContextValid(java.lang.String contextCfgItemName)
```
    Method to check whether a context is valid.
    
    Parameters:
    
    contextCfgItemName - contextCfgItemName.
    
    Returns:
    
    true/false
  - removeContext
```
@Deprecated
@LimitedPrivate("MapReduce")
public static void removeContext(java.lang.String contextCfgItemName)
```
    Deprecated.
    
    Removes the context from the context config items.
    
    Parameters:
    
    contextCfgItemName - contextCfgItemName.
  - ifExists
```
public boolean ifExists(java.lang.String pathStr,
                        Configuration conf)
```
    We search through all the configured dirs for the file's existence and return true when we find.
    
    Parameters:
    
    pathStr - the requested file (this will be searched)
    
    conf - the Configuration object
    
    Returns:
    
    true if files exist. false otherwise

Class LocalDirAllocator

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

SIZE_UNKNOWN

Constructor Detail

LocalDirAllocator

LocalDirAllocator

Method Detail

getLocalPathForWrite

getLocalPathForWrite

getLocalPathForWrite

getLocalPathToRead

getAllLocalPathsToRead

createTmpFileForWrite

isContextValid

removeContext

ifExists