Package org.apache.hadoop.hdfs.server.datanode

Class FileIoProvider

java.lang.Object

org.apache.hadoop.hdfs.server.datanode.FileIoProvider

@Private
@Unstable
public class FileIoProvider
extends java.lang.Object

This class abstracts out various file IO operations performed by the DataNode and invokes profiling (for collecting stats) and fault injection (for testing) event hooks before and after each file IO. Behavior can be injected into these events by enabling the profiling and/or fault injection event hooks through DFSConfigKeys.DFS_DATANODE_FILEIO_PROFILING_SAMPLING_PERCENTAGE_KEY and DFSConfigKeys.DFS_DATANODE_ENABLE_FILEIO_FAULT_INJECTION_KEY. These event hooks are disabled by default. Most functions accept an optional FsVolumeSpi parameter for instrumentation/logging. Some methods may look redundant, especially the multiple variations of move/rename/list. They exist to retain behavior compatibility for existing code.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FileIoProvider.OPERATION	Lists the types of file system operations.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
FileIoProvider(org.apache.hadoop.conf.Configuration conf, DataNode datanode)

Method Summary

Modifier and Type	Method	Description
boolean	createFile(FsVolumeSpi volume, java.io.File f)	Create a file.
boolean	delete(FsVolumeSpi volume, java.io.File f)	Delete a file.
boolean	deleteWithExistsCheck(FsVolumeSpi volume, java.io.File f)	Delete a file, first checking to see if it exists.
void	dirSync(FsVolumeSpi volume, java.io.File dir)	Sync the given directory changes to durable device.
boolean	exists(FsVolumeSpi volume, java.io.File f)	Check for file existence using File.exists().
void	flush(FsVolumeSpi volume, java.io.Flushable f)	See Flushable.flush().
boolean	fullyDelete(FsVolumeSpi volume, java.io.File dir)	Delete the given directory using FileUtil.fullyDelete(File).
java.io.FileInputStream	getFileInputStream(FsVolumeSpi volume, java.io.File f)	Create a FileInputStream using FileInputStream(File).
java.io.FileOutputStream	getFileOutputStream(FsVolumeSpi volume, java.io.File f)	Create a FileOutputStream using FileOutputStream(File, boolean).
java.io.FileOutputStream	getFileOutputStream(FsVolumeSpi volume, java.io.FileDescriptor fd)	Create a FileOutputStream using FileOutputStream(FileDescriptor).
java.io.FileOutputStream	getFileOutputStream(FsVolumeSpi volume, java.io.File f, boolean append)	Create a FileOutputStream using FileOutputStream(File, boolean).
int	getHardLinkCount(FsVolumeSpi volume, java.io.File f)	Retrieves the number of links to the specified file.
org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents	getProfilingEventHook()
java.io.RandomAccessFile	getRandomAccessFile(FsVolumeSpi volume, java.io.File f, java.lang.String mode)	Create a RandomAccessFile using RandomAccessFile(File, String).
java.io.FileInputStream	getShareDeleteFileInputStream(FsVolumeSpi volume, java.io.File f, long offset)	Create a FileInputStream using NativeIO.getShareDeleteFileDescriptor(java.io.File, long).
java.lang.String[]	list(FsVolumeSpi volume, java.io.File dir)	Get a listing of the given directory using FileUtil.listFiles(File).
java.util.List<java.lang.String>	listDirectory(FsVolumeSpi volume, java.io.File dir, java.io.FilenameFilter filter)	Get a listing of the given directory using IOUtils.listDirectory(File, FilenameFilter).
java.io.File[]	listFiles(FsVolumeSpi volume, java.io.File dir)	Get a listing of the given directory using FileUtil.listFiles(File).
boolean	mkdirs(FsVolumeSpi volume, java.io.File dir)	See File.mkdirs().
void	mkdirsWithExistsCheck(FsVolumeSpi volume, java.io.File dir)	Create the target directory using File.mkdirs() only if it doesn't exist already.
void	move(FsVolumeSpi volume, java.nio.file.Path src, java.nio.file.Path target, java.nio.file.CopyOption... options)	Move the src file to the target using Files.move(Path, Path, CopyOption...).
void	moveFile(FsVolumeSpi volume, java.io.File src, java.io.File target)	Move the src file to the target using FileUtils.moveFile(File, File).
void	nativeCopyFileUnbuffered(FsVolumeSpi volume, java.io.File src, java.io.File target, boolean preserveFileDate)	See Storage.nativeCopyFileUnbuffered(File, File, boolean).
java.io.FileInputStream	openAndSeek(FsVolumeSpi volume, java.io.File f, long offset)	Create a FileInputStream using FileInputStream(File) and position it at the given offset.
void	posixFadvise(FsVolumeSpi volume, java.lang.String identifier, java.io.FileDescriptor outFd, long offset, long length, int flags)	Call posix_fadvise on the given file descriptor.
void	rename(FsVolumeSpi volume, java.io.File src, java.io.File target)	Move the src file to the target using Storage.rename(File, File).
void	replaceFile(FsVolumeSpi volume, java.io.File src, java.io.File target)	Move the src file to the target using FileUtil.replaceFile(File, File).
void	sync(FsVolumeSpi volume, java.io.FileOutputStream fos)	Sync the given FileOutputStream.
void	syncFileRange(FsVolumeSpi volume, java.io.FileDescriptor outFd, long offset, long numBytes, int flags)	Call sync_file_range on the given file descriptor.
void	transferToSocketFully(FsVolumeSpi volume, org.apache.hadoop.net.SocketOutputStream sockOut, java.nio.channels.FileChannel fileCh, long position, int count, org.apache.hadoop.io.LongWritable waitTime, org.apache.hadoop.io.LongWritable transferTime)	Transfer data from a FileChannel to a SocketOutputStream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

FileIoProvider

public FileIoProvider(@Nullable
                      org.apache.hadoop.conf.Configuration conf,
                      DataNode datanode)

Parameters:

conf - Configuration object. May be null. When null, the event handlers are no-ops.

datanode - datanode that owns this FileIoProvider. Used for IO error based volume checker callback

Method Detail

flush

public void flush(@Nullable
                  FsVolumeSpi volume,
                  java.io.Flushable f)
           throws java.io.IOException

See Flushable.flush().

Parameters:

volume - target volume. null if unavailable.

Throws:

java.io.IOException

sync

public void sync(@Nullable
                 FsVolumeSpi volume,
                 java.io.FileOutputStream fos)
          throws java.io.IOException

Sync the given FileOutputStream.

Parameters:

volume - target volume. null if unavailable.

Throws:

java.io.IOException

dirSync

public void dirSync(@Nullable
                    FsVolumeSpi volume,
                    java.io.File dir)
             throws java.io.IOException

Sync the given directory changes to durable device.

Throws:

java.io.IOException

syncFileRange

public void syncFileRange(@Nullable
                          FsVolumeSpi volume,
                          java.io.FileDescriptor outFd,
                          long offset,
                          long numBytes,
                          int flags)
                   throws org.apache.hadoop.io.nativeio.NativeIOException

Call sync_file_range on the given file descriptor.

Parameters:

volume - target volume. null if unavailable.

Throws:

org.apache.hadoop.io.nativeio.NativeIOException

posixFadvise

public void posixFadvise(@Nullable
                         FsVolumeSpi volume,
                         java.lang.String identifier,
                         java.io.FileDescriptor outFd,
                         long offset,
                         long length,
                         int flags)
                  throws org.apache.hadoop.io.nativeio.NativeIOException

Call posix_fadvise on the given file descriptor.

Parameters:

volume - target volume. null if unavailable.

Throws:

org.apache.hadoop.io.nativeio.NativeIOException

delete

public boolean delete(@Nullable
                      FsVolumeSpi volume,
                      java.io.File f)

Delete a file.

Parameters:

volume - target volume. null if unavailable.

f - File to delete.

Returns:

true if the file was successfully deleted.

deleteWithExistsCheck

public boolean deleteWithExistsCheck(@Nullable
                                     FsVolumeSpi volume,
                                     java.io.File f)

Delete a file, first checking to see if it exists.

Parameters:

volume - target volume. null if unavailable.

f - File to delete

Returns:

true if the file was successfully deleted or if it never existed.

transferToSocketFully

public void transferToSocketFully(@Nullable
                                  FsVolumeSpi volume,
                                  org.apache.hadoop.net.SocketOutputStream sockOut,
                                  java.nio.channels.FileChannel fileCh,
                                  long position,
                                  int count,
                                  org.apache.hadoop.io.LongWritable waitTime,
                                  org.apache.hadoop.io.LongWritable transferTime)
                           throws java.io.IOException

Transfer data from a FileChannel to a SocketOutputStream.

Parameters:

volume - target volume. null if unavailable.

sockOut - SocketOutputStream to write the data.

fileCh - FileChannel from which to read data.

position - position within the channel where the transfer begins.

count - number of bytes to transfer.

waitTime - returns the nanoseconds spent waiting for the socket to become writable.

transferTime - returns the nanoseconds spent transferring data.

Throws:

java.io.IOException

createFile

public boolean createFile(@Nullable
                          FsVolumeSpi volume,
                          java.io.File f)
                   throws java.io.IOException

Create a file.

Parameters:

volume - target volume. null if unavailable.

f - File to be created.

Returns:

true if the file does not exist and was successfully created. false if the file already exists.

Throws:

java.io.IOException

getFileInputStream

public java.io.FileInputStream getFileInputStream(@Nullable
                                                  FsVolumeSpi volume,
                                                  java.io.File f)
                                           throws java.io.FileNotFoundException

Create a FileInputStream using FileInputStream(File). Wraps the created input stream to intercept read calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

f - File object.

Returns:

FileInputStream to the given file.

Throws:

java.io.FileNotFoundException

getFileOutputStream

public java.io.FileOutputStream getFileOutputStream(@Nullable
                                                    FsVolumeSpi volume,
                                                    java.io.File f,
                                                    boolean append)
                                             throws java.io.FileNotFoundException

Create a FileOutputStream using FileOutputStream(File, boolean). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

f - File object.

append - if true, then bytes will be written to the end of the file rather than the beginning.

Returns:

FileOutputStream to the given file object.

Throws:

java.io.FileNotFoundException

getFileOutputStream

public java.io.FileOutputStream getFileOutputStream(@Nullable
                                                    FsVolumeSpi volume,
                                                    java.io.File f)
                                             throws java.io.FileNotFoundException

Create a FileOutputStream using FileOutputStream(File, boolean). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

f - File object.

Returns:

FileOutputStream to the given file object.

Throws:

java.io.FileNotFoundException

getFileOutputStream

public java.io.FileOutputStream getFileOutputStream(@Nullable
                                                    FsVolumeSpi volume,
                                                    java.io.FileDescriptor fd)

Create a FileOutputStream using FileOutputStream(FileDescriptor). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

fd - File descriptor object.

Returns:

FileOutputStream to the given file object.

getShareDeleteFileInputStream

public java.io.FileInputStream getShareDeleteFileInputStream(@Nullable
                                                             FsVolumeSpi volume,
                                                             java.io.File f,
                                                             long offset)
                                                      throws java.io.IOException

Create a FileInputStream using NativeIO.getShareDeleteFileDescriptor(java.io.File, long). Wraps the created input stream to intercept input calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

f - File object.

offset - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.

Returns:

FileOutputStream to the given file object.

Throws:

java.io.FileNotFoundException

java.io.IOException

openAndSeek

public java.io.FileInputStream openAndSeek(@Nullable
                                           FsVolumeSpi volume,
                                           java.io.File f,
                                           long offset)
                                    throws java.io.IOException

Create a FileInputStream using FileInputStream(File) and position it at the given offset. Wraps the created input stream to intercept read calls before delegating to the wrapped stream.

Parameters:

volume - target volume. null if unavailable.

f - File object.

offset - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.

Throws:

java.io.FileNotFoundException

java.io.IOException

getRandomAccessFile

public java.io.RandomAccessFile getRandomAccessFile(@Nullable
                                                    FsVolumeSpi volume,
                                                    java.io.File f,
                                                    java.lang.String mode)
                                             throws java.io.FileNotFoundException

Create a RandomAccessFile using RandomAccessFile(File, String). Wraps the created input stream to intercept IO calls before delegating to the wrapped RandomAccessFile.

Parameters:

volume - target volume. null if unavailable.

f - File object.

mode - See RandomAccessFile for a description of the mode string.

Returns:

RandomAccessFile representing the given file.

Throws:

java.io.FileNotFoundException

fullyDelete

public boolean fullyDelete(@Nullable
                           FsVolumeSpi volume,
                           java.io.File dir)

Delete the given directory using FileUtil.fullyDelete(File).

Parameters:

volume - target volume. null if unavailable.

dir - directory to be deleted.

Returns:

true on success false on failure.

replaceFile

public void replaceFile(@Nullable
                        FsVolumeSpi volume,
                        java.io.File src,
                        java.io.File target)
                 throws java.io.IOException

Move the src file to the target using FileUtil.replaceFile(File, File).

Parameters:

volume - target volume. null if unavailable.

src - source path.

target - target path.

Throws:

java.io.IOException

rename

public void rename(@Nullable
                   FsVolumeSpi volume,
                   java.io.File src,
                   java.io.File target)
            throws java.io.IOException

Move the src file to the target using Storage.rename(File, File).

Parameters:

volume - target volume. null if unavailable.

src - source path.

target - target path.

Throws:

java.io.IOException

moveFile

public void moveFile(@Nullable
                     FsVolumeSpi volume,
                     java.io.File src,
                     java.io.File target)
              throws java.io.IOException

Move the src file to the target using FileUtils.moveFile(File, File).

Parameters:

volume - target volume. null if unavailable.

src - source path.

target - target path.

Throws:

java.io.IOException

move

public void move(@Nullable
                 FsVolumeSpi volume,
                 java.nio.file.Path src,
                 java.nio.file.Path target,
                 java.nio.file.CopyOption... options)
          throws java.io.IOException

Move the src file to the target using Files.move(Path, Path, CopyOption...).

Parameters:

volume - target volume. null if unavailable.

src - source path.

target - target path.

options - See Files.move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...) for a description of the options.

Throws:

java.io.IOException

nativeCopyFileUnbuffered

public void nativeCopyFileUnbuffered(@Nullable
                                     FsVolumeSpi volume,
                                     java.io.File src,
                                     java.io.File target,
                                     boolean preserveFileDate)
                              throws java.io.IOException

See Storage.nativeCopyFileUnbuffered(File, File, boolean).

Parameters:

volume - target volume. null if unavailable.

src - an existing file to copy, must not be null

target - the new file, must not be null

preserveFileDate - true if the file date of the copy should be the same as the original

Throws:

java.io.IOException

mkdirs

public boolean mkdirs(@Nullable
                      FsVolumeSpi volume,
                      java.io.File dir)
               throws java.io.IOException

See File.mkdirs().

Parameters:

volume - target volume. null if unavailable.

dir - directory to be created.

Returns:

true only if the directory was created. false if the directory already exists.

Throws:

java.io.IOException - if a directory with the given name does not exist and could not be created.

mkdirsWithExistsCheck

public void mkdirsWithExistsCheck(@Nullable
                                  FsVolumeSpi volume,
                                  java.io.File dir)
                           throws java.io.IOException

Create the target directory using File.mkdirs() only if it doesn't exist already.

Parameters:

volume - target volume. null if unavailable.

dir - directory to be created.

Throws:

java.io.IOException - if the directory could not created

listFiles

public java.io.File[] listFiles(@Nullable
                                FsVolumeSpi volume,
                                java.io.File dir)
                         throws java.io.IOException

Get a listing of the given directory using FileUtil.listFiles(File).

Parameters:

volume - target volume. null if unavailable.

dir - Directory to be listed.

Returns:

array of file objects representing the directory entries.

Throws:

java.io.IOException

list

public java.lang.String[] list(@Nullable
                               FsVolumeSpi volume,
                               java.io.File dir)
                        throws java.io.IOException

Get a listing of the given directory using FileUtil.listFiles(File).

Parameters:

volume - target volume. null if unavailable.

dir - directory to be listed.

Returns:

array of strings representing the directory entries.

Throws:

java.io.IOException

listDirectory

public java.util.List<java.lang.String> listDirectory(@Nullable
                                                      FsVolumeSpi volume,
                                                      java.io.File dir,
                                                      java.io.FilenameFilter filter)
                                               throws java.io.IOException

Get a listing of the given directory using IOUtils.listDirectory(File, FilenameFilter).

Parameters:

volume - target volume. null if unavailable.

dir - Directory to list.

filter - FilenameFilter to filter the directory entries.

Throws:

java.io.IOException

getHardLinkCount

public int getHardLinkCount(@Nullable
                            FsVolumeSpi volume,
                            java.io.File f)
                     throws java.io.IOException

Retrieves the number of links to the specified file.

Parameters:

volume - target volume. null if unavailable.

f - file whose link count is being queried.

Returns:

number of hard-links to the given file, including the given path itself.

Throws:

java.io.IOException

exists

public boolean exists(@Nullable
                      FsVolumeSpi volume,
                      java.io.File f)

Check for file existence using File.exists().

Parameters:

volume - target volume. null if unavailable.

f - file object.

Returns:

true if the file exists.

getProfilingEventHook

public org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents getProfilingEventHook()