Package org.apache.hadoop.fs.store

Class DataBlocks

java.lang.Object

org.apache.hadoop.fs.store.DataBlocks

public final class DataBlocks
extends java.lang.Object

A class to provide disk, byteBuffer and byteArray option for Filesystem OutputStreams.

• Disk: Uses Disk space to write the blocks. Is suited best to avoid OutOfMemory Errors in Java heap space.
• byteBuffer: Uses DirectByteBuffer to allocate memory off-heap to provide faster writing of DataBlocks with some risk of running OutOfMemory.
• byteArray: Uses byte[] to write a block of data. On heap and does have risk of running OutOfMemory fairly easily.

Implementation of DataBlocks taken from HADOOP-13560 to support huge file uploads in S3A with different options rather than one.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DataBlocks.BlockFactory	Base class for block factories.
static class	DataBlocks.BlockUploadData	The output information for an upload.
static class	DataBlocks.DataBlock	This represents a block being uploaded.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	DATA_BLOCKS_BUFFER_ARRAY	Use an in-memory array.
static java.lang.String	DATA_BLOCKS_BUFFER_DISK	Buffer blocks to disk.
static java.lang.String	DATA_BLOCKS_BYTEBUFFER	Use a byte buffer.

Method Summary

Modifier and Type	Method	Description
static DataBlocks.BlockFactory	createFactory(java.lang.String keyToBufferDir, Configuration configuration, java.lang.String name)	Create a factory.
static void	validateWriteArgs(byte[] b, int off, int len)	Validate args to a write command.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATA_BLOCKS_BUFFER_DISK

public static final java.lang.String DATA_BLOCKS_BUFFER_DISK

Buffer blocks to disk. Capacity is limited to available disk space.

See Also:

Constant Field Values

DATA_BLOCKS_BYTEBUFFER

public static final java.lang.String DATA_BLOCKS_BYTEBUFFER

Use a byte buffer.

See Also:

Constant Field Values

DATA_BLOCKS_BUFFER_ARRAY

public static final java.lang.String DATA_BLOCKS_BUFFER_ARRAY

Use an in-memory array. Fast but will run of heap rapidly.

See Also:

Constant Field Values

Method Detail

validateWriteArgs

public static void validateWriteArgs(byte[] b,
                                     int off,
                                     int len)
                              throws java.io.IOException

Validate args to a write command. These are the same validation checks expected for any implementation of OutputStream.write().

Parameters:

b - byte array containing data.

off - offset in array where to start.

len - number of bytes to be written.

Throws:

java.lang.NullPointerException - for a null buffer

java.lang.IndexOutOfBoundsException - if indices are out of range

java.io.IOException - raised on errors performing I/O.

createFactory

public static DataBlocks.BlockFactory createFactory(java.lang.String keyToBufferDir,
                                                    Configuration configuration,
                                                    java.lang.String name)

Create a factory.

Parameters:

keyToBufferDir - Key to buffer directory config for a FS.

configuration - factory configurations.

name - factory name -the option from CommonConfigurationKeys.

Returns:

the factory, ready to be initialized.

Throws:

java.lang.IllegalArgumentException - if the name is unknown.