Package org.apache.hadoop.io.erasurecode.rawcoder

Class RawErasureDecoder

java.lang.Object

org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder

Direct Known Subclasses:

DummyRawDecoder, NativeRSRawDecoder, NativeXORRawDecoder, RSLegacyRawDecoder, RSRawDecoder, XORRawDecoder

@Private
public abstract class RawErasureDecoder
extends java.lang.Object

An abstract raw erasure decoder that's to be inherited by new decoders. Raw erasure coder is part of erasure codec framework, where erasure coder is used to encode/decode a group of blocks (BlockGroup) according to the codec specific BlockGroup layout and logic. An erasure coder extracts chunks of data from the blocks and can employ various low level raw erasure coders to perform encoding/decoding against the chunks. To distinguish from erasure coder, here raw erasure coder is used to mean the low level constructs, since it only takes care of the math calculation with a group of byte buffers. Note it mainly provides decode() calls, which should be stateless and may be made thread-safe in future.

Constructor Summary

Constructors

Constructor	Description
RawErasureDecoder(ErasureCoderOptions coderOptions)

Method Summary

Modifier and Type	Method	Description
boolean	allowChangeInputs()	Allow change into input buffers or not while perform encoding/decoding.
boolean	allowVerboseDump()	Allow to dump verbose info during encoding/decoding.
void	decode(byte[][] inputs, int[] erasedIndexes, byte[][] outputs)	Decode with inputs and erasedIndexes, generates outputs.
void	decode(java.nio.ByteBuffer[] inputs, int[] erasedIndexes, java.nio.ByteBuffer[] outputs)	Decode with inputs and erasedIndexes, generates outputs.
void	decode(ECChunk[] inputs, int[] erasedIndexes, ECChunk[] outputs)	Decode with inputs and erasedIndexes, generates outputs.
protected abstract void	doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteArrayDecodingState decodingState)	Perform the real decoding using bytes array, supporting offsets and lengths.
protected abstract void	doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState decodingState)	Perform the real decoding using Direct ByteBuffer.
protected int	getNumAllUnits()
int	getNumDataUnits()
int	getNumParityUnits()
boolean	preferDirectBuffer()	Tell if direct buffer is preferred or not.
void	release()	Should be called when release this coder.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RawErasureDecoder

public RawErasureDecoder(ErasureCoderOptions coderOptions)

Method Detail

decode

public void decode(java.nio.ByteBuffer[] inputs,
                   int[] erasedIndexes,
                   java.nio.ByteBuffer[] outputs)
            throws java.io.IOException

Decode with inputs and erasedIndexes, generates outputs. How to prepare for inputs: 1. Create an array containing data units + parity units. Please note the data units should be first or before the parity units. 2. Set null in the array locations specified via erasedIndexes to indicate they're erased and no data are to read from; 3. Set null in the array locations for extra redundant items, as they're not necessary to read when decoding. For example in RS-6-3, if only 1 unit is really erased, then we have 2 extra items as redundant. They can be set as null to indicate no data will be used from them. For an example using RS (6, 3), assuming sources (d0, d1, d2, d3, d4, d5) and parities (p0, p1, p2), d2 being erased. We can and may want to use only 6 units like (d1, d3, d4, d5, p0, p2) to recover d2. We will have: inputs = [null(d0), d1, null(d2), d3, d4, d5, p0, null(p1), p2] erasedIndexes = [2] // index of d2 into inputs array outputs = [a-writable-buffer] Note, for both inputs and outputs, no mixing of on-heap buffers and direct buffers are allowed. If the coder option ALLOW_CHANGE_INPUTS is set true (false by default), the content of input buffers may change after the call, subject to concrete implementation.

Parameters:

inputs - input buffers to read data from. The buffers' remaining will be 0 after decoding

erasedIndexes - indexes of erased units in the inputs array

outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call

Throws:

java.io.IOException - raised on errors performing I/O.

doDecode

protected abstract void doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState decodingState)
                          throws java.io.IOException

Perform the real decoding using Direct ByteBuffer.

Parameters:

decodingState - the decoding state

Throws:

java.io.IOException - raised on errors performing I/O.

decode

public void decode(byte[][] inputs,
                   int[] erasedIndexes,
                   byte[][] outputs)
            throws java.io.IOException

Decode with inputs and erasedIndexes, generates outputs. More see above.

Parameters:

inputs - input buffers to read data from

erasedIndexes - indexes of erased units in the inputs array

outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call

Throws:

java.io.IOException - if the decoder is closed.

doDecode

protected abstract void doDecode(org.apache.hadoop.io.erasurecode.rawcoder.ByteArrayDecodingState decodingState)
                          throws java.io.IOException

Perform the real decoding using bytes array, supporting offsets and lengths.

Parameters:

decodingState - the decoding state

Throws:

java.io.IOException - if the decoder is closed.

decode

public void decode(ECChunk[] inputs,
                   int[] erasedIndexes,
                   ECChunk[] outputs)
            throws java.io.IOException

Decode with inputs and erasedIndexes, generates outputs. More see above. Note, for both input and output ECChunks, no mixing of on-heap buffers and direct buffers are allowed.

Parameters:

inputs - input buffers to read data from

erasedIndexes - indexes of erased units in the inputs array

outputs - output buffers to put decoded data into according to erasedIndexes, ready for read after the call

Throws:

java.io.IOException - if the decoder is closed

getNumDataUnits

public int getNumDataUnits()

getNumParityUnits

public int getNumParityUnits()

getNumAllUnits

protected int getNumAllUnits()

preferDirectBuffer

public boolean preferDirectBuffer()

Tell if direct buffer is preferred or not. It's for callers to decide how to allocate coding chunk buffers, using DirectByteBuffer or bytes array. It will return false by default.

Returns:

true if native buffer is preferred for performance consideration, otherwise false.

allowChangeInputs

public boolean allowChangeInputs()

Allow change into input buffers or not while perform encoding/decoding.

Returns:

true if it's allowed to change inputs, false otherwise

allowVerboseDump

public boolean allowVerboseDump()

Allow to dump verbose info during encoding/decoding.

Returns:

true if it's allowed to do verbose dump, false otherwise.

release

public void release()

Should be called when release this coder. Good chance to release encoding or decoding buffers