Package org.apache.hadoop.util

Class CrcComposer

java.lang.Object

org.apache.hadoop.util.CrcComposer

@LimitedPrivate({"Common","HDFS","MapReduce","Yarn"})
@Unstable
public class CrcComposer
extends java.lang.Object

Encapsulates logic for composing multiple CRCs into one or more combined CRCs corresponding to concatenated underlying data ranges. Optimized for composing a large number of CRCs that correspond to underlying chunks of data all of same size.

Method Summary

Modifier and Type	Method	Description
byte[]	digest()	Returns byte representation of composed CRCs; if no stripeLength was specified, the digest should be of length equal to exactly one CRC.
static CrcComposer	newCrcComposer(DataChecksum.Type type, long bytesPerCrcHint)	Returns a CrcComposer which will collapse all ingested CRCs into a single value.
static CrcComposer	newStripedCrcComposer(DataChecksum.Type type, long bytesPerCrcHint, long stripeLength)	Returns a CrcComposer which will collapse CRCs for every combined underlying data size which aligns with the specified stripe boundary.
void	update(byte[] crcBuffer, int offset, int length, long bytesPerCrc)	Composes length / CRC_SIZE_IN_BYTES more CRCs from crcBuffer, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.
void	update(int crcB, long bytesPerCrc)	Updates with a single additional CRC which corresponds to an underlying data size of bytesPerCrc.
void	update(java.io.DataInputStream checksumIn, long numChecksumsToRead, long bytesPerCrc)	Composes numChecksumsToRead additional CRCs into the current digest out of checksumIn, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

newCrcComposer

public static CrcComposer newCrcComposer(DataChecksum.Type type,
                                         long bytesPerCrcHint)
                                  throws java.io.IOException

Returns a CrcComposer which will collapse all ingested CRCs into a single value.

Parameters:

type - type.

bytesPerCrcHint - bytesPerCrcHint.

Returns:

a CrcComposer which will collapse all ingested CRCs into a single value.

Throws:

java.io.IOException - raised on errors performing I/O.

newStripedCrcComposer

public static CrcComposer newStripedCrcComposer(DataChecksum.Type type,
                                                long bytesPerCrcHint,
                                                long stripeLength)
                                         throws java.io.IOException

Returns a CrcComposer which will collapse CRCs for every combined underlying data size which aligns with the specified stripe boundary. For example, if "update" is called with 20 CRCs and bytesPerCrc == 5, and stripeLength == 10, then every two (10 / 5) consecutive CRCs will be combined with each other, yielding a list of 10 CRC "stripes" in the final digest, each corresponding to 10 underlying data bytes. Using a stripeLength greater than the total underlying data size is equivalent to using a non-striped CrcComposer.

Parameters:

type - type.

bytesPerCrcHint - bytesPerCrcHint.

stripeLength - stripeLength.

Returns:

a CrcComposer which will collapse CRCs for every combined. underlying data size which aligns with the specified stripe boundary.

Throws:

java.io.IOException - raised on errors performing I/O.

update

public void update(byte[] crcBuffer,
                   int offset,
                   int length,
                   long bytesPerCrc)
            throws java.io.IOException

Composes length / CRC_SIZE_IN_BYTES more CRCs from crcBuffer, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.

Parameters:

crcBuffer - crcBuffer.

offset - offset.

length - must be a multiple of the expected byte-size of a CRC.

bytesPerCrc - bytesPerCrc.

Throws:

java.io.IOException - raised on errors performing I/O.

update

public void update(java.io.DataInputStream checksumIn,
                   long numChecksumsToRead,
                   long bytesPerCrc)
            throws java.io.IOException

Composes numChecksumsToRead additional CRCs into the current digest out of checksumIn, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.

Parameters:

checksumIn - checksumIn.

numChecksumsToRead - numChecksumsToRead.

bytesPerCrc - bytesPerCrc.

Throws:

java.io.IOException - raised on errors performing I/O.

update

public void update(int crcB,
                   long bytesPerCrc)
            throws java.io.IOException

Updates with a single additional CRC which corresponds to an underlying data size of bytesPerCrc.

Parameters:

crcB - crcB.

bytesPerCrc - bytesPerCrc.

Throws:

java.io.IOException - raised on errors performing I/O.

digest

public byte[] digest()

Returns byte representation of composed CRCs; if no stripeLength was specified, the digest should be of length equal to exactly one CRC. Otherwise, the number of CRCs in the returned array is equal to the total sum bytesPerCrc divided by stripeLength. If the sum of bytesPerCrc is not a multiple of stripeLength, then the last CRC in the array corresponds to totalLength % stripeLength underlying data bytes.

Returns:

byte representation of composed CRCs.