Package org.apache.hadoop.hdfs.util

Class StripedBlockUtil

java.lang.Object

org.apache.hadoop.hdfs.util.StripedBlockUtil

@Private
public class StripedBlockUtil
extends java.lang.Object

When accessing a file in striped layout, operations on logical byte ranges in the file need to be mapped to physical byte ranges on block files stored on DataNodes. This utility class facilities this mapping by defining and exposing a number of striping-related concepts. The most basic ones are illustrated in the following diagram. Unless otherwise specified, all range-related calculations are inclusive (the end offset of the previous range should be 1 byte lower than the start offset of the next one).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	StripedBlockUtil.AlignedStripe	Given a requested byte range on a striped block group, an AlignedStripe represents an inclusive StripedBlockUtil.VerticalRange that is aligned with both the byte range and boundaries of all internal blocks.
static class	StripedBlockUtil.BlockReadStats	Struct holding the read statistics.
static class	StripedBlockUtil.ChunkByteBuffer	A utility to manage ByteBuffer slices for a reader.
static class	StripedBlockUtil.StripeRange	Used to indicate the buffered data's range in the block group.
static class	StripedBlockUtil.StripingCell	Cell is the unit of encoding used in DFSStripedOutputStream.
static class	StripedBlockUtil.StripingChunk	Indicates the coverage of an StripedBlockUtil.AlignedStripe on an internal block, and the state of the chunk in the context of the read request.
static class	StripedBlockUtil.StripingChunkReadResult	This class represents result from a striped read request.
static class	StripedBlockUtil.VerticalRange	A simple utility class representing an arbitrary vertical inclusive range starting at StripedBlockUtil.VerticalRange.offsetInBlock and lasting for StripedBlockUtil.VerticalRange.spanInBlock bytes in an internal block.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
StripedBlockUtil()

Method Summary

Modifier and Type	Method	Description
static void	checkBlocks(ExtendedBlock blockGroup, int i, ExtendedBlock blocki)	Check if the information such as IDs and generation stamps in block-i match the block group.
static ExtendedBlock	constructInternalBlock(ExtendedBlock blockGroup, int cellSize, int dataBlkNum, int idxInBlockGroup)	This method creates an internal ExtendedBlock at the given index of a block group.
static ExtendedBlock	constructInternalBlock(ExtendedBlock blockGroup, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)
static LocatedBlock	constructInternalBlock(LocatedStripedBlock bg, int idxInReturnedLocs, int cellSize, int dataBlkNum, int idxInBlockGroup)	This method creates an internal block at the given index of a block group.
static StripedBlockUtil.AlignedStripe[]	divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, java.nio.ByteBuffer buf)	This method divides a requested byte range into an array of inclusive StripedBlockUtil.AlignedStripe.
static StripedBlockUtil.AlignedStripe[]	divideOneStripe(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, java.nio.ByteBuffer buf)	Similar functionality with divideByteRangeIntoStripes(org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy, int, org.apache.hadoop.hdfs.protocol.LocatedStripedBlock, long, long, java.nio.ByteBuffer), but is used by stateful read and uses ByteBuffer as reading target buffer.
static int	getBlockIndex(Block reportedBlock)
static long	getInternalBlockLength(long dataSize, int cellSize, int numDataBlocks, int idxInBlockGroup)	Get the size of an internal block at the given index of a block group.
static long	getInternalBlockLength(long dataSize, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)
static StripedBlockUtil.StripingChunkReadResult	getNextCompletedStripedRead(java.util.concurrent.CompletionService<StripedBlockUtil.BlockReadStats> readService, java.util.Map<java.util.concurrent.Future<StripedBlockUtil.BlockReadStats>,java.lang.Integer> futures, long timeoutMillis)	Get the next completed striped read task.
static long	getSafeLength(ErasureCodingPolicy ecPolicy, long[] blockLens)	Compute the safe length given the internal block lengths.
static long	offsetInBlkToOffsetInBG(int cellSize, int dataBlkNum, long offsetInBlk, int idxInBlockGroup)	Given a byte's offset in an internal block, calculate the offset in the block group.
static LocatedBlock[]	parseStripedBlockGroup(LocatedStripedBlock bg, int cellSize, int dataBlkNum, int parityBlkNum)	This method parses a striped block group into individual blocks.
static long	spaceConsumedByStripedBlock(long numDataBlkBytes, int dataBlkNum, int parityBlkNum, int cellSize)	Get the total usage of the striped blocks, which is the total of data blocks and parity blocks.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

StripedBlockUtil

public StripedBlockUtil()

Method Detail

parseStripedBlockGroup

public static LocatedBlock[] parseStripedBlockGroup(LocatedStripedBlock bg,
                                                    int cellSize,
                                                    int dataBlkNum,
                                                    int parityBlkNum)

This method parses a striped block group into individual blocks.

Parameters:

bg - The striped block group

cellSize - The size of a striping cell

dataBlkNum - The number of data blocks

Returns:

An array containing the blocks in the group

constructInternalBlock

public static LocatedBlock constructInternalBlock(LocatedStripedBlock bg,
                                                  int idxInReturnedLocs,
                                                  int cellSize,
                                                  int dataBlkNum,
                                                  int idxInBlockGroup)

This method creates an internal block at the given index of a block group.

Parameters:

idxInReturnedLocs - The index in the stored locations in the LocatedStripedBlock object

idxInBlockGroup - The logical index in the striped block group

Returns:

The constructed internal block

constructInternalBlock

public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup,
                                                   ErasureCodingPolicy ecPolicy,
                                                   int idxInBlockGroup)

constructInternalBlock

public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup,
                                                   int cellSize,
                                                   int dataBlkNum,
                                                   int idxInBlockGroup)

This method creates an internal ExtendedBlock at the given index of a block group.

getInternalBlockLength

public static long getInternalBlockLength(long dataSize,
                                          ErasureCodingPolicy ecPolicy,
                                          int idxInBlockGroup)

getInternalBlockLength

public static long getInternalBlockLength(long dataSize,
                                          int cellSize,
                                          int numDataBlocks,
                                          int idxInBlockGroup)

Get the size of an internal block at the given index of a block group.

Parameters:

dataSize - Size of the block group only counting data blocks

cellSize - The size of a striping cell

numDataBlocks - The number of data blocks

idxInBlockGroup - The logical index in the striped block group

Returns:

The size of the internal block at the specified index

getSafeLength

public static long getSafeLength(ErasureCodingPolicy ecPolicy,
                                 long[] blockLens)

Compute the safe length given the internal block lengths.

Parameters:

ecPolicy - The EC policy used for the block group

blockLens - The lengths of internal blocks

Returns:

The safe length

offsetInBlkToOffsetInBG

public static long offsetInBlkToOffsetInBG(int cellSize,
                                           int dataBlkNum,
                                           long offsetInBlk,
                                           int idxInBlockGroup)

Given a byte's offset in an internal block, calculate the offset in the block group.

getNextCompletedStripedRead

public static StripedBlockUtil.StripingChunkReadResult getNextCompletedStripedRead(java.util.concurrent.CompletionService<StripedBlockUtil.BlockReadStats> readService,
                                                                                   java.util.Map<java.util.concurrent.Future<StripedBlockUtil.BlockReadStats>,java.lang.Integer> futures,
                                                                                   long timeoutMillis)
                                                                            throws java.lang.InterruptedException

Get the next completed striped read task.

Returns:

StripedBlockUtil.StripingChunkReadResult indicating the status of the read task succeeded, and the block index of the task. If the method times out without getting any completed read tasks, -1 is returned as block index.

Throws:

java.lang.InterruptedException

spaceConsumedByStripedBlock

public static long spaceConsumedByStripedBlock(long numDataBlkBytes,
                                               int dataBlkNum,
                                               int parityBlkNum,
                                               int cellSize)

Get the total usage of the striped blocks, which is the total of data blocks and parity blocks.

Parameters:

numDataBlkBytes - Size of the block group only counting data blocks

dataBlkNum - The number of data blocks

parityBlkNum - The number of parity blocks

cellSize - The size of a striping cell

Returns:

The total usage of data blocks and parity blocks

divideOneStripe

public static StripedBlockUtil.AlignedStripe[] divideOneStripe(ErasureCodingPolicy ecPolicy,
                                                               int cellSize,
                                                               LocatedStripedBlock blockGroup,
                                                               long rangeStartInBlockGroup,
                                                               long rangeEndInBlockGroup,
                                                               java.nio.ByteBuffer buf)

Similar functionality with divideByteRangeIntoStripes(org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy, int, org.apache.hadoop.hdfs.protocol.LocatedStripedBlock, long, long, java.nio.ByteBuffer), but is used by stateful read and uses ByteBuffer as reading target buffer. Besides the read range is within a single stripe thus the calculation logic is simpler.

divideByteRangeIntoStripes

public static StripedBlockUtil.AlignedStripe[] divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy,
                                                                          int cellSize,
                                                                          LocatedStripedBlock blockGroup,
                                                                          long rangeStartInBlockGroup,
                                                                          long rangeEndInBlockGroup,
                                                                          java.nio.ByteBuffer buf)

This method divides a requested byte range into an array of inclusive StripedBlockUtil.AlignedStripe.

Parameters:

ecPolicy - The codec policy for the file, which carries the numbers of data / parity blocks

cellSize - Cell size of stripe

blockGroup - The striped block group

rangeStartInBlockGroup - The byte range's start offset in block group

rangeEndInBlockGroup - The byte range's end offset in block group

buf - Destination buffer of the read operation for the byte range At most 5 stripes will be generated from each logical range, as demonstrated in the header of StripedBlockUtil.AlignedStripe.

checkBlocks

public static void checkBlocks(ExtendedBlock blockGroup,
                               int i,
                               ExtendedBlock blocki)
                        throws java.io.IOException

Check if the information such as IDs and generation stamps in block-i match the block group.

Throws:

java.io.IOException

getBlockIndex

public static int getBlockIndex(Block reportedBlock)