Package org.apache.hadoop.hdfs.protocol.datatransfer

Interface DataTransferProtocol

All Known Implementing Classes:

Sender

@Private
@Evolving
public interface DataTransferProtocol

Transfer data to/from datanode using a streaming protocol.

Field Summary

Fields

Modifier and Type	Field	Description
static int	DATA_TRANSFER_VERSION	Version for data transfers between clients and datanodes This should change when serialization of DatanodeInfo, not just when protocol changes.
static org.slf4j.Logger	LOG

Method Summary

Modifier and Type	Method	Description
void	blockChecksum(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, BlockChecksumOptions blockChecksumOptions)	Get block checksum (MD5 of CRC32).
void	blockGroupChecksum(StripedBlockInfo stripedBlockInfo, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long requestedNumBytes, BlockChecksumOptions blockChecksumOptions)	Get striped block group checksum (MD5 of CRC32).
void	copyBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken)	Copy a block.
void	readBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, java.lang.String clientName, long blockOffset, long length, boolean sendChecksum, CachingStrategy cachingStrategy)	Read a block.
void	releaseShortCircuitFds(ShortCircuitShm.SlotId slotId)	Release a pair of short-circuit FDs requested earlier.
void	replaceBlock(ExtendedBlock blk, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, java.lang.String delHint, DatanodeInfo source, java.lang.String storageId)	Receive a block from a source datanode and then notifies the namenode to remove the copy from the original datanode.
void	requestShortCircuitFds(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, ShortCircuitShm.SlotId slotId, int maxVersion, boolean supportsReceiptVerification)	Request short circuit access file descriptors from a DataNode.
void	requestShortCircuitShm(java.lang.String clientName)	Request a short circuit shared memory area from a DataNode.
void	transferBlock(ExtendedBlock blk, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, java.lang.String clientName, DatanodeInfo[] targets, org.apache.hadoop.fs.StorageType[] targetStorageTypes, java.lang.String[] targetStorageIDs)	Transfer a block to another datanode.
void	writeBlock(ExtendedBlock blk, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, java.lang.String clientName, DatanodeInfo[] targets, org.apache.hadoop.fs.StorageType[] targetStorageTypes, DatanodeInfo source, BlockConstructionStage stage, int pipelineSize, long minBytesRcvd, long maxBytesRcvd, long latestGenerationStamp, org.apache.hadoop.util.DataChecksum requestedChecksum, CachingStrategy cachingStrategy, boolean allowLazyPersist, boolean pinning, boolean[] targetPinnings, java.lang.String storageID, java.lang.String[] targetStorageIDs)	Write a block to a datanode pipeline.

Field Detail

LOG

static final org.slf4j.Logger LOG

DATA_TRANSFER_VERSION

static final int DATA_TRANSFER_VERSION

Version for data transfers between clients and datanodes This should change when serialization of DatanodeInfo, not just when protocol changes. It is not very obvious.

See Also:

Constant Field Values

Method Detail

readBlock

void readBlock(ExtendedBlock blk,
               org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
               java.lang.String clientName,
               long blockOffset,
               long length,
               boolean sendChecksum,
               CachingStrategy cachingStrategy)
        throws java.io.IOException

Read a block.

Parameters:

blk - the block being read.

blockToken - security token for accessing the block.

clientName - client's name.

blockOffset - offset of the block.

length - maximum number of bytes for this read.

sendChecksum - if false, the DN should skip reading and sending checksums

cachingStrategy - The caching strategy to use.

Throws:

java.io.IOException

writeBlock

void writeBlock(ExtendedBlock blk,
                org.apache.hadoop.fs.StorageType storageType,
                org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                java.lang.String clientName,
                DatanodeInfo[] targets,
                org.apache.hadoop.fs.StorageType[] targetStorageTypes,
                DatanodeInfo source,
                BlockConstructionStage stage,
                int pipelineSize,
                long minBytesRcvd,
                long maxBytesRcvd,
                long latestGenerationStamp,
                org.apache.hadoop.util.DataChecksum requestedChecksum,
                CachingStrategy cachingStrategy,
                boolean allowLazyPersist,
                boolean pinning,
                boolean[] targetPinnings,
                java.lang.String storageID,
                java.lang.String[] targetStorageIDs)
         throws java.io.IOException

Write a block to a datanode pipeline. The receiver datanode of this call is the next datanode in the pipeline. The other downstream datanodes are specified by the targets parameter. Note that the receiver DatanodeInfo is not required in the parameter list since the receiver datanode knows its info. However, the StorageType for storing the replica in the receiver datanode is a parameter since the receiver datanode may support multiple storage types.

Parameters:

blk - the block being written.

storageType - for storing the replica in the receiver datanode.

blockToken - security token for accessing the block.

clientName - client's name.

targets - other downstream datanodes in the pipeline.

targetStorageTypes - target StorageTypes corresponding to the target datanodes.

source - source datanode.

stage - pipeline stage.

pipelineSize - the size of the pipeline.

minBytesRcvd - minimum number of bytes received.

maxBytesRcvd - maximum number of bytes received.

latestGenerationStamp - the latest generation stamp of the block.

requestedChecksum - the requested checksum mechanism

cachingStrategy - the caching strategy

allowLazyPersist - hint to the DataNode that the block can be allocated on transient storage i.e. memory and written to disk lazily

pinning - whether to pin the block, so Balancer won't move it.

targetPinnings - whether to pin the block on target datanode

storageID - optional StorageIDs designating where to write the block. An empty String or null indicates that this has not been provided.

targetStorageIDs - target StorageIDs corresponding to the target datanodes.

Throws:

java.io.IOException

transferBlock

void transferBlock(ExtendedBlock blk,
                   org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                   java.lang.String clientName,
                   DatanodeInfo[] targets,
                   org.apache.hadoop.fs.StorageType[] targetStorageTypes,
                   java.lang.String[] targetStorageIDs)
            throws java.io.IOException

Transfer a block to another datanode. The block stage must be either BlockConstructionStage.TRANSFER_RBW or BlockConstructionStage.TRANSFER_FINALIZED.

Parameters:

blk - the block being transferred.

blockToken - security token for accessing the block.

clientName - client's name.

targets - target datanodes.

targetStorageIDs - StorageID designating where to write the block.

Throws:

java.io.IOException

requestShortCircuitFds

void requestShortCircuitFds(ExtendedBlock blk,
                            org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                            ShortCircuitShm.SlotId slotId,
                            int maxVersion,
                            boolean supportsReceiptVerification)
                     throws java.io.IOException

Request short circuit access file descriptors from a DataNode.

Parameters:

blk - The block to get file descriptors for.

blockToken - Security token for accessing the block.

slotId - The shared memory slot id to use, or null to use no slot id.

maxVersion - Maximum version of the block data the client can understand.

supportsReceiptVerification - True if the client supports receipt verification.

Throws:

java.io.IOException

releaseShortCircuitFds

void releaseShortCircuitFds(ShortCircuitShm.SlotId slotId)
                     throws java.io.IOException

Release a pair of short-circuit FDs requested earlier.

Parameters:

slotId - SlotID used by the earlier file descriptors.

Throws:

java.io.IOException

requestShortCircuitShm

void requestShortCircuitShm(java.lang.String clientName)
                     throws java.io.IOException

Request a short circuit shared memory area from a DataNode.

Parameters:

clientName - The name of the client.

Throws:

java.io.IOException

replaceBlock

void replaceBlock(ExtendedBlock blk,
                  org.apache.hadoop.fs.StorageType storageType,
                  org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                  java.lang.String delHint,
                  DatanodeInfo source,
                  java.lang.String storageId)
           throws java.io.IOException

Receive a block from a source datanode and then notifies the namenode to remove the copy from the original datanode. Note that the source datanode and the original datanode can be different. It is used for balancing purpose.

Parameters:

blk - the block being replaced.

storageType - the StorageType for storing the block.

blockToken - security token for accessing the block.

delHint - the hint for deleting the block in the original datanode.

source - the source datanode for receiving the block.

storageId - an optional storage ID to designate where the block is replaced to.

Throws:

java.io.IOException

copyBlock

void copyBlock(ExtendedBlock blk,
               org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken)
        throws java.io.IOException

Copy a block. It is used for balancing purpose.

Parameters:

blk - the block being copied.

blockToken - security token for accessing the block.

Throws:

java.io.IOException

blockChecksum

void blockChecksum(ExtendedBlock blk,
                   org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                   BlockChecksumOptions blockChecksumOptions)
            throws java.io.IOException

Get block checksum (MD5 of CRC32).

Parameters:

blk - a block.

blockToken - security token for accessing the block.

blockChecksumOptions - determines how the block-level checksum is computed from underlying block metadata.

Throws:

java.io.IOException

blockGroupChecksum

void blockGroupChecksum(StripedBlockInfo stripedBlockInfo,
                        org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                        long requestedNumBytes,
                        BlockChecksumOptions blockChecksumOptions)
                 throws java.io.IOException

Get striped block group checksum (MD5 of CRC32).

Parameters:

stripedBlockInfo - a striped block info.

blockToken - security token for accessing the block.

requestedNumBytes - requested number of bytes in the block group to compute the checksum.

blockChecksumOptions - determines how the block-level checksum is computed from underlying block metadata.

Throws:

java.io.IOException