Package org.apache.hadoop.hdfs.client.impl

Class BlockReaderRemote

java.lang.Object

org.apache.hadoop.hdfs.client.impl.BlockReaderRemote

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.fs.ByteBufferReadable, BlockReader

@Private
public class BlockReaderRemote
extends java.lang.Object
implements BlockReader

This is a wrapper around connection to datanode and understands checksum, offset etc. Terminology:

block

The hdfs block, typically large (~64MB).

chunk

A block is divided into chunks, each comes with a checksum. We want transfers to be chunk-aligned, to be able to verify checksums.

packet

A grouping of chunks used for transport. It contains a header, followed by checksum data, followed by real data.

Please see DataNode for the RPC specification. This is a new implementation introduced in Hadoop 0.23 which is more efficient and simpler than the older BlockReader implementation. It is renamed to BlockReaderRemote from BlockReaderRemote2.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BlockReaderRemote(java.lang.String file, long blockId, org.apache.hadoop.util.DataChecksum checksum, boolean verifyChecksum, long startOffset, long firstChunkOffset, long bytesToRead, Peer peer, DatanodeID datanodeID, PeerCache peerCache, int networkDistance)

Method Summary

Modifier and Type	Method	Description
int	available()	Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without performing network I/O.
void	close()	Close the block reader.
ClientMmap	getClientMmap(java.util.EnumSet<org.apache.hadoop.fs.ReadOption> opts)	Get a ClientMmap object for this BlockReader.
org.apache.hadoop.util.DataChecksum	getDataChecksum()
static java.lang.String	getFileName(java.net.InetSocketAddress s, java.lang.String poolId, long blockId)	File name to print when accessing a block directly (from servlets)
int	getNetworkDistance()	Return the network distance between local machine and the remote machine.
Peer	getPeer()
boolean	isShortCircuit()
static BlockReader	newBlockReader(java.lang.String file, ExtendedBlock block, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, long startOffset, long len, boolean verifyChecksum, java.lang.String clientName, Peer peer, DatanodeID datanodeID, PeerCache peerCache, CachingStrategy cachingStrategy, int networkDistance, org.apache.hadoop.conf.Configuration configuration)	Create a new BlockReader specifically to satisfy a read.
int	read(byte[] buf, int off, int len)
int	read(java.nio.ByteBuffer buf)
int	readAll(byte[] buf, int offset, int len)	Similar to BlockReader.readFully(byte[], int, int) except that it will not throw an exception on EOF.
void	readFully(byte[] buf, int off, int len)	Read exactly the given amount of data, throwing an exception if EOF is reached before that amount
long	skip(long n)	Skip the given number of bytes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockReaderRemote

protected BlockReaderRemote(java.lang.String file,
                            long blockId,
                            org.apache.hadoop.util.DataChecksum checksum,
                            boolean verifyChecksum,
                            long startOffset,
                            long firstChunkOffset,
                            long bytesToRead,
                            Peer peer,
                            DatanodeID datanodeID,
                            PeerCache peerCache,
                            int networkDistance)

Method Detail

getPeer

@VisibleForTesting
public Peer getPeer()

read

public int read(byte[] buf,
                int off,
                int len)
         throws java.io.IOException

Specified by:

read in interface BlockReader

Throws:

java.io.IOException

read

public int read(java.nio.ByteBuffer buf)
         throws java.io.IOException

Specified by:

read in interface org.apache.hadoop.fs.ByteBufferReadable

Throws:

java.io.IOException

skip

public long skip(long n)
          throws java.io.IOException

Description copied from interface: BlockReader

Skip the given number of bytes

Specified by:

skip in interface BlockReader

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Description copied from interface: BlockReader

Close the block reader.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface BlockReader

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

getFileName

public static java.lang.String getFileName(java.net.InetSocketAddress s,
                                           java.lang.String poolId,
                                           long blockId)

File name to print when accessing a block directly (from servlets)

Parameters:

s - Address of the block location

poolId - Block pool ID of the block

blockId - Block ID of the block

Returns:

string that has a file name for debug purposes

readAll

public int readAll(byte[] buf,
                   int offset,
                   int len)
            throws java.io.IOException

Description copied from interface: BlockReader

Similar to BlockReader.readFully(byte[], int, int) except that it will not throw an exception on EOF. However, it differs from the simple BlockReader.read(byte[], int, int) call in that it is guaranteed to read the data if it is available. In other words, if this call does not throw an exception, then either the buffer has been filled or the next call will return EOF.

Specified by:

readAll in interface BlockReader

Throws:

java.io.IOException

readFully

public void readFully(byte[] buf,
                      int off,
                      int len)
               throws java.io.IOException

Description copied from interface: BlockReader

Read exactly the given amount of data, throwing an exception if EOF is reached before that amount

Specified by:

readFully in interface BlockReader

Throws:

java.io.IOException

newBlockReader

public static BlockReader newBlockReader(java.lang.String file,
                                         ExtendedBlock block,
                                         org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
                                         long startOffset,
                                         long len,
                                         boolean verifyChecksum,
                                         java.lang.String clientName,
                                         Peer peer,
                                         DatanodeID datanodeID,
                                         PeerCache peerCache,
                                         CachingStrategy cachingStrategy,
                                         int networkDistance,
                                         org.apache.hadoop.conf.Configuration configuration)
                                  throws java.io.IOException

Create a new BlockReader specifically to satisfy a read. This method also sends the OP_READ_BLOCK request.

Parameters:

file - File location.

block - The block object.

blockToken - The block token for security.

startOffset - The read offset, relative to block head.

len - The number of bytes to read.

verifyChecksum - Whether to verify checksum.

clientName - Client name.

peer - The Peer to use.

datanodeID - The DatanodeID this peer is connected to.

peerCache - Caches TCP and UNIX domain sockets for reuse.

cachingStrategy - Caching strategy to use when reading the block.

networkDistance - Return the distance between two nodes by comparing their network paths.

configuration - Configuration of client.

Returns:

New BlockReader instance, or null on error.

Throws:

java.io.IOException

available

public int available()

Description copied from interface: BlockReader

Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without performing network I/O. This may return more than what is actually present in the block.

Specified by:

available in interface BlockReader

isShortCircuit

public boolean isShortCircuit()

Specified by:

isShortCircuit in interface BlockReader

Returns:

true only if this is a short-circuit read. All short-circuit reads are also local.

getClientMmap

public ClientMmap getClientMmap(java.util.EnumSet<org.apache.hadoop.fs.ReadOption> opts)

Description copied from interface: BlockReader

Get a ClientMmap object for this BlockReader.

Specified by:

getClientMmap in interface BlockReader

Parameters:

opts - The read options to use.

Returns:

The ClientMmap object, or null if mmap is not supported.

getDataChecksum

public org.apache.hadoop.util.DataChecksum getDataChecksum()

Specified by:

getDataChecksum in interface BlockReader

Returns:

The DataChecksum used by the read block

getNetworkDistance

public int getNetworkDistance()

Description copied from interface: BlockReader

Return the network distance between local machine and the remote machine.

Specified by:

getNetworkDistance in interface BlockReader