Package org.apache.hadoop.hdfs

Class DFSInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.hdfs.DFSInputStream

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.fs.ByteBufferPositionedReadable, org.apache.hadoop.fs.ByteBufferReadable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.HasEnhancedByteBufferAccess, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.StreamCapabilities

Direct Known Subclasses:

DFSStripedInputStream

@Private
public class DFSInputStream
extends org.apache.hadoop.fs.FSInputStream
implements org.apache.hadoop.fs.ByteBufferReadable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.HasEnhancedByteBufferAccess, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.ByteBufferPositionedReadable

DFSInputStream provides bytes from a named file. It handles negotiation of the namenode and various datanodes as necessary.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field	Description
protected long	blockEnd
protected CachingStrategy	cachingStrategy
protected java.util.concurrent.atomic.AtomicBoolean	closed
protected LocatedBlock	currentLocatedBlock
protected DFSClient	dfsClient
protected int	failures	This variable tracks the number of failures since the start of the most recent user-facing operation.
protected java.lang.Object	infoLock
protected LocatedBlocks	locatedBlocks
protected long	pos
protected ReadStatistics	readStatistics
protected java.lang.String	src
static boolean	tcpReadsDisabledForTesting
protected boolean	verifyChecksum

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Method Summary

Modifier and Type	Method	Description
protected void	addToLocalDeadNodes(DatanodeInfo dnInfo)
int	available()	Return the size of the remaining available bytes if the size is less than or equal to Integer.MAX_VALUE, otherwise, return Integer.MAX_VALUE.
void	clearReadStatistics()	Clear statistics about the reads which this DFSInputStream has done.
void	close()	Close it down!
protected void	closeCurrentBlockReaders()
protected LocatedBlock	fetchBlockAt(long offset)	Fetch a block from namenode and cache it
protected void	fetchBlockByteRange(LocatedBlock block, long start, long end, java.nio.ByteBuffer buf, DFSUtilClient.CorruptedBlocks corruptedBlocks)
java.util.List<LocatedBlock>	getAllBlocks()	Return collection of blocks that has already been located.
protected org.apache.hadoop.hdfs.DFSInputStream.DNAddrPair	getBestNodeDNAddrPair(LocatedBlock block, java.util.Collection<DatanodeInfo> ignoredNodes)	Get the best node from which to stream the data.
protected LocatedBlock	getBlockAt(long offset)	Get block at the specified position.
protected BlockReader	getBlockReader(LocatedBlock targetBlock, long offsetInBlock, long length, java.net.InetSocketAddress targetAddr, org.apache.hadoop.fs.StorageType storageType, DatanodeInfo datanode)
ExtendedBlock	getCurrentBlock()	Returns the block containing the target position.
protected int	getCurrentBlockLocationsLength()
DatanodeInfo	getCurrentDatanode()	Returns the datanode from which the stream is currently reading.
protected DFSClient	getDFSClient()
org.apache.hadoop.fs.FileEncryptionInfo	getFileEncryptionInfo()
long	getFileLength()
long	getHedgedReadOpsLoopNumForTesting()
protected java.util.concurrent.ConcurrentHashMap<DatanodeInfo,DatanodeInfo>	getLocalDeadNodes()
protected LocatedBlocks	getLocatedBlocks()
long	getPos()
ReadStatistics	getReadStatistics()	Get statistics about the reads which this DFSInputStream has done.
protected java.lang.String	getSrc()
boolean	hasCapability(java.lang.String capability)
void	mark(int readLimit)
boolean	markSupported()	We definitely don't support marks
protected void	maybeRegisterBlockRefresh()	Many DFSInputStreams can be opened and closed in quick succession, in which case they would be registered/deregistered but never need to be refreshed.
int	read()
int	read(byte[] buf, int off, int len)	Read the entire buffer.
int	read(long position, byte[] buffer, int offset, int length)	Read bytes starting from the specified position.
int	read(long position, java.nio.ByteBuffer buf)
int	read(java.nio.ByteBuffer buf)
java.nio.ByteBuffer	read(org.apache.hadoop.io.ByteBufferPool bufferPool, int maxLength, java.util.EnumSet<org.apache.hadoop.fs.ReadOption> opts)
void	readFully(long position, java.nio.ByteBuffer buf)
protected int	readWithStrategy(org.apache.hadoop.hdfs.ReaderStrategy strategy)
protected LocatedBlock	refreshLocatedBlock(LocatedBlock block)	Refresh cached block locations.
void	releaseBuffer(java.nio.ByteBuffer buffer)
protected void	removeFromLocalDeadNodes(DatanodeInfo dnInfo)
protected void	reportCheckSumFailure(DFSUtilClient.CorruptedBlocks corruptedBlocks, int dataNodeCount, boolean isStriped)	DFSInputStream reports checksum failure.
protected void	reportLostBlock(LocatedBlock lostBlock, java.util.Collection<DatanodeInfo> ignoredNodes)	Warn the user of a lost block
void	reset()
void	seek(long targetPos)	Seek to a new arbitrary location
boolean	seekToNewSource(long targetPos)	Seek to given position on a node other than the current node.
void	setDropBehind(java.lang.Boolean dropBehind)
void	setReadahead(java.lang.Long readahead)
long	skip(long n)
protected static boolean	tokenRefetchNeeded(java.io.IOException ex, java.net.InetSocketAddress targetAddr)	Should the block access token be refetched on an exception
void	unbuffer()

Methods inherited from class org.apache.hadoop.fs.FSInputStream

readFully, readFully, toString, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

nullInputStream, read, readAllBytes, readNBytes, readNBytes, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

maxReadSizeForVectorReads, minSeekForVectorReads, readVectored

Field Detail

tcpReadsDisabledForTesting

@VisibleForTesting
public static boolean tcpReadsDisabledForTesting

dfsClient

protected final DFSClient dfsClient

closed

protected java.util.concurrent.atomic.AtomicBoolean closed

src

protected final java.lang.String src

verifyChecksum

protected final boolean verifyChecksum

currentLocatedBlock

protected LocatedBlock currentLocatedBlock

pos

protected long pos

blockEnd

protected long blockEnd

locatedBlocks

protected LocatedBlocks locatedBlocks

cachingStrategy

protected CachingStrategy cachingStrategy

readStatistics

protected final ReadStatistics readStatistics

infoLock

protected final java.lang.Object infoLock

failures

protected int failures

This variable tracks the number of failures since the start of the most recent user-facing operation. That is to say, it should be reset whenever the user makes a call on this stream, and if at any point during the retry logic, the failure count exceeds a threshold, the errors will be thrown back to the operation. Specifically this counts the number of times the client has gone back to the namenode to get a new list of block locations, and is capped at maxBlockAcquireFailures

Method Detail

addToLocalDeadNodes

protected void addToLocalDeadNodes(DatanodeInfo dnInfo)

removeFromLocalDeadNodes

protected void removeFromLocalDeadNodes(DatanodeInfo dnInfo)

getLocalDeadNodes

protected java.util.concurrent.ConcurrentHashMap<DatanodeInfo,DatanodeInfo> getLocalDeadNodes()

getDFSClient

protected DFSClient getDFSClient()

getFileLength

public long getFileLength()

getCurrentDatanode

public DatanodeInfo getCurrentDatanode()

Returns the datanode from which the stream is currently reading.

getCurrentBlock

public ExtendedBlock getCurrentBlock()

Returns the block containing the target position.

getAllBlocks

public java.util.List<LocatedBlock> getAllBlocks()
                                          throws java.io.IOException

Return collection of blocks that has already been located.

Throws:

java.io.IOException

getSrc

protected java.lang.String getSrc()

getLocatedBlocks

protected LocatedBlocks getLocatedBlocks()

getBlockAt

protected LocatedBlock getBlockAt(long offset)
                           throws java.io.IOException

Get block at the specified position. Fetch it from the namenode if not cached.

Parameters:

offset - block corresponding to this offset in file is returned

Returns:

located block

Throws:

java.io.IOException

fetchBlockAt

protected LocatedBlock fetchBlockAt(long offset)
                             throws java.io.IOException

Fetch a block from namenode and cache it

Throws:

java.io.IOException

getBlockReader

protected BlockReader getBlockReader(LocatedBlock targetBlock,
                                     long offsetInBlock,
                                     long length,
                                     java.net.InetSocketAddress targetAddr,
                                     org.apache.hadoop.fs.StorageType storageType,
                                     DatanodeInfo datanode)
                              throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Close it down!

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Overrides:

close in class java.io.InputStream

Throws:

java.io.IOException

read

public int read()
         throws java.io.IOException

Specified by:

read in class java.io.InputStream

Throws:

java.io.IOException

readWithStrategy

protected int readWithStrategy(org.apache.hadoop.hdfs.ReaderStrategy strategy)
                        throws java.io.IOException

Throws:

java.io.IOException

getCurrentBlockLocationsLength

protected int getCurrentBlockLocationsLength()

read

public int read(@Nonnull
                byte[] buf,
                int off,
                int len)
         throws java.io.IOException

Read the entire buffer.

Overrides:

read in class java.io.InputStream

Throws:

java.io.IOException

read

public int read(java.nio.ByteBuffer buf)
         throws java.io.IOException

Specified by:

read in interface org.apache.hadoop.fs.ByteBufferReadable

Throws:

java.io.IOException

getBestNodeDNAddrPair

protected org.apache.hadoop.hdfs.DFSInputStream.DNAddrPair getBestNodeDNAddrPair(LocatedBlock block,
                                                                                 java.util.Collection<DatanodeInfo> ignoredNodes)

Get the best node from which to stream the data.

Parameters:

block - LocatedBlock, containing nodes in priority order.

ignoredNodes - Do not choose nodes in this array (may be null)

Returns:

The DNAddrPair of the best node. Null if no node can be chosen.

reportLostBlock

protected void reportLostBlock(LocatedBlock lostBlock,
                               java.util.Collection<DatanodeInfo> ignoredNodes)

Warn the user of a lost block

fetchBlockByteRange

protected void fetchBlockByteRange(LocatedBlock block,
                                   long start,
                                   long end,
                                   java.nio.ByteBuffer buf,
                                   DFSUtilClient.CorruptedBlocks corruptedBlocks)
                            throws java.io.IOException

Throws:

java.io.IOException

refreshLocatedBlock

protected LocatedBlock refreshLocatedBlock(LocatedBlock block)
                                    throws java.io.IOException

Refresh cached block locations.

Parameters:

block - The currently cached block locations

Returns:

Refreshed block locations

Throws:

java.io.IOException

getHedgedReadOpsLoopNumForTesting

@VisibleForTesting
public long getHedgedReadOpsLoopNumForTesting()

tokenRefetchNeeded

protected static boolean tokenRefetchNeeded(java.io.IOException ex,
                                            java.net.InetSocketAddress targetAddr)

Should the block access token be refetched on an exception

Parameters:

ex - Exception received

targetAddr - Target datanode address from where exception was received

Returns:

true if block access token has expired or invalid and it should be refetched

read

public int read(long position,
                byte[] buffer,
                int offset,
                int length)
         throws java.io.IOException

Read bytes starting from the specified position.

Specified by:

read in interface org.apache.hadoop.fs.PositionedReadable

Overrides:

read in class org.apache.hadoop.fs.FSInputStream

Parameters:

position - start read from this position

buffer - read buffer

offset - offset into buffer

length - number of bytes to read

Returns:

actual number of bytes read

Throws:

java.io.IOException

reportCheckSumFailure

protected void reportCheckSumFailure(DFSUtilClient.CorruptedBlocks corruptedBlocks,
                                     int dataNodeCount,
                                     boolean isStriped)

DFSInputStream reports checksum failure. For replicated blocks, we have the following logic: Case I : client has tried multiple data nodes and at least one of the attempts has succeeded. We report the other failures as corrupted block to namenode. Case II: client has tried out all data nodes, but all failed. We only report if the total number of replica is 1. We do not report otherwise since this maybe due to the client is a handicapped client (who can not read). For erasure-coded blocks, each block in corruptedBlockMap is an internal block in a block group, and there is usually only one DataNode corresponding to each internal block. For this case we simply report the corrupted blocks to NameNode and ignore the above logic.

Parameters:

corruptedBlocks - map of corrupted blocks

dataNodeCount - number of data nodes who contains the block replicas

skip

public long skip(long n)
          throws java.io.IOException

Overrides:

skip in class java.io.InputStream

Throws:

java.io.IOException

seek

public void seek(long targetPos)
          throws java.io.IOException

Seek to a new arbitrary location

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Throws:

java.io.IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws java.io.IOException

Seek to given position on a node other than the current node. If a node other than the current node is found, then returns true. If another node could not be found, then returns false.

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

java.io.IOException

getPos

public long getPos()

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

available

public int available()
              throws java.io.IOException

Return the size of the remaining available bytes if the size is less than or equal to Integer.MAX_VALUE, otherwise, return Integer.MAX_VALUE.

Overrides:

available in class java.io.InputStream

Throws:

java.io.IOException

markSupported

public boolean markSupported()

We definitely don't support marks

Overrides:

markSupported in class java.io.InputStream

mark

public void mark(int readLimit)

Overrides:

mark in class java.io.InputStream

reset

public void reset()
           throws java.io.IOException

Overrides:

reset in class java.io.InputStream

Throws:

java.io.IOException

read

public int read(long position,
                java.nio.ByteBuffer buf)
         throws java.io.IOException

Specified by:

read in interface org.apache.hadoop.fs.ByteBufferPositionedReadable

Throws:

java.io.IOException

readFully

public void readFully(long position,
                      java.nio.ByteBuffer buf)
               throws java.io.IOException

Specified by:

readFully in interface org.apache.hadoop.fs.ByteBufferPositionedReadable

Throws:

java.io.IOException

getReadStatistics

public ReadStatistics getReadStatistics()

Get statistics about the reads which this DFSInputStream has done.

clearReadStatistics

public void clearReadStatistics()

Clear statistics about the reads which this DFSInputStream has done.

getFileEncryptionInfo

public org.apache.hadoop.fs.FileEncryptionInfo getFileEncryptionInfo()

closeCurrentBlockReaders

protected void closeCurrentBlockReaders()

setReadahead

public void setReadahead(java.lang.Long readahead)
                  throws java.io.IOException

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

Throws:

java.io.IOException

setDropBehind

public void setDropBehind(java.lang.Boolean dropBehind)
                   throws java.io.IOException

Specified by:

setDropBehind in interface org.apache.hadoop.fs.CanSetDropBehind

Throws:

java.io.IOException

read

public java.nio.ByteBuffer read(org.apache.hadoop.io.ByteBufferPool bufferPool,
                                int maxLength,
                                java.util.EnumSet<org.apache.hadoop.fs.ReadOption> opts)
                         throws java.io.IOException,
                                java.lang.UnsupportedOperationException

Specified by:

read in interface org.apache.hadoop.fs.HasEnhancedByteBufferAccess

Throws:

java.io.IOException

java.lang.UnsupportedOperationException

releaseBuffer

public void releaseBuffer(java.nio.ByteBuffer buffer)

Specified by:

releaseBuffer in interface org.apache.hadoop.fs.HasEnhancedByteBufferAccess

unbuffer

public void unbuffer()

Specified by:

unbuffer in interface org.apache.hadoop.fs.CanUnbuffer

hasCapability

public boolean hasCapability(java.lang.String capability)

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

maybeRegisterBlockRefresh

protected void maybeRegisterBlockRefresh()

Many DFSInputStreams can be opened and closed in quick succession, in which case they would be registered/deregistered but never need to be refreshed. Defers registering with the located block refresher, in order to avoid an additional source of unnecessary synchronization for short-lived DFSInputStreams.