Package org.apache.hadoop.fs

Class FSDataInputStream

java.lang.Object

java.io.InputStream

java.io.FilterInputStream

java.io.DataInputStream

org.apache.hadoop.fs.FSDataInputStream

All Implemented Interfaces:

java.io.Closeable, java.io.DataInput, java.lang.AutoCloseable, ByteBufferPositionedReadable, ByteBufferReadable, CanSetDropBehind, CanSetReadahead, CanUnbuffer, HasEnhancedByteBufferAccess, HasFileDescriptor, PositionedReadable, Seekable, IOStatisticsSource, StreamCapabilities

Direct Known Subclasses:

CryptoFSDataInputStream

@Public
@Stable
public class FSDataInputStream
extends java.io.DataInputStream
implements Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor, CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess, CanUnbuffer, StreamCapabilities, ByteBufferPositionedReadable, IOStatisticsSource

Utility that wraps a FSInputStream in a DataInputStream and buffers input through a BufferedInputStream.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FSDataInputStream.FadviseType	Type of file advise to be passed on to the underlying file system.

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

StreamCapabilities.StreamCapability

Field Summary

Fields inherited from class java.io.FilterInputStream

in

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Constructor Summary

Constructors

Constructor	Description
FSDataInputStream(java.io.InputStream in)

Method Summary

Modifier and Type	Method	Description
void	adviseFile(FSDataInputStream.FadviseType type, long offset, long count)	Specifies the kind of advise to provide for this stream and the file offsets to which they apply.
long	getChunkSize()	Returns the file chunk size.
long[]	getFidServers()	Returns the server IPs in which the file is stored.
java.lang.String	getFidStr()	Returns the file id as string.
java.io.FileDescriptor	getFileDescriptor()
long	getFileLength()	Returns the file length.
IOStatistics	getIOStatistics()	Get the IO Statistics of the nested stream, falling back to null if the stream does not implement the interface IOStatisticsSource.
long	getPos()	Get the current position in the input stream.
java.io.InputStream	getWrappedStream()	Get a reference to the wrapped input stream.
boolean	hasCapability(java.lang.String capability)	Query the stream for a specific capability.
int	maxReadSizeForVectorReads()	What is the largest size that we should group ranges together as?
int	minSeekForVectorReads()	What is the smallest reasonable seek?
int	read(long position, byte[] buffer, int offset, int length)	Read bytes from the given position in the stream to the given buffer.
int	read(long position, java.nio.ByteBuffer buf)	Reads up to buf.remaining() bytes into buf from a given position in the file and returns the number of bytes read.
int	read(java.nio.ByteBuffer buf)	Reads up to buf.remaining() bytes into buf.
java.nio.ByteBuffer	read(ByteBufferPool bufferPool, int maxLength)
java.nio.ByteBuffer	read(ByteBufferPool bufferPool, int maxLength, java.util.EnumSet<ReadOption> opts)	Get a ByteBuffer containing file data.
void	readFully(long position, byte[] buffer)	See readFully(long, byte[], int, int).
void	readFully(long position, byte[] buffer, int offset, int length)	Read bytes from the given position in the stream to the given buffer.
void	readFully(long position, java.nio.ByteBuffer buf)	Delegate to the underlying stream.
void	readVectored(java.util.List<? extends FileRange> ranges, java.util.function.IntFunction<java.nio.ByteBuffer> allocate)	Read fully a list of file ranges asynchronously from this file.
void	releaseBuffer(java.nio.ByteBuffer buffer)	Release a ByteBuffer which was created by the enhanced ByteBuffer read function.
void	seek(long desired)	Seek to the given offset.
boolean	seekToNewSource(long targetPos)	Seek to the given position on an alternate copy of the data.
void	setDropBehind(java.lang.Boolean dropBehind)	Configure whether the stream should drop the cache.
void	setReadahead(java.lang.Long readahead)	Set the readahead on this stream.
java.lang.String	toString()	String value.
void	unbuffer()	Reduce the buffering.

Methods inherited from class java.io.DataInputStream

read, read, readBoolean, readByte, readChar, readDouble, readFloat, readFully, readFully, readInt, readLine, readLong, readShort, readUnsignedByte, readUnsignedShort, readUTF, readUTF, skipBytes

Methods inherited from class java.io.FilterInputStream

available, close, mark, markSupported, read, reset, skip

Methods inherited from class java.io.InputStream

nullInputStream, readAllBytes, readNBytes, readNBytes, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FSDataInputStream

public FSDataInputStream(java.io.InputStream in)

Method Detail

seek

public void seek(long desired)
          throws java.io.IOException

Seek to the given offset.

Specified by:

seek in interface Seekable

Parameters:

desired - offset to seek to

Throws:

java.io.IOException - raised on errors performing I/O.

getPos

public long getPos()
            throws java.io.IOException

Get the current position in the input stream.

Specified by:

getPos in interface Seekable

Returns:

current position in the input stream

Throws:

java.io.IOException - raised on errors performing I/O.

read

public int read(long position,
                byte[] buffer,
                int offset,
                int length)
         throws java.io.IOException

Read bytes from the given position in the stream to the given buffer.

Specified by:

read in interface PositionedReadable

Parameters:

position - position in the input stream to seek

buffer - buffer into which data is read

offset - offset into the buffer in which data is written

length - maximum number of bytes to read

Returns:

total number of bytes read into the buffer, or -1 if there is no more data because the end of the stream has been reached

Throws:

java.io.IOException - IO problems.

readFully

public void readFully(long position,
                      byte[] buffer,
                      int offset,
                      int length)
               throws java.io.IOException

Read bytes from the given position in the stream to the given buffer. Continues to read until length bytes have been read.

Specified by:

readFully in interface PositionedReadable

Parameters:

position - position in the input stream to seek

buffer - buffer into which data is read

offset - offset into the buffer in which data is written

length - the number of bytes to read

Throws:

java.io.IOException - IO problems

java.io.EOFException - If the end of stream is reached while reading. If an exception is thrown an undetermined number of bytes in the buffer may have been written.

readFully

public void readFully(long position,
                      byte[] buffer)
               throws java.io.IOException

See readFully(long, byte[], int, int).

Specified by:

readFully in interface PositionedReadable

Parameters:

position - position within file

buffer - destination buffer

Throws:

java.io.IOException - IO problems.

java.io.EOFException - the end of the data was reached before the read operation completed

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws java.io.IOException

Seek to the given position on an alternate copy of the data.

Specified by:

seekToNewSource in interface Seekable

Parameters:

targetPos - position to seek to

Returns:

true if a new source is found, false otherwise

Throws:

java.io.IOException - raised on errors performing I/O.

getWrappedStream

@Public
@Stable
public java.io.InputStream getWrappedStream()

Get a reference to the wrapped input stream. Used by unit tests.

Returns:

the underlying input stream

read

public int read(java.nio.ByteBuffer buf)
         throws java.io.IOException

Description copied from interface: ByteBufferReadable

Reads up to buf.remaining() bytes into buf. Callers should use buf.limit(..) to control the size of the desired read.

After a successful call, buf.position() will be advanced by the number of bytes read and buf.limit() will be unchanged.

In the case of an exception, the state of the buffer (the contents of the buffer, the buf.position(), the buf.limit(), etc.) is undefined, and callers should be prepared to recover from this eventuality.

Callers should use StreamCapabilities.hasCapability(String) with StreamCapabilities.READBYTEBUFFER to check if the underlying stream supports this interface, otherwise they might get a UnsupportedOperationException.

Implementations should treat 0-length requests as legitimate, and must not signal an error upon their receipt.

Specified by:

read in interface ByteBufferReadable

Parameters:

buf - the ByteBuffer to receive the results of the read operation.

Returns:

the number of bytes read, possibly zero, or -1 if reach end-of-stream

Throws:

java.io.IOException - if there is some error performing the read

getFileDescriptor

public java.io.FileDescriptor getFileDescriptor()
                                         throws java.io.IOException

Specified by:

getFileDescriptor in interface HasFileDescriptor

Returns:

the FileDescriptor

Throws:

java.io.IOException - raised on errors performing I/O.

setReadahead

public void setReadahead(java.lang.Long readahead)
                  throws java.io.IOException,
                         java.lang.UnsupportedOperationException

Description copied from interface: CanSetReadahead

Set the readahead on this stream.

Specified by:

setReadahead in interface CanSetReadahead

Parameters:

readahead - The readahead to use. null means to use the default.

Throws:

java.io.IOException - If there was an error changing the dropBehind setting. UnsupportedOperationException If this stream doesn't support setting readahead.

java.lang.UnsupportedOperationException

setDropBehind

public void setDropBehind(java.lang.Boolean dropBehind)
                   throws java.io.IOException,
                          java.lang.UnsupportedOperationException

Description copied from interface: CanSetDropBehind

Configure whether the stream should drop the cache.

Specified by:

setDropBehind in interface CanSetDropBehind

Parameters:

dropBehind - Whether to drop the cache. null means to use the default value.

Throws:

java.io.IOException - If there was an error changing the dropBehind setting. UnsupportedOperationException If this stream doesn't support setting the drop-behind.

java.lang.UnsupportedOperationException

read

public java.nio.ByteBuffer read(ByteBufferPool bufferPool,
                                int maxLength,
                                java.util.EnumSet<ReadOption> opts)
                         throws java.io.IOException,
                                java.lang.UnsupportedOperationException

Description copied from interface: HasEnhancedByteBufferAccess

Get a ByteBuffer containing file data. This ByteBuffer may come from the stream itself, via a call like mmap, or it may come from the ByteBufferFactory which is passed in as an argument.

Specified by:

read in interface HasEnhancedByteBufferAccess

Parameters:

bufferPool - If this is non-null, it will be used to create a fallback ByteBuffer when the stream itself cannot create one.

maxLength - The maximum length of buffer to return. We may return a buffer which is shorter than this.

opts - Options to use when reading.

Returns:

We will always return an empty buffer if maxLength was 0, whether or not we are at EOF. If maxLength > 0, we will return null if the stream has reached EOF. Otherwise, we will return a ByteBuffer containing at least one byte. You must free this ByteBuffer when you are done with it by calling releaseBuffer on it. The buffer will continue to be readable until it is released in this manner. However, the input stream's close method may warn about unclosed buffers.

Throws:

java.io.IOException - if there was an error reading.

java.lang.UnsupportedOperationException - if factory was null, and we needed an external byte buffer.

read

public final java.nio.ByteBuffer read(ByteBufferPool bufferPool,
                                      int maxLength)
                               throws java.io.IOException,
                                      java.lang.UnsupportedOperationException

Throws:

java.io.IOException

java.lang.UnsupportedOperationException

releaseBuffer

public void releaseBuffer(java.nio.ByteBuffer buffer)

Description copied from interface: HasEnhancedByteBufferAccess

Release a ByteBuffer which was created by the enhanced ByteBuffer read function. You must not continue using the ByteBuffer after calling this function.

Specified by:

releaseBuffer in interface HasEnhancedByteBufferAccess

Parameters:

buffer - The ByteBuffer to release.

unbuffer

public void unbuffer()

Description copied from interface: CanUnbuffer

Reduce the buffering. This will also free sockets and file descriptors held by the stream, if possible.

Specified by:

unbuffer in interface CanUnbuffer

hasCapability

public boolean hasCapability(java.lang.String capability)

Description copied from interface: StreamCapabilities

Query the stream for a specific capability.

Specified by:

hasCapability in interface StreamCapabilities

Parameters:

capability - string to query the stream support for.

Returns:

True if the stream supports capability.

toString

public java.lang.String toString()

String value. Includes the string value of the inner stream

Overrides:

toString in class java.lang.Object

Returns:

the stream

read

public int read(long position,
                java.nio.ByteBuffer buf)
         throws java.io.IOException

Description copied from interface: ByteBufferPositionedReadable

Reads up to buf.remaining() bytes into buf from a given position in the file and returns the number of bytes read. Callers should use buf.limit(...) to control the size of the desired read and buf.position(...) to control the offset into the buffer the data should be written to.

After a successful call, buf.position() will be advanced by the number of bytes read and buf.limit() will be unchanged.

In the case of an exception, the state of the buffer (the contents of the buffer, the buf.position(), the buf.limit(), etc.) is undefined, and callers should be prepared to recover from this eventuality.

Callers should use StreamCapabilities.hasCapability(String) with StreamCapabilities.PREADBYTEBUFFER to check if the underlying stream supports this interface, otherwise they might get a UnsupportedOperationException.

Implementations should treat 0-length requests as legitimate, and must not signal an error upon their receipt.

This does not change the current offset of a file, and is thread-safe.

Specified by:

read in interface ByteBufferPositionedReadable

Parameters:

position - position within file

buf - the ByteBuffer to receive the results of the read operation.

Returns:

the number of bytes read, possibly zero, or -1 if reached end-of-stream

Throws:

java.io.IOException - if there is some error performing the read

readFully

public void readFully(long position,
                      java.nio.ByteBuffer buf)
               throws java.io.IOException

Delegate to the underlying stream.

Specified by:

readFully in interface ByteBufferPositionedReadable

Parameters:

position - position within file

buf - the ByteBuffer to receive the results of the read operation.

Throws:

java.io.IOException - on a failure from the nested stream.

java.lang.UnsupportedOperationException - if the inner stream does not support this operation.

See Also:

ByteBufferPositionedReadable.read(long, ByteBuffer)

getIOStatistics

public IOStatistics getIOStatistics()

Get the IO Statistics of the nested stream, falling back to null if the stream does not implement the interface IOStatisticsSource.

Specified by:

getIOStatistics in interface IOStatisticsSource

Returns:

an IOStatistics instance or null

minSeekForVectorReads

public int minSeekForVectorReads()

Description copied from interface: PositionedReadable

What is the smallest reasonable seek?

Specified by:

minSeekForVectorReads in interface PositionedReadable

Returns:

the minimum number of bytes

maxReadSizeForVectorReads

public int maxReadSizeForVectorReads()

Description copied from interface: PositionedReadable

What is the largest size that we should group ranges together as?

Specified by:

maxReadSizeForVectorReads in interface PositionedReadable

Returns:

the number of bytes to read at once

readVectored

public void readVectored(java.util.List<? extends FileRange> ranges,
                         java.util.function.IntFunction<java.nio.ByteBuffer> allocate)
                  throws java.io.IOException

Description copied from interface: PositionedReadable

Read fully a list of file ranges asynchronously from this file. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers. As a result of the call, each range will have FileRange.setData(CompletableFuture) called with a future that when complete will have a ByteBuffer with the data from the file's range.

The position returned by getPos() after readVectored() is undefined.

If a file is changed while the readVectored() operation is in progress, the output is undefined. Some ranges may have old data, some may have new and some may have both.

While a readVectored() operation is in progress, normal read api calls may block.

Specified by:

readVectored in interface PositionedReadable

Parameters:

ranges - the byte ranges to read

allocate - the function to allocate ByteBuffer

Throws:

java.io.IOException - any IOE.

adviseFile

public void adviseFile(FSDataInputStream.FadviseType type,
                       long offset,
                       long count)
                throws java.io.IOException

Specifies the kind of advise to provide for this stream and the file offsets to which they apply. The default implementation does nothing. Sub classes can override this behavior.

Parameters:

type - advise type

offset - starting file offset

count - number of bytes starting from the offset

Throws:

java.io.IOException

getFileLength

public long getFileLength()
                   throws java.io.IOException

Returns the file length.

Returns:

file length

Throws:

java.io.IOException

getFidStr

public java.lang.String getFidStr()

Returns the file id as string.

Returns:

file id as string

getFidServers

public long[] getFidServers()

Returns the server IPs in which the file is stored. Each IP is stored in a long. For e.g., the first 4 bytes can be used to store the IP in hexadecimal format and the last 4 bytes to store the port number.

Returns:

array of server IPs in which the file is stored

getChunkSize

public long getChunkSize()

Returns the file chunk size.

Returns:

file chunk size