Package org.apache.hadoop.fs

Class FSInputChecker

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.FSInputChecker

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, PositionedReadable, Seekable

@LimitedPrivate("HDFS")
@Unstable
public abstract class FSInputChecker
extends FSInputStream

This is a generic input stream for verifying checksums for data before it is read by a user.

Field Summary

Fields

Modifier and Type	Field	Description
protected static int	CHECKSUM_SIZE
protected Path	file	The file name from which data is read from
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	FSInputChecker(Path file, int numOfRetries)	Constructor
protected	FSInputChecker(Path file, int numOfRetries, boolean verifyChecksum, java.util.zip.Checksum sum, int chunkSize, int checksumSize)	Constructor

Method Summary

Modifier and Type	Method	Description
int	available()
static long	checksum2long(byte[] checksum)	Deprecated.
protected abstract long	getChunkPosition(long pos)	Return position of beginning of chunk containing pos.
long	getPos()	Return the current offset from the start of the file
void	mark(int readlimit)
boolean	markSupported()
protected boolean	needChecksum()	Return true if there is a need for checksum verification.
int	read()	Read one checksum-verified byte
int	read(byte[] b, int off, int len)	Read checksum verified bytes from this byte-input stream into the specified byte array, starting at the given offset.
protected int	readAndDiscard(int len)	Like read(byte[], int, int), but does not provide a dest buffer, so the read data is discarded.
protected abstract int	readChunk(long pos, byte[] buf, int offset, int len, byte[] checksum)	Reads in checksum chunks into buf at offset and checksum into checksum.
protected static int	readFully(java.io.InputStream stm, byte[] buf, int offset, int len)	A utility function that tries to read up to len bytes from stm
void	reset()
void	seek(long pos)	Seek to the given position in the stream.
protected void	set(boolean verifyChecksum, java.util.zip.Checksum sum, int maxChunkSize, int checksumSize)	Set the checksum related parameters
long	skip(long n)	Skips over and discards n bytes of data from the input stream.

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, readFully, seekToNewSource, toString, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

close, nullInputStream, read, readAllBytes, readNBytes, readNBytes, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

maxReadSizeForVectorReads, minSeekForVectorReads, readVectored

Field Detail

LOG

public static final org.slf4j.Logger LOG

file

protected Path file

The file name from which data is read from

CHECKSUM_SIZE

protected static final int CHECKSUM_SIZE

See Also:

Constant Field Values

Constructor Detail

FSInputChecker

protected FSInputChecker(Path file,
                         int numOfRetries)

Constructor

Parameters:

file - The name of the file to be read

numOfRetries - Number of read retries when ChecksumError occurs

FSInputChecker

protected FSInputChecker(Path file,
                         int numOfRetries,
                         boolean verifyChecksum,
                         java.util.zip.Checksum sum,
                         int chunkSize,
                         int checksumSize)

Constructor

Parameters:

file - The name of the file to be read

numOfRetries - Number of read retries when ChecksumError occurs

sum - the type of Checksum engine

chunkSize - maximun chunk size

checksumSize - the number byte of each checksum

verifyChecksum - verify check sum.

Method Detail

readChunk

protected abstract int readChunk(long pos,
                                 byte[] buf,
                                 int offset,
                                 int len,
                                 byte[] checksum)
                          throws java.io.IOException

Reads in checksum chunks into buf at offset and checksum into checksum. Since checksums can be disabled, there are two cases implementors need to worry about: (a) needChecksum() will return false: - len can be any positive value - checksum will be null Implementors should simply pass through to the underlying data stream. or (b) needChecksum() will return true: - len >= maxChunkSize - checksum.length is a multiple of CHECKSUM_SIZE Implementors should read an integer number of data chunks into buf. The amount read should be bounded by len or by checksum.length / CHECKSUM_SIZE * maxChunkSize. Note that len may be a value that is not a multiple of maxChunkSize, in which case the implementation may return less than len. The method is used for implementing read, therefore, it should be optimized for sequential reading.

Parameters:

pos - chunkPos

buf - destination buffer

offset - offset in buf at which to store data

len - maximum number of bytes to read

checksum - the data buffer into which to write checksums

Returns:

number of bytes read

Throws:

java.io.IOException - raised on errors performing I/O.

getChunkPosition

protected abstract long getChunkPosition(long pos)

Return position of beginning of chunk containing pos.

Parameters:

pos - a position in the file

Returns:

the starting position of the chunk which contains the byte

needChecksum

protected boolean needChecksum()

Return true if there is a need for checksum verification.

Returns:

if there is a need for checksum verification true, not false.

read

public int read()
         throws java.io.IOException

Read one checksum-verified byte

Specified by:

read in class java.io.InputStream

Returns:

the next byte of data, or -1 if the end of the stream is reached.

Throws:

java.io.IOException - if an I/O error occurs.

read

public int read(byte[] b,
                int off,
                int len)
         throws java.io.IOException

Read checksum verified bytes from this byte-input stream into the specified byte array, starting at the given offset.

This method implements the general contract of the corresponding read method of the InputStream class. As an additional convenience, it attempts to read as many bytes as possible by repeatedly invoking the read method of the underlying stream. This iterated read continues until one of the following conditions becomes true:

• The specified number of bytes have been read,
• The read method of the underlying stream returns -1, indicating end-of-file.

If the first read on the underlying stream returns -1 to indicate end-of-file then this method returns -1. Otherwise this method returns the number of bytes actually read.

Overrides:

read in class java.io.InputStream

Parameters:

b - destination buffer.

off - offset at which to start storing bytes.

len - maximum number of bytes to read.

Returns:

the number of bytes read, or -1 if the end of the stream has been reached.

Throws:

java.io.IOException - if an I/O error occurs. ChecksumException if any checksum error occurs

readAndDiscard

protected final int readAndDiscard(int len)
                            throws java.io.IOException

Like read(byte[], int, int), but does not provide a dest buffer, so the read data is discarded.

Parameters:

len - maximum number of bytes to read.

Returns:

the number of bytes read.

Throws:

java.io.IOException - if an I/O error occurs.

checksum2long

@Deprecated
public static long checksum2long(byte[] checksum)

Deprecated.

Convert a checksum byte array to a long This is deprecated since 0.22 since it is no longer in use by this class.

Parameters:

checksum - check sum.

Returns:

crc.

getPos

public long getPos()
            throws java.io.IOException

Description copied from class: FSInputStream

Return the current offset from the start of the file

Specified by:

getPos in interface Seekable

Specified by:

getPos in class FSInputStream

Returns:

offset from the start of the file.

Throws:

java.io.IOException - raised on errors performing I/O.

available

public int available()
              throws java.io.IOException

Overrides:

available in class java.io.InputStream

Throws:

java.io.IOException

skip

public long skip(long n)
          throws java.io.IOException

Skips over and discards n bytes of data from the input stream.

This method may skip more bytes than are remaining in the backing file. This produces no exception and the number of bytes skipped may include some number of bytes that were beyond the EOF of the backing file. Attempting to read from the stream after skipping past the end will result in -1 indicating the end of the file.

If n is negative, no bytes are skipped.

Overrides:

skip in class java.io.InputStream

Parameters:

n - the number of bytes to be skipped.

Returns:

the actual number of bytes skipped.

Throws:

java.io.IOException - if an I/O error occurs. ChecksumException if the chunk to skip to is corrupted

seek

public void seek(long pos)
          throws java.io.IOException

Seek to the given position in the stream. The next read() will be from that position.

This method may seek past the end of the file. This produces no exception and an attempt to read from the stream will result in -1 indicating the end of the file.

Specified by:

seek in interface Seekable

Specified by:

seek in class FSInputStream

Parameters:

pos - the position to seek to.

Throws:

java.io.IOException - if an I/O error occurs. ChecksumException if the chunk to seek to is corrupted

readFully

protected static int readFully(java.io.InputStream stm,
                               byte[] buf,
                               int offset,
                               int len)
                        throws java.io.IOException

A utility function that tries to read up to len bytes from stm

Parameters:

stm - an input stream

buf - destination buffer

offset - offset at which to store data

len - number of bytes to read

Returns:

actual number of bytes read

Throws:

java.io.IOException - if there is any IO error

set

protected final void set(boolean verifyChecksum,
                         java.util.zip.Checksum sum,
                         int maxChunkSize,
                         int checksumSize)

Set the checksum related parameters

Parameters:

verifyChecksum - whether to verify checksum

sum - which type of checksum to use

maxChunkSize - maximun chunk size

checksumSize - checksum size

markSupported

public final boolean markSupported()

Overrides:

markSupported in class java.io.InputStream

mark

public final void mark(int readlimit)

Overrides:

mark in class java.io.InputStream

reset

public final void reset()
                 throws java.io.IOException

Overrides:

reset in class java.io.InputStream

Throws:

java.io.IOException