Package org.apache.hadoop.hdfs.server.namenode

Class FSTreeTraverser

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser

@Private
public abstract class FSTreeTraverser
extends java.lang.Object

FSTreeTraverser traverse directory recursively and process files in batches.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FSTreeTraverser.TraverseInfo	Class will represent the additional info required for traverse.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
FSTreeTraverser(FSDirectory dir, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
protected abstract boolean	canTraverseDir(INode inode)	Check whether dir is traversable or not.
protected abstract void	checkINodeReady(long startId)	Check whether inode is ready for traverse.
protected abstract void	checkPauseForTesting()
FSDirectory	getFSDirectory()
protected abstract boolean	processFileInode(INode inode, FSTreeTraverser.TraverseInfo traverseInfo)	Process an Inode.
protected void	readLock()
protected void	readUnlock()
protected abstract boolean	shouldSubmitCurrentBatch()	Check whether current batch can be submitted for the processing.
protected abstract void	submitCurrentBatch(java.lang.Long startId)	Submit the current batch for processing.
protected abstract void	throttle()	Throttles the FSTreeTraverser.
protected void	traverseDir(INodeDirectory parent, long startId, byte[] startAfter, FSTreeTraverser.TraverseInfo traverseInfo)	Iterate through all files directly inside parent, and recurse down directories.
protected INode	traverseDirInt(long startId, INode curr, java.util.List<byte[]> startAfters, FSTreeTraverser.TraverseInfo traverseInfo)	Iterates the parent directory, and add direct children files to current batch.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

FSTreeTraverser

public FSTreeTraverser(FSDirectory dir,
                       org.apache.hadoop.conf.Configuration conf)

Method Detail

getFSDirectory

public FSDirectory getFSDirectory()

traverseDir

protected void traverseDir(INodeDirectory parent,
                           long startId,
                           byte[] startAfter,
                           FSTreeTraverser.TraverseInfo traverseInfo)
                    throws java.io.IOException,
                           java.lang.InterruptedException

Iterate through all files directly inside parent, and recurse down directories. The listing is done in batch, and can optionally start after a position. The iteration of the inode tree is done in a depth-first fashion. But instead of holding all INodeDirectory's in memory on the fly, only the path components to the current inode is held. This is to reduce memory consumption.

Parameters:

parent - The inode id of parent directory

startId - Id of the start inode.

startAfter - Full path of a file the traverse should start after.

traverseInfo - info which may required for processing the child's.

Throws:

java.io.IOException

java.lang.InterruptedException

traverseDirInt

protected INode traverseDirInt(long startId,
                               INode curr,
                               java.util.List<byte[]> startAfters,
                               FSTreeTraverser.TraverseInfo traverseInfo)
                        throws java.io.IOException,
                               java.lang.InterruptedException

Iterates the parent directory, and add direct children files to current batch. If batch size meets configured threshold, current batch will be submitted for the processing.

Locks could be released and reacquired when a batch submission is finished.

Parameters:

startId - Id of the start inode.

Returns:

The inode which was just processed, if lock is held in the entire process. Null if lock is released.

Throws:

java.io.IOException

java.lang.InterruptedException

readLock

protected void readLock()

readUnlock

protected void readUnlock()

checkPauseForTesting

protected abstract void checkPauseForTesting()
                                      throws java.lang.InterruptedException

Throws:

java.lang.InterruptedException

processFileInode

protected abstract boolean processFileInode(INode inode,
                                            FSTreeTraverser.TraverseInfo traverseInfo)
                                     throws java.io.IOException,
                                            java.lang.InterruptedException

Process an Inode. Add to current batch if it's a file, no-op otherwise.

Parameters:

inode - the inode

Returns:

true if inode is added to currentBatch and should be process for next operation. false otherwise: could be inode is not a file.

Throws:

java.io.IOException

java.lang.InterruptedException

shouldSubmitCurrentBatch

protected abstract boolean shouldSubmitCurrentBatch()

Check whether current batch can be submitted for the processing.

Returns:

true if batch size meets the condition, otherwise false.

checkINodeReady

protected abstract void checkINodeReady(long startId)
                                 throws java.io.IOException

Check whether inode is ready for traverse. Throws IOE if it's not.

Parameters:

startId - Id of the start inode.

Throws:

java.io.IOException

submitCurrentBatch

protected abstract void submitCurrentBatch(java.lang.Long startId)
                                    throws java.io.IOException,
                                           java.lang.InterruptedException

Submit the current batch for processing.

Parameters:

startId - Id of the start inode.

Throws:

java.io.IOException

java.lang.InterruptedException

throttle

protected abstract void throttle()
                          throws java.lang.InterruptedException

Throttles the FSTreeTraverser.

Throws:

java.lang.InterruptedException

canTraverseDir

protected abstract boolean canTraverseDir(INode inode)
                                   throws java.io.IOException

Check whether dir is traversable or not.

Parameters:

inode - Dir inode

Returns:

true if dir is traversable otherwise false.

Throws:

java.io.IOException