Package org.apache.hadoop.tools

Class SimpleCopyListing

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.CopyListing

org.apache.hadoop.tools.SimpleCopyListing

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class SimpleCopyListing
extends CopyListing

The SimpleCopyListing is responsible for making the exhaustive list of all files/directories under its specified list of input-paths. These are written into the specified copy-listing file. Note: The SimpleCopyListing doesn't handle wild-cards in the input-paths.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.tools.CopyListing

CopyListing.AclsNotSupportedException, CopyListing.XAttrsNotSupportedException

Field Summary

Fields

Modifier and Type	Field	Description
static int	DEFAULT_FILE_STATUS_SIZE
static boolean	DEFAULT_LISTING_CHECK
static boolean	DEFAULT_RANDOMIZE_FILE_LISTING
static org.slf4j.Logger	LOG
static java.util.Map<java.lang.String,java.util.Set<java.lang.String>>	loopLocator

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.security.Credentials credentials)	Protected constructor, to initialize configuration.
protected	SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.security.Credentials credentials, int numListstatusThreads, int fileStatusLimit, boolean randomizeFileListing)
protected	SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.security.Credentials credentials, org.apache.hadoop.tools.DistCpSync distCpSync)

Method Summary

Modifier and Type	Method	Description
protected void	doBuildListing(org.apache.hadoop.fs.Path pathToListingFile, DistCpContext context)	The interface to be implemented by sub-classes, to create the source/target file listing.
protected void	doBuildListing(org.apache.hadoop.io.SequenceFile.Writer fileListWriter, DistCpContext context)	Collect the list of <sourceRelativePath, sourceFileStatus> to be copied and write to the sequence file.
protected void	doBuildListingWithSnapshotDiff(org.apache.hadoop.io.SequenceFile.Writer fileListWriter, DistCpContext context)	Build a copy list based on the snapshot diff report.
protected long	getBytesToCopy()	Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
protected long	getNumberOfPaths()	Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
protected boolean	shouldCopy(org.apache.hadoop.fs.Path path)	Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME
protected void	validatePaths(DistCpContext context)	Validate input and output paths

Methods inherited from class org.apache.hadoop.tools.CopyListing

buildListing, getCopyListing, getCredentials, getFileListingKey, getFileListingValue, setCredentials

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

DEFAULT_FILE_STATUS_SIZE

public static final int DEFAULT_FILE_STATUS_SIZE

See Also:

Constant Field Values

DEFAULT_RANDOMIZE_FILE_LISTING

public static final boolean DEFAULT_RANDOMIZE_FILE_LISTING

See Also:

Constant Field Values

DEFAULT_LISTING_CHECK

public static final boolean DEFAULT_LISTING_CHECK

See Also:

Constant Field Values

loopLocator

public static java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator

Constructor Detail

SimpleCopyListing

protected SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration,
                            org.apache.hadoop.security.Credentials credentials)

Protected constructor, to initialize configuration.

Parameters:

configuration - The input configuration, with which the source/target FileSystems may be accessed.

credentials - - Credentials object on which the FS delegation tokens are cached. If null delegation token caching is skipped

SimpleCopyListing

@VisibleForTesting
protected SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration,
                            org.apache.hadoop.security.Credentials credentials,
                            int numListstatusThreads,
                            int fileStatusLimit,
                            boolean randomizeFileListing)

SimpleCopyListing

protected SimpleCopyListing(org.apache.hadoop.conf.Configuration configuration,
                            org.apache.hadoop.security.Credentials credentials,
                            org.apache.hadoop.tools.DistCpSync distCpSync)

Method Detail

validatePaths

protected void validatePaths(DistCpContext context)
                      throws java.io.IOException,
                             org.apache.hadoop.tools.CopyListing.InvalidInputException

Description copied from class: CopyListing

Validate input and output paths

Specified by:

validatePaths in class CopyListing

Parameters:

context - - Distcp context

Throws:

java.io.IOException - any Exception with FS

org.apache.hadoop.tools.CopyListing.InvalidInputException - If inputs are invalid

doBuildListing

protected void doBuildListing(org.apache.hadoop.fs.Path pathToListingFile,
                              DistCpContext context)
                       throws java.io.IOException

Description copied from class: CopyListing

The interface to be implemented by sub-classes, to create the source/target file listing.

Specified by:

doBuildListing in class CopyListing

Parameters:

pathToListingFile - Path on HDFS where the listing file is written.

context - - Distcp context

Throws:

java.io.IOException - Thrown on failure to create the listing file.

doBuildListingWithSnapshotDiff

@VisibleForTesting
protected void doBuildListingWithSnapshotDiff(org.apache.hadoop.io.SequenceFile.Writer fileListWriter,
                                              DistCpContext context)
                                       throws java.io.IOException

Build a copy list based on the snapshot diff report. Any file/directory changed or created will be in the list. Deleted files/directories will not be in the list, since they are handled by DistCpSync.sync(). An item can be created/modified and renamed, in which case, the target path is put into the list.

Parameters:

fileListWriter - the list for holding processed results

context - The DistCp context with associated input options

Throws:

java.io.IOException - if unable to construct the fileList

doBuildListing

@VisibleForTesting
protected void doBuildListing(org.apache.hadoop.io.SequenceFile.Writer fileListWriter,
                              DistCpContext context)
                       throws java.io.IOException

Collect the list of <sourceRelativePath, sourceFileStatus> to be copied and write to the sequence file. In essence, any file or directory that need to be copied or sync-ed is written as an entry to the sequence file, with the possible exception of the source root: when either -update (sync) or -overwrite switch is specified, and if the the source root is a directory, then the source root entry is not written to the sequence file, because only the contents of the source directory need to be copied in this case. See DistCpUtils.getRelativePath(org.apache.hadoop.tools.FileListingEntry) for how relative path is computed. See computeSourceRootPath method for how the root path of the source is computed.

Parameters:

fileListWriter -

context - The distcp context with associated input options

Throws:

java.io.IOException - if unable to construct the fileList

shouldCopy

protected boolean shouldCopy(org.apache.hadoop.fs.Path path)

Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME

Parameters:

path - - Path being considered for copy while building the file listing

Returns:

- True if the path should be considered for copy, false otherwise

getBytesToCopy

protected long getBytesToCopy()

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

Specified by:

getBytesToCopy in class CopyListing

Returns:

total bytes to copy

getNumberOfPaths

protected long getNumberOfPaths()

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

Specified by:

getNumberOfPaths in class CopyListing

Returns:

Total number of paths to distcp