Package org.apache.hadoop.tools.util

Class DistCpUtils

java.lang.Object

org.apache.hadoop.tools.util.DistCpUtils

public class DistCpUtils
extends java.lang.Object

Utility functions used in DistCp.

Constructor Summary

Constructors

Constructor	Description
DistCpUtils()

Method Summary

Modifier and Type	Method	Description
static void	checkFileSystemAclSupport(org.apache.hadoop.fs.FileSystem fs)	Determines if a file system supports ACLs by running a canary getAclStatus request on the file system root.
static void	checkFileSystemXAttrSupport(org.apache.hadoop.fs.FileSystem fs)	Determines if a file system supports XAttrs by running a getXAttrs request on the file system root.
static CopyMapper.ChecksumComparison	checksumsAreEqual(org.apache.hadoop.fs.FileSystem sourceFS, org.apache.hadoop.fs.Path source, org.apache.hadoop.fs.FileChecksum sourceChecksum, org.apache.hadoop.fs.FileSystem targetFS, org.apache.hadoop.fs.Path target, long sourceLen)	Utility to compare checksums for the paths specified.
static void	compareFileLengthsAndChecksums(long srcLen, org.apache.hadoop.fs.FileSystem sourceFS, org.apache.hadoop.fs.Path source, org.apache.hadoop.fs.FileChecksum sourceChecksum, org.apache.hadoop.fs.FileSystem targetFS, org.apache.hadoop.fs.Path target, boolean skipCrc, long targetLen)	Utility to compare file lengths and checksums for source and target.
static org.apache.hadoop.fs.Path	discloseRelativeSymlink(org.apache.hadoop.fs.FileStatus stat)
static java.util.List<org.apache.hadoop.fs.permission.AclEntry>	getAcl(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.FileStatus fileStatus)	Returns a file's full logical ACL.
static long	getFileSize(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration configuration)	Retrieves size of the file at the specified path.
static java.text.DecimalFormat	getFormatter()
static int	getInt(org.apache.hadoop.conf.Configuration configuration, java.lang.String label)	Utility to retrieve a specified key from a Configuration.
static long	getLong(org.apache.hadoop.conf.Configuration configuration, java.lang.String label)	Utility to retrieve a specified key from a Configuration.
static FileListingEntry	getOriginalFileStatus(org.apache.hadoop.fs.FileStatus sourceStatus, org.apache.hadoop.conf.Configuration conf, boolean keepLinks, java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator)
static java.lang.String	getRelativePath(org.apache.hadoop.fs.Path sourceRootPath, org.apache.hadoop.fs.Path childPath)
static java.lang.String	getRelativePath(FileListingEntry listingEntry)	Gets relative path of child path with respect to a root path For ex.
static java.lang.String	getRelativePath(FileListingEntry root, FileListingEntry child)
static org.apache.hadoop.fs.Path	getSplitChunkPath(org.apache.hadoop.fs.Path targetFile, CopyListingFileStatus srcFileStatus)
static java.lang.Class<? extends org.apache.hadoop.mapreduce.InputFormat>	getStrategy(org.apache.hadoop.conf.Configuration conf, DistCpContext context)	Returns the class that implements a copy strategy.
static java.lang.String	getStringDescriptionFor(long nBytes)
static org.apache.hadoop.fs.Path	getTmpFile(org.apache.hadoop.fs.Path target, org.apache.hadoop.mapreduce.Mapper.Context context, boolean splitSource)
static java.util.Map<java.lang.String,byte[]>	getXAttrs(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path)	Returns a file's all xAttrs.
static boolean	isLoop(org.apache.hadoop.fs.FileStatus src, java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator)
static boolean	isParentLoop(org.apache.hadoop.fs.FileStatus sourceStatus, org.apache.hadoop.conf.Configuration conf)
static java.lang.String	packAttributes(java.util.EnumSet<DistCpOptions.FileAttribute> attributes)	Pack file preservation attributes into a string, containing just the first character of each preservation attribute
static FileListingEntry	pathToFileListingEntry(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FileSystem fs)
static void	preserve(org.apache.hadoop.fs.FileSystem targetFS, org.apache.hadoop.fs.Path path, CopyListingFileStatus srcFileStatus, java.util.EnumSet<DistCpOptions.FileAttribute> attributes, boolean preserveRawXattrs)	Preserve attribute on file matching that of the file status being sent as argument.
static <T> void	publish(org.apache.hadoop.conf.Configuration configuration, java.lang.String label, T value)	Utility to publish a value to a configuration.
static org.apache.hadoop.fs.Path	sortListing(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path sourceListing)	Sort sequence file containing FileStatus and Text as key and value respectively.
static void	sortListing(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path sourceListing, org.apache.hadoop.fs.Path output)	Sort sequence file containing FileStatus and Text as key and value respectively, saving the result to the output path, which will be deleted first.
static void	toCopyListingFileStatus(org.apache.hadoop.fs.FileSystem fileSystem, FileListingEntry listingEntry, boolean preserveAcls, boolean preserveXAttrs, boolean preserveRawXAttrs, int blocksPerChunk)	Converts FileStatus to a list of CopyListingFileStatus.
static CopyListingFileStatus	toCopyListingFileStatusHelper(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.FileStatus fileStatus, boolean preserveAcls, boolean preserveXAttrs, boolean preserveRawXAttrs, long chunkOffset, long chunkLength)	Converts a FileStatus to a CopyListingFileStatus.
static java.util.EnumSet<DistCpOptions.FileAttribute>	unpackAttributes(java.lang.String attributes)	Unpacks preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistCpUtils

public DistCpUtils()

Method Detail

getFileSize

public static long getFileSize(org.apache.hadoop.fs.Path path,
                               org.apache.hadoop.conf.Configuration configuration)
                        throws java.io.IOException

Retrieves size of the file at the specified path.

Parameters:

path - The path of the file whose size is sought.

configuration - Configuration, to retrieve the appropriate FileSystem.

Returns:

The file-size, in number of bytes.

Throws:

java.io.IOException

publish

public static <T> void publish(org.apache.hadoop.conf.Configuration configuration,
                               java.lang.String label,
                               T value)

Utility to publish a value to a configuration.

Type Parameters:

T - The type of the value.

Parameters:

configuration - The Configuration to which the value must be written.

label - The label for the value being published.

value - The value being published.

getInt

public static int getInt(org.apache.hadoop.conf.Configuration configuration,
                         java.lang.String label)

Utility to retrieve a specified key from a Configuration. Throw exception if not found.

Parameters:

configuration - The Configuration in which the key is sought.

label - The key being sought.

Returns:

Integer value of the key.

getLong

public static long getLong(org.apache.hadoop.conf.Configuration configuration,
                           java.lang.String label)

Utility to retrieve a specified key from a Configuration. Throw exception if not found.

Parameters:

configuration - The Configuration in which the key is sought.

label - The key being sought.

Returns:

Long value of the key.

getStrategy

public static java.lang.Class<? extends org.apache.hadoop.mapreduce.InputFormat> getStrategy(org.apache.hadoop.conf.Configuration conf,
                                                                                             DistCpContext context)

Returns the class that implements a copy strategy. Looks up the implementation for a particular strategy from distcp-default.xml

Parameters:

conf - - Configuration object

context - - Distcp context with associated input options

Returns:

Class implementing the strategy specified in options.

getRelativePath

public static java.lang.String getRelativePath(FileListingEntry listingEntry)

Gets relative path of child path with respect to a root path For ex. If childPath = /tmp/abc/xyz/file and sourceRootPath = /tmp/abc Relative path would be /xyz/file If childPath = /file and sourceRootPath = / Relative path would be /file

Parameters:

listingEntry - - listingEntry

Returns:

- Relative portion of the child path (always prefixed with / unless it is empty

getRelativePath

public static java.lang.String getRelativePath(org.apache.hadoop.fs.Path sourceRootPath,
                                               org.apache.hadoop.fs.Path childPath)

getRelativePath

public static java.lang.String getRelativePath(FileListingEntry root,
                                               FileListingEntry child)

pathToFileListingEntry

public static FileListingEntry pathToFileListingEntry(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.FileSystem fs)
                                               throws java.io.IOException

Throws:

java.io.IOException

packAttributes

public static java.lang.String packAttributes(java.util.EnumSet<DistCpOptions.FileAttribute> attributes)

Pack file preservation attributes into a string, containing just the first character of each preservation attribute

Parameters:

attributes - - Attribute set to preserve

Returns:

- String containing first letters of each attribute to preserve

unpackAttributes

public static java.util.EnumSet<DistCpOptions.FileAttribute> unpackAttributes(java.lang.String attributes)

Unpacks preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve

Parameters:

attributes - - Attribute string

Returns:

- Attribute set

preserve

public static void preserve(org.apache.hadoop.fs.FileSystem targetFS,
                            org.apache.hadoop.fs.Path path,
                            CopyListingFileStatus srcFileStatus,
                            java.util.EnumSet<DistCpOptions.FileAttribute> attributes,
                            boolean preserveRawXattrs)
                     throws java.io.IOException

Preserve attribute on file matching that of the file status being sent as argument. Barring the block size, all the other attributes are preserved by this function

Parameters:

targetFS - - File system

path - - Path that needs to preserve original file status

srcFileStatus - - Original file status

attributes - - Attribute set that needs to be preserved

preserveRawXattrs - if true, raw.* xattrs should be preserved

Throws:

java.io.IOException - - Exception if any (particularly relating to group/owner change or any transient error)

getAcl

public static java.util.List<org.apache.hadoop.fs.permission.AclEntry> getAcl(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                              org.apache.hadoop.fs.FileStatus fileStatus)
                                                                       throws java.io.IOException

Returns a file's full logical ACL.

Parameters:

fileSystem - FileSystem containing the file

fileStatus - FileStatus of file

Returns:

List containing full logical ACL

Throws:

java.io.IOException - if there is an I/O error

getXAttrs

public static java.util.Map<java.lang.String,byte[]> getXAttrs(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                     org.apache.hadoop.fs.Path path)
                                                              throws java.io.IOException

Returns a file's all xAttrs.

Parameters:

fileSystem - FileSystem containing the file

path - file path

Returns:

Map containing all xAttrs

Throws:

java.io.IOException - if there is an I/O error

toCopyListingFileStatus

public static void toCopyListingFileStatus(org.apache.hadoop.fs.FileSystem fileSystem,
                                           FileListingEntry listingEntry,
                                           boolean preserveAcls,
                                           boolean preserveXAttrs,
                                           boolean preserveRawXAttrs,
                                           int blocksPerChunk)
                                    throws java.io.IOException

Converts FileStatus to a list of CopyListingFileStatus. The resulted list contains either one CopyListingFileStatus per chunk of file-blocks (if file-size exceeds blockSize * blocksPerChunk, and there are more blocks in the file than blocksperChunk), or a single CopyListingFileStatus for the entire file (if file-size is too small to split). If preserving ACLs, populates the CopyListingFileStatus with the ACLs. If preserving XAttrs, populates the CopyListingFileStatus with the XAttrs.

Parameters:

fileSystem - FileSystem containing the file

listingEntry - listingEntry

preserveAcls - boolean true if preserving ACLs

preserveXAttrs - boolean true if preserving XAttrs

preserveRawXAttrs - boolean true if preserving raw.* XAttrs

blocksPerChunk - size of chunks when copying chunks in parallel

Throws:

java.io.IOException - if there is an I/O error

getOriginalFileStatus

public static FileListingEntry getOriginalFileStatus(org.apache.hadoop.fs.FileStatus sourceStatus,
                                                     org.apache.hadoop.conf.Configuration conf,
                                                     boolean keepLinks,
                                                     java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator)

isParentLoop

public static boolean isParentLoop(org.apache.hadoop.fs.FileStatus sourceStatus,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws java.io.IOException

Throws:

java.io.IOException

isLoop

public static boolean isLoop(org.apache.hadoop.fs.FileStatus src,
                             java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator)
                      throws java.io.IOException

Throws:

java.io.IOException

discloseRelativeSymlink

public static org.apache.hadoop.fs.Path discloseRelativeSymlink(org.apache.hadoop.fs.FileStatus stat)
                                                         throws java.io.IOException

Throws:

java.io.IOException

toCopyListingFileStatusHelper

public static CopyListingFileStatus toCopyListingFileStatusHelper(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                  org.apache.hadoop.fs.FileStatus fileStatus,
                                                                  boolean preserveAcls,
                                                                  boolean preserveXAttrs,
                                                                  boolean preserveRawXAttrs,
                                                                  long chunkOffset,
                                                                  long chunkLength)
                                                           throws java.io.IOException

Converts a FileStatus to a CopyListingFileStatus. If preserving ACLs, populates the CopyListingFileStatus with the ACLs. If preserving XAttrs, populates the CopyListingFileStatus with the XAttrs.

Parameters:

fileSystem - FileSystem containing the file

fileStatus - FileStatus of file

preserveAcls - boolean true if preserving ACLs

preserveXAttrs - boolean true if preserving XAttrs

preserveRawXAttrs - boolean true if preserving raw.* XAttrs

chunkOffset - chunk offset in bytes

chunkLength - chunk length in bytes

Returns:

CopyListingFileStatus

Throws:

java.io.IOException - if there is an I/O error

sortListing

public static org.apache.hadoop.fs.Path sortListing(org.apache.hadoop.conf.Configuration conf,
                                                    org.apache.hadoop.fs.Path sourceListing)
                                             throws java.io.IOException

Sort sequence file containing FileStatus and Text as key and value respectively.

Parameters:

conf - - Configuration

sourceListing - - Source listing file

Returns:

Path of the sorted file. Is source file with _sorted appended to the name

Throws:

java.io.IOException - - Any exception during sort.

sortListing

public static void sortListing(org.apache.hadoop.conf.Configuration conf,
                               org.apache.hadoop.fs.Path sourceListing,
                               org.apache.hadoop.fs.Path output)
                        throws java.io.IOException

Sort sequence file containing FileStatus and Text as key and value respectively, saving the result to the output path, which will be deleted first.

Parameters:

conf - - Configuration

sourceListing - - Source listing file

output - output path

Throws:

java.io.IOException - - Any exception during sort.

checkFileSystemAclSupport

public static void checkFileSystemAclSupport(org.apache.hadoop.fs.FileSystem fs)
                                      throws CopyListing.AclsNotSupportedException

Determines if a file system supports ACLs by running a canary getAclStatus request on the file system root. This method is used before distcp job submission to fail fast if the user requested preserving ACLs, but the file system cannot support ACLs.

Parameters:

fs - FileSystem to check

Throws:

CopyListing.AclsNotSupportedException - if fs does not support ACLs

checkFileSystemXAttrSupport

public static void checkFileSystemXAttrSupport(org.apache.hadoop.fs.FileSystem fs)
                                        throws CopyListing.XAttrsNotSupportedException

Determines if a file system supports XAttrs by running a getXAttrs request on the file system root. This method is used before distcp job submission to fail fast if the user requested preserving XAttrs, but the file system cannot support XAttrs.

Parameters:

fs - FileSystem to check

Throws:

CopyListing.XAttrsNotSupportedException - if fs does not support XAttrs

getFormatter

public static java.text.DecimalFormat getFormatter()

getStringDescriptionFor

public static java.lang.String getStringDescriptionFor(long nBytes)

checksumsAreEqual

public static CopyMapper.ChecksumComparison checksumsAreEqual(org.apache.hadoop.fs.FileSystem sourceFS,
                                                              org.apache.hadoop.fs.Path source,
                                                              org.apache.hadoop.fs.FileChecksum sourceChecksum,
                                                              org.apache.hadoop.fs.FileSystem targetFS,
                                                              org.apache.hadoop.fs.Path target,
                                                              long sourceLen)
                                                       throws java.io.IOException

Utility to compare checksums for the paths specified. If checksums can't be retrieved, it doesn't fail the test Only time the comparison would fail is when checksums are available and they don't match

Parameters:

sourceFS - FileSystem for the source path.

source - The source path.

sourceChecksum - The checksum of the source file. If it is null we still need to retrieve it through sourceFS.

targetFS - FileSystem for the target path.

target - The target path.

Returns:

If either checksum couldn't be retrieved, the function returns false. If checksums are retrieved, the function returns true if they match, and false otherwise.

Throws:

java.io.IOException - if there's an exception while retrieving checksums.

compareFileLengthsAndChecksums

public static void compareFileLengthsAndChecksums(long srcLen,
                                                  org.apache.hadoop.fs.FileSystem sourceFS,
                                                  org.apache.hadoop.fs.Path source,
                                                  org.apache.hadoop.fs.FileChecksum sourceChecksum,
                                                  org.apache.hadoop.fs.FileSystem targetFS,
                                                  org.apache.hadoop.fs.Path target,
                                                  boolean skipCrc,
                                                  long targetLen)
                                           throws java.io.IOException

Utility to compare file lengths and checksums for source and target.

Parameters:

sourceFS - FileSystem for the source path.

source - The source path.

sourceChecksum - The checksum of the source file. If it is null we still need to retrieve it through sourceFS.

targetFS - FileSystem for the target path.

target - The target path.

skipCrc - The flag to indicate whether to skip checksums.

Throws:

java.io.IOException - if there's a mismatch in file lengths or checksums.

getSplitChunkPath

public static org.apache.hadoop.fs.Path getSplitChunkPath(org.apache.hadoop.fs.Path targetFile,
                                                          CopyListingFileStatus srcFileStatus)

getTmpFile

public static org.apache.hadoop.fs.Path getTmpFile(org.apache.hadoop.fs.Path target,
                                                   org.apache.hadoop.mapreduce.Mapper.Context context,
                                                   boolean splitSource)