CopyCommandWithMultiThread (Apache Hadoop Common 3.4.1.200-dep-1000 API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.fs.shell.Command
  - - org.apache.hadoop.fs.shell.FsCommand
    - - org.apache.hadoop.fs.shell.CopyCommandWithMultiThread

All Implemented Interfaces:

Configurable
```
public abstract class CopyCommandWithMultiThread
extends FsCommand
```
Abstract command to enable sub copy commands run with multi-thread.

Field Summary

Fields
Modifier and Type	Field	Description
`static int`	`DEFAULT_QUEUE_SIZE`
`protected PathData`	`dst`
`protected static org.slf4j.Logger`	`LOG`
`static java.util.Map<java.lang.String,java.util.Set<java.lang.String>>`	`loopLocator`

Fields inherited from class org.apache.hadoop.fs.shell.Command
args, COMMAND_DESCRIPTION_FIELD, COMMAND_NAME_FIELD, COMMAND_USAGE_FIELD, err, exceptions, exitCode, name, numErrors, out, recursive

Constructor Summary

Constructors
Constructor Description

CopyCommandWithMultiThread()

Constructors
Constructor	Description
`CopyCommandWithMultiThread()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`protected void`	`copyFileToTarget(PathData src, PathData target)`	Copies the source file to the target.
`protected void`	`copyStreamToTarget(java.io.InputStream in, PathData target)`	If direct write is disabled ,copies the stream contents to a temporary file "target._COPYING_".
`protected java.util.concurrent.ThreadPoolExecutor`	`getExecutor()`
`protected void`	`getLocalDestination(java.util.LinkedList<java.lang.String> args)`	The last arg is expected to be a local path, if only one argument is given then the destination will be the current directory
`protected void`	`getRemoteDestination(java.util.LinkedList<java.lang.String> args)`	The last arg is expected to be a remote path, if only one argument is given then the destination will be the remote user's directory
`protected PathData`	`getTargetPath(PathData src)`
`protected int`	`getThreadCount()`
`protected int`	`getThreadPoolQueueSize()`
`protected boolean`	`isMultiThreadNecessary(java.util.LinkedList<PathData> args)`
`protected void`	`preserve(org.apache.hadoop.fs.shell.CommandWithDestination.FileAttribute fileAttribute)`	Add file attributes that need to be preserved.
`protected void`	`preserveAttributes(PathData src, PathData target, boolean preserveRawXAttrs)`	Preserve the attributes of the source to the target.
`protected void`	`processArguments(java.util.LinkedList<PathData> args)`	Processes the command's list of expanded arguments.
`protected void`	`processPath(PathData src)`	Hook for commands to implement an operation to be applied on each path for the command.
`protected void`	`processPath(PathData src, PathData dst)`	Called with a source and target destination pair
`protected void`	`processPathArgument(PathData src)`	This is the last chance to modify an argument before going into the (possibly) recursive `Command.processPaths(PathData, PathData...)` -> `Command.processPath(PathData)` loop.
`protected void`	`recursePath(PathData src)`	Gets the directory listing for a path and invokes `Command.processPaths(PathData, PathData...)`
`protected void`	`setDirectWrite(boolean flag)`
`protected void`	`setKeepLinks(boolean flag)`
`protected void`	`setLazyPersist(boolean flag)`
`protected void`	`setOverwrite(boolean flag)`	This method is used to enable the force(-f) option while copying the files.
`protected void`	`setPreserve(boolean preserve)`	If true, the last modified time, last access time, owner, group and permission information of the source file will be preserved as far as target `FileSystem` implementation allows.
`protected void`	`setThreadCount(java.lang.String optValue)`	set thread count by option value, if the value less than 1, use 1 instead.
`protected void`	`setThreadPoolQueueSize(java.lang.String optValue)`	set thread pool queue size by option value, if the value less than 1, use DEFAULT_QUEUE_SIZE instead.
`protected void`	`setVerifyChecksum(boolean flag)`
`protected void`	`setWriteChecksum(boolean flag)`

Methods inherited from class org.apache.hadoop.fs.shell.FsCommand
getCommandName, processRawArguments, registerCommands, run, runAll

Methods inherited from class org.apache.hadoop.fs.shell.Command
displayError, displayError, displayWarning, exitCodeForError, expandArgument, expandArguments, getCommandFactory, getDepth, getDescription, getListingGroupSize, getName, getReplacementCommand, getUsage, isDeprecated, isPathRecursable, isRecursive, isSorted, postProcessPath, processArgument, processNonexistentPath, processOptions, processPaths, processPaths, run, run, setCommandFactory, setName, setRecursive

Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEFAULT_QUEUE_SIZE
```
public static final int DEFAULT_QUEUE_SIZE
```
    See Also:
    
    Constant Field Values
  - LOG
```
protected static final org.slf4j.Logger LOG
```
  - dst
```
protected PathData dst
```
  - loopLocator
```
public static java.util.Map<java.lang.String,java.util.Set<java.lang.String>> loopLocator
```
- Constructor Detail
  - CopyCommandWithMultiThread
```
public CopyCommandWithMultiThread()
```
- Method Detail
  - setThreadCount
```
protected void setThreadCount(java.lang.String optValue)
```
    set thread count by option value, if the value less than 1, use 1 instead.
    
    Parameters:
    
    optValue - option value
  - setThreadPoolQueueSize
```
protected void setThreadPoolQueueSize(java.lang.String optValue)
```
    set thread pool queue size by option value, if the value less than 1, use DEFAULT_QUEUE_SIZE instead.
    
    Parameters:
    
    optValue - option value
  - getThreadCount
```
@VisibleForTesting
protected int getThreadCount()
```
  - getThreadPoolQueueSize
```
@VisibleForTesting
protected int getThreadPoolQueueSize()
```
  - getExecutor
```
@VisibleForTesting
protected java.util.concurrent.ThreadPoolExecutor getExecutor()
```
  - processArguments
```
protected void processArguments(java.util.LinkedList<PathData> args)
                         throws java.io.IOException
```
    Description copied from class: Command
    
    Processes the command's list of expanded arguments. Command.processArgument(PathData) will be invoked with each item in the list. The loop catches IOExceptions, increments the error count, and displays the exception.
    
    Parameters:
    
    args - a list of PathData to process
    
    Throws:
    
    java.io.IOException - if anything goes wrong...
  - isMultiThreadNecessary
```
@VisibleForTesting
protected boolean isMultiThreadNecessary(java.util.LinkedList<PathData> args)
                                  throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - copyFileToTarget
```
protected void copyFileToTarget(PathData src,
                                PathData target)
                         throws java.io.IOException
```
    Copies the source file to the target.
    
    Parameters:
    
    src - item to copy
    
    target - where to copy the item
    
    Throws:
    
    java.io.IOException - if copy fails
  - setOverwrite
```
protected void setOverwrite(boolean flag)
```
    This method is used to enable the force(-f) option while copying the files.
    
    Parameters:
    
    flag - true/false
  - setLazyPersist
```
protected void setLazyPersist(boolean flag)
```
  - setVerifyChecksum
```
protected void setVerifyChecksum(boolean flag)
```
  - setWriteChecksum
```
protected void setWriteChecksum(boolean flag)
```
  - setDirectWrite
```
protected void setDirectWrite(boolean flag)
```
  - setKeepLinks
```
protected void setKeepLinks(boolean flag)
```
  - setPreserve
```
protected void setPreserve(boolean preserve)
```
    If true, the last modified time, last access time, owner, group and permission information of the source file will be preserved as far as target FileSystem implementation allows.
    
    Parameters:
    
    preserve - preserve.
  - preserve
```
protected void preserve(org.apache.hadoop.fs.shell.CommandWithDestination.FileAttribute fileAttribute)
```
    Add file attributes that need to be preserved. This method may be called multiple times to add attributes.
    
    Parameters:
    
    fileAttribute - - Attribute to add, one at a time
  - getLocalDestination
```
protected void getLocalDestination(java.util.LinkedList<java.lang.String> args)
                            throws java.io.IOException
```
    The last arg is expected to be a local path, if only one argument is given then the destination will be the current directory
    
    Parameters:
    
    args - is the list of arguments
    
    Throws:
    
    java.io.IOException - raised on errors performing I/O.
  - getRemoteDestination
```
protected void getRemoteDestination(java.util.LinkedList<java.lang.String> args)
                             throws java.io.IOException
```
    The last arg is expected to be a remote path, if only one argument is given then the destination will be the remote user's directory
    
    Parameters:
    
    args - is the list of arguments
    
    Throws:
    
    PathIOException - if path doesn't exist or matches too many times
    
    java.io.IOException
  - processPathArgument
```
protected void processPathArgument(PathData src)
                            throws java.io.IOException
```
    Description copied from class: Command
    
    This is the last chance to modify an argument before going into the (possibly) recursive Command.processPaths(PathData, PathData...) -> Command.processPath(PathData) loop. Ex. ls and du use this to expand out directories.
    
    Overrides:
    
    processPathArgument in class Command
    
    Parameters:
    
    src - a PathData representing a path which exists
    
    Throws:
    
    java.io.IOException - if anything goes wrong...
  - processPath
```
protected void processPath(PathData src)
                    throws java.io.IOException
```
    Description copied from class: Command
    
    Hook for commands to implement an operation to be applied on each path for the command. Note implementation of this method is optional if earlier methods in the chain handle the operation.
    
    Overrides:
    
    processPath in class Command
    
    Parameters:
    
    src - a PathData object
    
    Throws:
    
    java.io.IOException - if anything else goes wrong in the user-implementation
  - processPath
```
protected void processPath(PathData src,
                           PathData dst)
                    throws java.io.IOException
```
    Called with a source and target destination pair
    
    Parameters:
    
    src - for the operation
    
    dst - for the operation
    
    Throws:
    
    java.io.IOException - if anything goes wrong
  - recursePath
```
protected void recursePath(PathData src)
                    throws java.io.IOException
```
    Description copied from class: Command
    
    Gets the directory listing for a path and invokes Command.processPaths(PathData, PathData...)
    
    Overrides:
    
    recursePath in class Command
    
    Parameters:
    
    src - PathData for directory to recurse into
    
    Throws:
    
    java.io.IOException - if anything goes wrong...
  - getTargetPath
```
protected PathData getTargetPath(PathData src)
                          throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - copyStreamToTarget
```
protected void copyStreamToTarget(java.io.InputStream in,
                                  PathData target)
                           throws java.io.IOException
```
    If direct write is disabled ,copies the stream contents to a temporary file "target._COPYING_". If the copy is successful, the temporary file will be renamed to the real path, else the temporary file will be deleted. if direct write is enabled , then creation temporary file is skipped.
    
    Parameters:
    
    in - the input stream for the copy
    
    target - where to store the contents of the stream
    
    Throws:
    
    java.io.IOException - if copy fails
  - preserveAttributes
```
protected void preserveAttributes(PathData src,
                                  PathData target,
                                  boolean preserveRawXAttrs)
                           throws java.io.IOException
```
    Preserve the attributes of the source to the target. The method calls shouldPreserve(FileAttribute) to check what attribute to preserve.
    
    Parameters:
    
    src - source to preserve
    
    target - where to preserve attributes
    
    preserveRawXAttrs - true if raw.* xattrs should be preserved
    
    Throws:
    
    java.io.IOException - if fails to preserve attributes

Class CopyCommandWithMultiThread

Field Summary

Fields inherited from class org.apache.hadoop.fs.shell.Command

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.fs.shell.FsCommand

Methods inherited from class org.apache.hadoop.fs.shell.Command

Methods inherited from class org.apache.hadoop.conf.Configured

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_QUEUE_SIZE

LOG

dst

loopLocator

Constructor Detail

CopyCommandWithMultiThread

Method Detail

setThreadCount

setThreadPoolQueueSize

getThreadCount

getThreadPoolQueueSize

getExecutor

processArguments

isMultiThreadNecessary

copyFileToTarget

setOverwrite

setLazyPersist

setVerifyChecksum

setWriteChecksum

setDirectWrite

setKeepLinks

setPreserve

preserve

getLocalDestination

getRemoteDestination

processPathArgument

processPath

processPath

recursePath

getTargetPath

copyStreamToTarget

preserveAttributes