Package org.apache.hadoop.tools

Class DistCp

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.DistCp

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@Public
@Evolving
public class DistCp
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

DistCp is the main driver-class for DistCpV2. For command-line use, DistCp::main() orchestrates the parsing of command-line parameters and the launch of the DistCp job. For programmatic use, a DistCp object can be constructed by specifying options (in a DistCpOptions object), and DistCp::execute() may be used to launch the copy-job. DistCp may alternatively be sub-classed to fine-tune behaviour.

Constructor Summary

Constructors

Constructor	Description
DistCp(org.apache.hadoop.conf.Configuration configuration, DistCpOptions inputOptions)	Public Constructor.

Method Summary

Modifier and Type	Method	Description
protected void	cleanup()	Clean the staging folder created by distcp.
org.apache.hadoop.mapreduce.Job	createAndSubmitJob()	Create and submit the mapreduce job.
protected org.apache.hadoop.fs.Path	createInputFileListing(org.apache.hadoop.mapreduce.Job job)	Create input listing by invoking an appropriate copy listing implementation.
org.apache.hadoop.mapreduce.Job	execute()	Implements the core-execution.
protected DistCpContext	getContext()	Returns the context.
protected org.apache.hadoop.fs.Path	getFileListingPath()	Get default name of the copy listing file.
static void	main(java.lang.String[] argv)	Main function of the DistCp program.
int	run(java.lang.String[] argv)	Implementation of Tool::run().
void	waitForJobCompletion(org.apache.hadoop.mapreduce.Job job)	Wait for the given job to complete.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

DistCp

public DistCp(org.apache.hadoop.conf.Configuration configuration,
              DistCpOptions inputOptions)
       throws java.lang.Exception

Public Constructor. Creates DistCp object with specified input-parameters. (E.g. source-paths, target-location, etc.)

Parameters:

configuration - configuration against which the Copy-mapper must run

inputOptions - Immutable options

Throws:

java.lang.Exception

Method Detail

run

public int run(java.lang.String[] argv)

Implementation of Tool::run(). Orchestrates the copy of source file(s) to target location, by: 1. Creating a list of files to be copied to target. 2. Launching a Map-only job to copy the files. (Delegates to execute().) The MR job is not closed as part of run if its a blocking call to run

Specified by:

run in interface org.apache.hadoop.util.Tool

Parameters:

argv - List of arguments passed to DistCp, from the ToolRunner.

Returns:

On success, it returns 0. Else, -1.

execute

public org.apache.hadoop.mapreduce.Job execute()
                                        throws java.lang.Exception

Implements the core-execution. Creates the file-list for copy, and launches the Hadoop-job, to do the copy.

Returns:

Job handle

Throws:

java.lang.Exception

createAndSubmitJob

public org.apache.hadoop.mapreduce.Job createAndSubmitJob()
                                                   throws java.lang.Exception

Create and submit the mapreduce job.

Returns:

The mapreduce job object that has been submitted

Throws:

java.lang.Exception

waitForJobCompletion

public void waitForJobCompletion(org.apache.hadoop.mapreduce.Job job)
                          throws java.lang.Exception

Wait for the given job to complete.

Parameters:

job - the given mapreduce job that has already been submitted

Throws:

java.lang.Exception

createInputFileListing

protected org.apache.hadoop.fs.Path createInputFileListing(org.apache.hadoop.mapreduce.Job job)
                                                    throws java.io.IOException

Create input listing by invoking an appropriate copy listing implementation. Also add delegation tokens for each path to job's credential store

Parameters:

job - - Handle to job

Returns:

Returns the path where the copy listing is created

Throws:

java.io.IOException - - If any

getFileListingPath

protected org.apache.hadoop.fs.Path getFileListingPath()
                                                throws java.io.IOException

Get default name of the copy listing file. Use the meta folder to create the copy listing file

Returns:

- Path where the copy listing file has to be saved

Throws:

java.io.IOException - - Exception if any

getContext

protected DistCpContext getContext()

Returns the context.

Returns:

the context

main

public static void main(java.lang.String[] argv)

Main function of the DistCp program. Parses the input arguments (via OptionsParser), and invokes the DistCp::run() method, via the ToolRunner.

Parameters:

argv - Command-line arguments sent to DistCp.

cleanup

protected void cleanup()

Clean the staging folder created by distcp.