Package org.apache.hadoop.streaming

Class StreamJob

java.lang.Object

org.apache.hadoop.streaming.StreamJob

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class StreamJob
extends java.lang.Object
implements org.apache.hadoop.util.Tool

All the client-side work happens here. (Jar packaging, MapRed job submission and monitoring)

Field Summary

Fields

Modifier and Type	Field	Description
protected java.lang.String	additionalConfSpec_
protected java.lang.String	addTaskEnvironment_
protected java.net.URI[]	archiveURIs
protected java.lang.String[]	argv_
protected boolean	background_
protected java.lang.String	cacheArchives
protected java.lang.String	cacheFiles
protected java.lang.String	comCmd_
protected org.apache.hadoop.conf.Configuration	config_
protected int	debug_
protected boolean	detailedUsage_
protected Environment	env_
protected java.net.URI[]	fileURIs
protected boolean	hasSimpleInputSpecs_
protected java.lang.String	inputFormatSpec_
protected java.util.ArrayList<java.lang.String>	inputSpecs_
protected java.lang.String	inReaderSpec_
protected java.lang.String	ioSpec_
protected java.lang.String	jar_
protected org.apache.hadoop.mapred.JobClient	jc_
protected org.apache.hadoop.mapred.JobConf	jobConf_
protected org.apache.hadoop.mapred.JobID	jobId_
protected boolean	lazyOutput_
protected static java.lang.String	LINK_URI
protected boolean	localHadoop_
protected static org.slf4j.Logger	LOG
protected java.lang.String	mapCmd_
protected java.lang.String	mapDebugSpec_
protected long	minRecWrittenToEnableSkip_
protected java.lang.String	numReduceTasksSpec_
protected java.lang.String	output_
protected java.lang.String	outputFormatSpec_
protected boolean	outputSingleNode_
protected java.util.ArrayList<java.lang.String>	packageFiles_
protected java.lang.String	partitionerSpec_
protected boolean	printUsage
protected java.lang.String	redCmd_
protected java.lang.String	reduceDebugSpec_
protected org.apache.hadoop.mapred.RunningJob	running_
protected java.util.TreeSet<java.lang.String>	seenPrimary_
protected java.util.ArrayList<java.lang.String>	shippedCanonFiles_
protected boolean	verbose_

Constructor Summary

Constructors

Constructor	Description
StreamJob()
StreamJob(java.lang.String[] argv, boolean mayExit)	Deprecated. use StreamJob() with ToolRunner or set the Configuration using setConf(Configuration) and run with run(String[]).

Method Summary

Modifier and Type	Method	Description
static org.apache.hadoop.mapred.JobConf	createJob(java.lang.String[] argv)	This method creates a streaming job from the given argument list.
void	exitUsage(boolean detailed)
void	fail(java.lang.String message)
protected java.lang.String	getClusterNick()	Deprecated.
org.apache.hadoop.conf.Configuration	getConf()
protected java.lang.String	getHadoopClientHome()
protected java.lang.String	getJobTrackerHostPort()
protected void	getURIs(java.lang.String lcacheArchives, java.lang.String lcacheFiles)	get the uris of all the files/caches
int	go()	Deprecated. use run(String[]) instead.
protected void	init()
protected boolean	isLocalHadoop()
protected void	listJobConfProperties()	Prints out the jobconf properties on stdout when verbose is specified.
protected void	msg(java.lang.String msg)
protected java.lang.String	packageJobJar()
int	run(java.lang.String[] args)
void	setConf(org.apache.hadoop.conf.Configuration conf)
protected void	setJobConf()
int	submitAndMonitorJob()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

argv_

protected java.lang.String[] argv_

background_

protected boolean background_

verbose_

protected boolean verbose_

detailedUsage_

protected boolean detailedUsage_

printUsage

protected boolean printUsage

debug_

protected int debug_

env_

protected Environment env_

jar_

protected java.lang.String jar_

localHadoop_

protected boolean localHadoop_

config_

protected org.apache.hadoop.conf.Configuration config_

jobConf_

protected org.apache.hadoop.mapred.JobConf jobConf_

jc_

protected org.apache.hadoop.mapred.JobClient jc_

inputSpecs_

protected java.util.ArrayList<java.lang.String> inputSpecs_

seenPrimary_

protected java.util.TreeSet<java.lang.String> seenPrimary_

hasSimpleInputSpecs_

protected boolean hasSimpleInputSpecs_

packageFiles_

protected java.util.ArrayList<java.lang.String> packageFiles_

shippedCanonFiles_

protected java.util.ArrayList<java.lang.String> shippedCanonFiles_

output_

protected java.lang.String output_

mapCmd_

protected java.lang.String mapCmd_

comCmd_

protected java.lang.String comCmd_

redCmd_

protected java.lang.String redCmd_

cacheFiles

protected java.lang.String cacheFiles

cacheArchives

protected java.lang.String cacheArchives

fileURIs

protected java.net.URI[] fileURIs

archiveURIs

protected java.net.URI[] archiveURIs

inReaderSpec_

protected java.lang.String inReaderSpec_

inputFormatSpec_

protected java.lang.String inputFormatSpec_

outputFormatSpec_

protected java.lang.String outputFormatSpec_

partitionerSpec_

protected java.lang.String partitionerSpec_

numReduceTasksSpec_

protected java.lang.String numReduceTasksSpec_

additionalConfSpec_

protected java.lang.String additionalConfSpec_

mapDebugSpec_

protected java.lang.String mapDebugSpec_

reduceDebugSpec_

protected java.lang.String reduceDebugSpec_

ioSpec_

protected java.lang.String ioSpec_

lazyOutput_

protected boolean lazyOutput_

addTaskEnvironment_

protected java.lang.String addTaskEnvironment_

outputSingleNode_

protected boolean outputSingleNode_

minRecWrittenToEnableSkip_

protected long minRecWrittenToEnableSkip_

running_

protected org.apache.hadoop.mapred.RunningJob running_

jobId_

protected org.apache.hadoop.mapred.JobID jobId_

LINK_URI

protected static final java.lang.String LINK_URI

See Also:

Constant Field Values

Constructor Detail

StreamJob

@Deprecated
public StreamJob(java.lang.String[] argv,
                 boolean mayExit)

Deprecated.

use StreamJob() with ToolRunner or set the Configuration using setConf(Configuration) and run with run(String[]).

StreamJob

public StreamJob()

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

run

public int run(java.lang.String[] args)
        throws java.lang.Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

java.lang.Exception

createJob

public static org.apache.hadoop.mapred.JobConf createJob(java.lang.String[] argv)
                                                  throws java.io.IOException

This method creates a streaming job from the given argument list. The created object can be used and/or submitted to a jobtracker for execution by a job agent such as JobControl

Parameters:

argv - the list args for creating a streaming job

Returns:

the created JobConf object

Throws:

java.io.IOException

go

@Deprecated
public int go()
       throws java.io.IOException

Deprecated.

use run(String[]) instead.

This is the method that actually initializes the job conf and submits the job to the jobtracker

Throws:

java.io.IOException

init

protected void init()

msg

protected void msg(java.lang.String msg)

exitUsage

public void exitUsage(boolean detailed)

fail

public void fail(java.lang.String message)

getHadoopClientHome

protected java.lang.String getHadoopClientHome()

isLocalHadoop

protected boolean isLocalHadoop()

getClusterNick

@Deprecated
protected java.lang.String getClusterNick()

Deprecated.

packageJobJar

protected java.lang.String packageJobJar()
                                  throws java.io.IOException

Returns:

path to the created Jar file or null if no files are necessary.

Throws:

java.io.IOException

getURIs

protected void getURIs(java.lang.String lcacheArchives,
                       java.lang.String lcacheFiles)

get the uris of all the files/caches

setJobConf

protected void setJobConf()
                   throws java.io.IOException

Throws:

java.io.IOException

listJobConfProperties

protected void listJobConfProperties()

Prints out the jobconf properties on stdout when verbose is specified.

getJobTrackerHostPort

protected java.lang.String getJobTrackerHostPort()

submitAndMonitorJob

public int submitAndMonitorJob()
                        throws java.io.IOException

Throws:

java.io.IOException