Package org.apache.hadoop.mapred.gridmix

Class Gridmix

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.mapred.gridmix.Gridmix

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class Gridmix
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Driver class for the Gridmix3 benchmark. Gridmix accepts a timestamped stream (trace) of job/task descriptions. For each job in the trace, the client will submit a corresponding, synthetic job to the target cluster at the rate in the original trace. The intent is to provide a benchmark that can be configured and extended to closely match the measured resource profile of actual, production loads.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS	The configuration key which determines the duration for which the job-monitor sleeps while polling for job status.
static int	GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS_DEFAULT	Default value for GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS.
static java.lang.String	GRIDMIX_JOBMONITOR_THREADS	The configuration key which determines the total number of job-status monitoring threads.
static int	GRIDMIX_JOBMONITOR_THREADS_DEFAULT	Default value for GRIDMIX_JOBMONITOR_THREADS.
static java.lang.String	GRIDMIX_OUT_DIR	Output (scratch) directory for submitted jobs.
static java.lang.String	GRIDMIX_QUE_DEP	The depth of the queue of job descriptions.
static java.lang.String	GRIDMIX_SUB_MUL	Multiplier to accelerate or decelerate job submission.
static java.lang.String	GRIDMIX_SUB_THR	Number of submitting threads at the client and upper bound for in-memory split data.
static java.lang.String	GRIDMIX_USR_RSV	Class used to resolve users in the trace to the list of target users on the cluster.
static org.slf4j.Logger	LOG
static java.lang.String	ORIGINAL_JOB_ID	Configuration property set in simulated job's configuration whose value is set to the corresponding original job's id.
static java.lang.String	ORIGINAL_JOB_NAME	Configuration property set in simulated job's configuration whose value is set to the corresponding original job's name.

Constructor Summary

Constructors

Constructor	Description
Gridmix()

Method Summary

Modifier and Type	Method	Description
protected org.apache.hadoop.mapred.gridmix.JobFactory	createJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter, java.lang.String traceIn, org.apache.hadoop.fs.Path scratchDir, org.apache.hadoop.conf.Configuration conf, java.util.concurrent.CountDownLatch startFlag, UserResolver resolver)
protected org.apache.hadoop.mapred.gridmix.JobMonitor	createJobMonitor(Statistics stats, org.apache.hadoop.conf.Configuration conf)
protected org.apache.hadoop.tools.rumen.JobStoryProducer	createJobStoryProducer(java.lang.String traceIn, org.apache.hadoop.conf.Configuration conf)	Create an appropriate JobStoryProducer object for the given trace.
protected org.apache.hadoop.mapred.gridmix.JobSubmitter	createJobSubmitter(org.apache.hadoop.mapred.gridmix.JobMonitor monitor, int threads, int queueDepth, org.apache.hadoop.mapred.gridmix.FilePool pool, UserResolver resolver, Statistics statistics)
UserResolver	getCurrentUserResolver()
protected static org.apache.hadoop.mapred.gridmix.GridmixJobSubmissionPolicy	getJobSubmissionPolicy(org.apache.hadoop.conf.Configuration conf)
protected org.apache.hadoop.mapred.gridmix.Summarizer	getSummarizer()
static void	main(java.lang.String[] argv)
protected void	printUsage(java.io.PrintStream out)
int	run(java.lang.String[] argv)
protected void	writeDistCacheData(org.apache.hadoop.conf.Configuration conf)	Write random bytes in the distributed cache files that will be used by all simulated jobs of current gridmix run, if files are to be generated.
protected int	writeInputData(long genbytes, org.apache.hadoop.fs.Path inputDir)	Write random bytes at the path <inputDir> if needed.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

public static final org.slf4j.Logger LOG

GRIDMIX_OUT_DIR

public static final java.lang.String GRIDMIX_OUT_DIR

Output (scratch) directory for submitted jobs. Relative paths are resolved against the path provided as input and absolute paths remain independent of it. The default is "gridmix".

See Also:

Constant Field Values

GRIDMIX_SUB_THR

public static final java.lang.String GRIDMIX_SUB_THR

Number of submitting threads at the client and upper bound for in-memory split data. Submitting threads precompute InputSplits for submitted jobs. This limits the number of splits held in memory waiting for submission and also permits parallel computation of split data.

See Also:

Constant Field Values

GRIDMIX_QUE_DEP

public static final java.lang.String GRIDMIX_QUE_DEP

The depth of the queue of job descriptions. Before splits are computed, a queue of pending descriptions is stored in memoory. This parameter limits the depth of that queue.

See Also:

Constant Field Values

GRIDMIX_SUB_MUL

public static final java.lang.String GRIDMIX_SUB_MUL

Multiplier to accelerate or decelerate job submission. As a crude means of sizing a job trace to a cluster, the time separating two jobs is multiplied by this factor.

See Also:

Constant Field Values

GRIDMIX_USR_RSV

public static final java.lang.String GRIDMIX_USR_RSV

Class used to resolve users in the trace to the list of target users on the cluster.

See Also:

Constant Field Values

GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS

public static final java.lang.String GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS

The configuration key which determines the duration for which the job-monitor sleeps while polling for job status. This value should be specified in milliseconds.

See Also:

Constant Field Values

GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS_DEFAULT

public static final int GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS_DEFAULT

Default value for GRIDMIX_JOBMONITOR_SLEEPTIME_MILLIS.

See Also:

Constant Field Values

GRIDMIX_JOBMONITOR_THREADS

public static final java.lang.String GRIDMIX_JOBMONITOR_THREADS

The configuration key which determines the total number of job-status monitoring threads.

See Also:

Constant Field Values

GRIDMIX_JOBMONITOR_THREADS_DEFAULT

public static final int GRIDMIX_JOBMONITOR_THREADS_DEFAULT

Default value for GRIDMIX_JOBMONITOR_THREADS.

See Also:

Constant Field Values

ORIGINAL_JOB_NAME

public static final java.lang.String ORIGINAL_JOB_NAME

Configuration property set in simulated job's configuration whose value is set to the corresponding original job's name. This is not configurable by gridmix user.

See Also:

Constant Field Values

ORIGINAL_JOB_ID

public static final java.lang.String ORIGINAL_JOB_ID

Configuration property set in simulated job's configuration whose value is set to the corresponding original job's id. This is not configurable by gridmix user.

See Also:

Constant Field Values

Constructor Detail

Gridmix

public Gridmix()

Method Detail

writeInputData

protected int writeInputData(long genbytes,
                             org.apache.hadoop.fs.Path inputDir)
                      throws java.io.IOException,
                             java.lang.InterruptedException

Write random bytes at the path <inputDir> if needed.

Returns:

exit status

Throws:

java.io.IOException

java.lang.InterruptedException

See Also:

GenerateData

writeDistCacheData

protected void writeDistCacheData(org.apache.hadoop.conf.Configuration conf)
                           throws java.io.IOException,
                                  java.lang.InterruptedException

Write random bytes in the distributed cache files that will be used by all simulated jobs of current gridmix run, if files are to be generated. Do this as part of the MapReduce job GenerateDistCacheData.JOB_NAME

Throws:

java.io.IOException

java.lang.InterruptedException

See Also:

GenerateDistCacheData

createJobStoryProducer

protected org.apache.hadoop.tools.rumen.JobStoryProducer createJobStoryProducer(java.lang.String traceIn,
                                                                                org.apache.hadoop.conf.Configuration conf)
                                                                         throws java.io.IOException

Create an appropriate JobStoryProducer object for the given trace.

Parameters:

traceIn - the path to the trace file. The special path "-" denotes the standard input stream.

conf - the configuration to be used.

Throws:

java.io.IOException - if there was an error.

getJobSubmissionPolicy

protected static org.apache.hadoop.mapred.gridmix.GridmixJobSubmissionPolicy getJobSubmissionPolicy(org.apache.hadoop.conf.Configuration conf)

createJobMonitor

protected org.apache.hadoop.mapred.gridmix.JobMonitor createJobMonitor(Statistics stats,
                                                                       org.apache.hadoop.conf.Configuration conf)
                                                                throws java.io.IOException

Throws:

java.io.IOException

createJobSubmitter

protected org.apache.hadoop.mapred.gridmix.JobSubmitter createJobSubmitter(org.apache.hadoop.mapred.gridmix.JobMonitor monitor,
                                                                           int threads,
                                                                           int queueDepth,
                                                                           org.apache.hadoop.mapred.gridmix.FilePool pool,
                                                                           UserResolver resolver,
                                                                           Statistics statistics)
                                                                    throws java.io.IOException

Throws:

java.io.IOException

createJobFactory

protected org.apache.hadoop.mapred.gridmix.JobFactory createJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter,
                                                                       java.lang.String traceIn,
                                                                       org.apache.hadoop.fs.Path scratchDir,
                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                       java.util.concurrent.CountDownLatch startFlag,
                                                                       UserResolver resolver)
                                                                throws java.io.IOException

Throws:

java.io.IOException

getCurrentUserResolver

public UserResolver getCurrentUserResolver()

run

public int run(java.lang.String[] argv)
        throws java.io.IOException,
               java.lang.InterruptedException

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

java.io.IOException

java.lang.InterruptedException

main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception

Throws:

java.lang.Exception

printUsage

protected void printUsage(java.io.PrintStream out)

getSummarizer

protected org.apache.hadoop.mapred.gridmix.Summarizer getSummarizer()