Package org.apache.hadoop.mapred.gridmix

Class StressJobFactory

java.lang.Object

org.apache.hadoop.mapred.gridmix.StressJobFactory

public class StressJobFactory
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.conf.Configuration	conf
static java.lang.String	CONF_MAX_JOB_TRACKER_RATIO
static java.lang.String	CONF_MAX_MAPSLOT_SHARE_PER_JOB
static java.lang.String	CONF_MAX_REDUCESLOT_SHARE_PER_JOB
static java.lang.String	CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO
static java.lang.String	CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO
protected java.io.IOException	error
protected JobCreator	jobCreator
protected org.apache.hadoop.tools.rumen.JobStoryProducer	jobProducer
protected java.util.concurrent.locks.ReentrantLock	lock
static org.slf4j.Logger	LOG
protected int	numJobsInTrace
protected float	rateFactor
protected java.lang.Thread	rThread
protected org.apache.hadoop.fs.Path	scratch
protected java.util.concurrent.atomic.AtomicInteger	sequence
protected java.util.concurrent.CountDownLatch	startFlag
protected org.apache.hadoop.mapred.gridmix.JobSubmitter	submitter
protected UserResolver	userResolver

Constructor Summary

Constructors

Constructor	Description
StressJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter, org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer, org.apache.hadoop.fs.Path scratch, org.apache.hadoop.conf.Configuration conf, java.util.concurrent.CountDownLatch startFlag, UserResolver resolver)	Creating a new instance does not start the thread.

Method Summary

Modifier and Type	Method	Description
void	abort()	Interrupt the reader thread.
void	add(java.lang.Void ignored)	Add is disabled.
protected void	checkLoadAndGetSlotsToBackfill()	We try to use some light-weight mechanism to determine cluster load.
java.lang.Thread	createReaderThread()
java.io.IOException	error()	Obtain the error that caused the thread to exit unexpectedly.
protected org.apache.hadoop.tools.rumen.JobStory	getNextJobFiltered()
void	join(long millis)	Wait for the reader thread to exhaust the job trace.
void	shutdown()	Interrupt the reader thread.
void	start()	Start the reader thread, wait for latch if necessary.
void	update(org.apache.hadoop.mapred.gridmix.Statistics.ClusterStats item)	STRESS Once you get the notification from StatsCollector.Collect the clustermetrics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO

public static final java.lang.String CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO

See Also:

Constant Field Values

CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO

public static final java.lang.String CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO

See Also:

Constant Field Values

CONF_MAX_MAPSLOT_SHARE_PER_JOB

public static final java.lang.String CONF_MAX_MAPSLOT_SHARE_PER_JOB

See Also:

Constant Field Values

CONF_MAX_REDUCESLOT_SHARE_PER_JOB

public static final java.lang.String CONF_MAX_REDUCESLOT_SHARE_PER_JOB

See Also:

Constant Field Values

CONF_MAX_JOB_TRACKER_RATIO

public static final java.lang.String CONF_MAX_JOB_TRACKER_RATIO

See Also:

Constant Field Values

scratch

protected final org.apache.hadoop.fs.Path scratch

rateFactor

protected final float rateFactor

conf

protected final org.apache.hadoop.conf.Configuration conf

rThread

protected final java.lang.Thread rThread

sequence

protected final java.util.concurrent.atomic.AtomicInteger sequence

submitter

protected final org.apache.hadoop.mapred.gridmix.JobSubmitter submitter

startFlag

protected final java.util.concurrent.CountDownLatch startFlag

userResolver

protected final UserResolver userResolver

jobCreator

protected final JobCreator jobCreator

error

protected volatile java.io.IOException error

jobProducer

protected final org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer

lock

protected final java.util.concurrent.locks.ReentrantLock lock

numJobsInTrace

protected int numJobsInTrace

Constructor Detail

StressJobFactory

public StressJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter,
                        org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer,
                        org.apache.hadoop.fs.Path scratch,
                        org.apache.hadoop.conf.Configuration conf,
                        java.util.concurrent.CountDownLatch startFlag,
                        UserResolver resolver)
                 throws java.io.IOException

Creating a new instance does not start the thread.

Parameters:

submitter - Component to which deserialized jobs are passed

jobProducer - Stream of job traces with which to construct a ZombieJobProducer

scratch - Directory into which to write output from simulated jobs

conf - Config passed to all jobs to be submitted

startFlag - Latch released from main to start pipeline

Throws:

java.io.IOException

Method Detail

createReaderThread

public java.lang.Thread createReaderThread()

update

public void update(org.apache.hadoop.mapred.gridmix.Statistics.ClusterStats item)

STRESS Once you get the notification from StatsCollector.Collect the clustermetrics. Update current loadStatus with new load status of JT.

Parameters:

item -

checkLoadAndGetSlotsToBackfill

protected void checkLoadAndGetSlotsToBackfill()
                                       throws java.io.IOException,
                                              java.lang.InterruptedException

We try to use some light-weight mechanism to determine cluster load.

Throws:

java.io.IOException

java.lang.InterruptedException

start

public void start()

Start the reader thread, wait for latch if necessary.

getNextJobFiltered

protected org.apache.hadoop.tools.rumen.JobStory getNextJobFiltered()
                                                             throws java.io.IOException

Throws:

java.io.IOException

error

public java.io.IOException error()

Obtain the error that caused the thread to exit unexpectedly.

add

public void add(java.lang.Void ignored)

Add is disabled.

Throws:

java.lang.UnsupportedOperationException

join

public void join(long millis)
          throws java.lang.InterruptedException

Wait for the reader thread to exhaust the job trace.

Throws:

java.lang.InterruptedException

shutdown

public void shutdown()

Interrupt the reader thread.

abort

public void abort()

Interrupt the reader thread. This requires no special consideration, as the thread has no pending work queue.