Package org.apache.hadoop.mapreduce.task.reduce

Class DirectShuffleFetcher<K,V>

java.lang.Object

java.lang.Thread

org.apache.hadoop.mapreduce.task.reduce.DirectShuffleFetcher<K,V>

All Implemented Interfaces:

java.lang.Runnable

public class DirectShuffleFetcher<K,V>
extends java.lang.Thread

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.fs.FileSystem	rfs

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor

Description

DirectShuffleFetcher(int id, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapreduce.TaskAttemptID reduceId, DirectShuffleSchedulerImpl<K,V> scheduler, org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V> merger, org.apache.hadoop.mapred.Reporter reporter, org.apache.hadoop.mapreduce.task.reduce.ShuffleClientMetrics metrics, org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter, org.apache.hadoop.mapred.MapOutputFile mapOutputFile)

Represents the result of an attempt to copy a map output

Method Summary

Modifier and Type	Method	Description
protected long	copyOutput(MapOutputLocation loc)	Copies a a map output from a remote host.
boolean	fail()	Fail the current file that we are fetching
MapOutputLocation	getLocation()	Get the current map output location.
void	run()	Loop forever and fetch map outputs as they become available.
void	shutDown()

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, onSpinWait, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

rfs

protected org.apache.hadoop.fs.FileSystem rfs

Constructor Detail

DirectShuffleFetcher

public DirectShuffleFetcher(int id,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            org.apache.hadoop.mapreduce.TaskAttemptID reduceId,
                            DirectShuffleSchedulerImpl<K,V> scheduler,
                            org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V> merger,
                            org.apache.hadoop.mapred.Reporter reporter,
                            org.apache.hadoop.mapreduce.task.reduce.ShuffleClientMetrics metrics,
                            org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter,
                            org.apache.hadoop.mapred.MapOutputFile mapOutputFile)

Represents the result of an attempt to copy a map output

Method Detail

fail

public boolean fail()

Fail the current file that we are fetching

Returns:

were we currently fetching?

getLocation

public MapOutputLocation getLocation()

Get the current map output location.

shutDown

public void shutDown()

run

public void run()

Loop forever and fetch map outputs as they become available.

Specified by:

run in interface java.lang.Runnable

Overrides:

run in class java.lang.Thread

copyOutput

@VisibleForTesting
protected long copyOutput(MapOutputLocation loc)
                   throws java.io.IOException,
                          java.lang.InterruptedException

Copies a a map output from a remote host.

Throws:

java.io.IOException - if there is an error copying the file

java.lang.InterruptedException - if the copier should give up