org.apache.hadoop.mapreduce.task.reduce

Class DirectShuffleMergeManagerImpl<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

@InterfaceAudience.LimitedPrivate(value="MapReduce")
@InterfaceStability.Unstable
public class DirectShuffleMergeManagerImpl<K,V>
extends Object
implements org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapred.MapOutputFile	mapOutputFile

Constructor Summary

Constructors

Constructor and Description

DirectShuffleMergeManagerImpl(org.apache.hadoop.mapreduce.TaskAttemptID reduceId, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.mapred.Reporter reporter, org.apache.hadoop.io.compress.CompressionCodec codec, Class<? extends org.apache.hadoop.mapred.Reducer> combinerClass, org.apache.hadoop.mapred.Task.CombineOutputCollector<K,V> combineCollector, org.apache.hadoop.mapred.Counters.Counter spilledRecordsCounter, org.apache.hadoop.mapred.Counters.Counter reduceCombineInputCounter, org.apache.hadoop.mapred.Counters.Counter mergedMapOutputsCounter, org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter, org.apache.hadoop.util.Progress mergePhase, org.apache.hadoop.mapred.MapOutputFile mapOutputFile)

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	canShuffleToMemory(long requestedSize)
org.apache.hadoop.mapred.RawKeyValueIterator	close()
void	closeInMemoryFile(DirectInMemoryOutput<K,V> mapOutput)
void	closeInMemoryMergedFile(DirectInMemoryOutput<K,V> mapOutput)
void	closeOnDiskFile(org.apache.hadoop.fs.FileStatus fileStatus)
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<DirectInMemoryOutput<K,V>,K,V>	createInMemoryMerger()
org.apache.hadoop.mapreduce.task.reduce.MapOutput<K,V>	reserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId, long requestedSize, int fetcher)
protected DirectInMemoryOutput<K,V>	unconditionalReserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId, long requestedSize, boolean primaryMapOutput) Unconditional Reserve is used by the Memory-to-Memory thread
void	waitForResource()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mapOutputFile

protected org.apache.hadoop.mapred.MapOutputFile mapOutputFile

Constructor Detail

DirectShuffleMergeManagerImpl

public DirectShuffleMergeManagerImpl(org.apache.hadoop.mapreduce.TaskAttemptID reduceId,
                             org.apache.hadoop.mapred.JobConf jobConf,
                             org.apache.hadoop.fs.FileSystem localFS,
                             org.apache.hadoop.mapred.Reporter reporter,
                             org.apache.hadoop.io.compress.CompressionCodec codec,
                             Class<? extends org.apache.hadoop.mapred.Reducer> combinerClass,
                             org.apache.hadoop.mapred.Task.CombineOutputCollector<K,V> combineCollector,
                             org.apache.hadoop.mapred.Counters.Counter spilledRecordsCounter,
                             org.apache.hadoop.mapred.Counters.Counter reduceCombineInputCounter,
                             org.apache.hadoop.mapred.Counters.Counter mergedMapOutputsCounter,
                             org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter,
                             org.apache.hadoop.util.Progress mergePhase,
                             org.apache.hadoop.mapred.MapOutputFile mapOutputFile)
                              throws IOException

Throws:

IOException

Method Detail

createInMemoryMerger

protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<DirectInMemoryOutput<K,V>,K,V> createInMemoryMerger()

waitForResource

public void waitForResource()
                     throws InterruptedException

Specified by:

waitForResource in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

InterruptedException

canShuffleToMemory

protected boolean canShuffleToMemory(long requestedSize)

reserve

public org.apache.hadoop.mapreduce.task.reduce.MapOutput<K,V> reserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId,
                                                             long requestedSize,
                                                             int fetcher)
                                                               throws IOException

Specified by:

reserve in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

IOException

unconditionalReserve

protected DirectInMemoryOutput<K,V> unconditionalReserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId,
                                             long requestedSize,
                                             boolean primaryMapOutput)

Unconditional Reserve is used by the Memory-to-Memory thread

Returns:

closeInMemoryFile

public void closeInMemoryFile(DirectInMemoryOutput<K,V> mapOutput)

closeInMemoryMergedFile

public void closeInMemoryMergedFile(DirectInMemoryOutput<K,V> mapOutput)

closeOnDiskFile

public void closeOnDiskFile(org.apache.hadoop.fs.FileStatus fileStatus)

close

public org.apache.hadoop.mapred.RawKeyValueIterator close()
                                                   throws Throwable

Specified by:

close in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

Throwable