Package org.apache.hadoop.mapreduce.task.reduce

Class DirectShuffleMergeManagerImpl<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.task.reduce.DirectShuffleMergeManagerImpl<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

@LimitedPrivate("MapReduce")
@Unstable
public class DirectShuffleMergeManagerImpl<K,V>
extends java.lang.Object
implements org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.mapred.MapOutputFile	mapOutputFile

Constructor Summary

Constructors

Constructor

Description

DirectShuffleMergeManagerImpl(org.apache.hadoop.mapreduce.TaskAttemptID reduceId, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.mapred.Reporter reporter, org.apache.hadoop.io.compress.CompressionCodec codec, java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> combinerClass, org.apache.hadoop.mapred.Task.CombineOutputCollector<K,V> combineCollector, org.apache.hadoop.mapred.Counters.Counter spilledRecordsCounter, org.apache.hadoop.mapred.Counters.Counter reduceCombineInputCounter, org.apache.hadoop.mapred.Counters.Counter mergedMapOutputsCounter, org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter, org.apache.hadoop.util.Progress mergePhase, org.apache.hadoop.mapred.MapOutputFile mapOutputFile)

Method Summary

Modifier and Type	Method	Description
protected boolean	canShuffleToMemory(long requestedSize)
org.apache.hadoop.mapred.RawKeyValueIterator	close()
void	closeInMemoryFile(DirectInMemoryOutput<K,V> mapOutput)
void	closeInMemoryMergedFile(DirectInMemoryOutput<K,V> mapOutput)
void	closeOnDiskFile(org.apache.hadoop.fs.FileStatus fileStatus)
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<DirectInMemoryOutput<K,V>,K,V>	createInMemoryMerger()
org.apache.hadoop.mapreduce.task.reduce.MapOutput<K,V>	reserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId, long requestedSize, int fetcher)
protected DirectInMemoryOutput<K,V>	unconditionalReserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId, long requestedSize, boolean primaryMapOutput)	Unconditional Reserve is used by the Memory-to-Memory thread
void	waitForResource()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mapOutputFile

protected org.apache.hadoop.mapred.MapOutputFile mapOutputFile

Constructor Detail

DirectShuffleMergeManagerImpl

public DirectShuffleMergeManagerImpl(org.apache.hadoop.mapreduce.TaskAttemptID reduceId,
                                     org.apache.hadoop.mapred.JobConf jobConf,
                                     org.apache.hadoop.fs.FileSystem localFS,
                                     org.apache.hadoop.mapred.Reporter reporter,
                                     org.apache.hadoop.io.compress.CompressionCodec codec,
                                     java.lang.Class<? extends org.apache.hadoop.mapred.Reducer> combinerClass,
                                     org.apache.hadoop.mapred.Task.CombineOutputCollector<K,V> combineCollector,
                                     org.apache.hadoop.mapred.Counters.Counter spilledRecordsCounter,
                                     org.apache.hadoop.mapred.Counters.Counter reduceCombineInputCounter,
                                     org.apache.hadoop.mapred.Counters.Counter mergedMapOutputsCounter,
                                     org.apache.hadoop.mapreduce.task.reduce.ExceptionReporter exceptionReporter,
                                     org.apache.hadoop.util.Progress mergePhase,
                                     org.apache.hadoop.mapred.MapOutputFile mapOutputFile)
                              throws java.io.IOException

Throws:

java.io.IOException

Method Detail

createInMemoryMerger

protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<DirectInMemoryOutput<K,V>,K,V> createInMemoryMerger()

waitForResource

public void waitForResource()
                     throws java.lang.InterruptedException

Specified by:

waitForResource in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

java.lang.InterruptedException

canShuffleToMemory

@VisibleForTesting
protected boolean canShuffleToMemory(long requestedSize)

reserve

public org.apache.hadoop.mapreduce.task.reduce.MapOutput<K,V> reserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId,
                                                                            long requestedSize,
                                                                            int fetcher)
                                                                     throws java.io.IOException

Specified by:

reserve in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

java.io.IOException

unconditionalReserve

@VisibleForTesting
protected DirectInMemoryOutput<K,V> unconditionalReserve(org.apache.hadoop.mapreduce.TaskAttemptID mapId,
                                                               long requestedSize,
                                                               boolean primaryMapOutput)

Unconditional Reserve is used by the Memory-to-Memory thread

Returns:

closeInMemoryFile

public void closeInMemoryFile(DirectInMemoryOutput<K,V> mapOutput)

closeInMemoryMergedFile

public void closeInMemoryMergedFile(DirectInMemoryOutput<K,V> mapOutput)

closeOnDiskFile

public void closeOnDiskFile(org.apache.hadoop.fs.FileStatus fileStatus)

close

public org.apache.hadoop.mapred.RawKeyValueIterator close()
                                                   throws java.lang.Throwable

Specified by:

close in interface org.apache.hadoop.mapreduce.task.reduce.MergeManager<K,V>

Throws:

java.lang.Throwable