org.apache.hadoop.mapreduce.task.reduce
Class MergeManagerImpl<K,V>

java.lang.Object
  extended by org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl<K,V>
All Implemented Interfaces:
MergeManager<K,V>

@InterfaceAudience.LimitedPrivate(value="MapReduce")
@InterfaceStability.Unstable
public class MergeManagerImpl<K,V>
extends Object
implements MergeManager<K,V>


Field Summary
protected  MapOutputFile mapOutputFile
           
 
Constructor Summary
MergeManagerImpl(TaskAttemptID reduceId, JobConf jobConf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator, Reporter reporter, org.apache.hadoop.io.compress.CompressionCodec codec, Class<? extends Reducer> combinerClass, Task.CombineOutputCollector<K,V> combineCollector, Counters.Counter spilledRecordsCounter, Counters.Counter reduceCombineInputCounter, Counters.Counter mergedMapOutputsCounter, ExceptionReporter exceptionReporter, org.apache.hadoop.util.Progress mergePhase, MapOutputFile mapOutputFile)
           
 
Method Summary
 RawKeyValueIterator close()
          Called at the end of shuffle.
 void closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
           
 void closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
           
 void closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)
           
protected  org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V> createInMemoryMerger()
           
 MapOutput<K,V> reserve(TaskAttemptID mapId, long requestedSize, int fetcher)
          To reserve resources for data to be shuffled.
 void waitForResource()
          To wait until merge has some freed resources available so that it can accept shuffled data.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

mapOutputFile

protected MapOutputFile mapOutputFile
Constructor Detail

MergeManagerImpl

public MergeManagerImpl(TaskAttemptID reduceId,
                        JobConf jobConf,
                        org.apache.hadoop.fs.FileSystem localFS,
                        org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
                        Reporter reporter,
                        org.apache.hadoop.io.compress.CompressionCodec codec,
                        Class<? extends Reducer> combinerClass,
                        Task.CombineOutputCollector<K,V> combineCollector,
                        Counters.Counter spilledRecordsCounter,
                        Counters.Counter reduceCombineInputCounter,
                        Counters.Counter mergedMapOutputsCounter,
                        ExceptionReporter exceptionReporter,
                        org.apache.hadoop.util.Progress mergePhase,
                        MapOutputFile mapOutputFile)
Method Detail

createInMemoryMerger

protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V> createInMemoryMerger()

waitForResource

public void waitForResource()
                     throws InterruptedException
Description copied from interface: MergeManager
To wait until merge has some freed resources available so that it can accept shuffled data. This will be called before a network connection is established to get the map output.

Specified by:
waitForResource in interface MergeManager<K,V>
Throws:
InterruptedException

reserve

public MapOutput<K,V> reserve(TaskAttemptID mapId,
                              long requestedSize,
                              int fetcher)
                       throws IOException
Description copied from interface: MergeManager
To reserve resources for data to be shuffled. This will be called after a network connection is made to shuffle the data.

Specified by:
reserve in interface MergeManager<K,V>
Parameters:
mapId - mapper from which data will be shuffled.
requestedSize - size in bytes of data that will be shuffled.
fetcher - id of the map output fetcher that will shuffle the data.
Returns:
a MapOutput object that can be used by shuffle to shuffle data. If required resources cannot be reserved immediately, a null can be returned.
Throws:
IOException

closeInMemoryFile

public void closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)

closeInMemoryMergedFile

public void closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)

closeOnDiskFile

public void closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)

close

public RawKeyValueIterator close()
                          throws Throwable
Description copied from interface: MergeManager
Called at the end of shuffle.

Specified by:
close in interface MergeManager<K,V>
Returns:
a key value iterator object.
Throws:
Throwable


Copyright © 2014 Apache Software Foundation. All Rights Reserved.