org.apache.hadoop.mapreduce.task.reduce
Class MergeManagerImpl<K,V>
java.lang.Object
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl<K,V>
- All Implemented Interfaces:
- MergeManager<K,V>
@InterfaceAudience.LimitedPrivate(value="MapReduce")
@InterfaceStability.Unstable
public class MergeManagerImpl<K,V>
- extends Object
- implements MergeManager<K,V>
Constructor Summary |
MergeManagerImpl(TaskAttemptID reduceId,
JobConf jobConf,
org.apache.hadoop.fs.FileSystem localFS,
org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
Reporter reporter,
org.apache.hadoop.io.compress.CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter mergedMapOutputsCounter,
ExceptionReporter exceptionReporter,
org.apache.hadoop.util.Progress mergePhase,
MapOutputFile mapOutputFile)
|
Method Summary |
RawKeyValueIterator |
close()
Called at the end of shuffle. |
void |
closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
|
void |
closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
|
void |
closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)
|
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V> |
createInMemoryMerger()
|
MapOutput<K,V> |
reserve(TaskAttemptID mapId,
long requestedSize,
int fetcher)
To reserve resources for data to be shuffled. |
void |
waitForResource()
To wait until merge has some freed resources available so that it can
accept shuffled data. |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
mapOutputFile
protected MapOutputFile mapOutputFile
MergeManagerImpl
public MergeManagerImpl(TaskAttemptID reduceId,
JobConf jobConf,
org.apache.hadoop.fs.FileSystem localFS,
org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
Reporter reporter,
org.apache.hadoop.io.compress.CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter mergedMapOutputsCounter,
ExceptionReporter exceptionReporter,
org.apache.hadoop.util.Progress mergePhase,
MapOutputFile mapOutputFile)
createInMemoryMerger
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V> createInMemoryMerger()
waitForResource
public void waitForResource()
throws InterruptedException
- Description copied from interface:
MergeManager
- To wait until merge has some freed resources available so that it can
accept shuffled data. This will be called before a network connection is
established to get the map output.
- Specified by:
waitForResource
in interface MergeManager<K,V>
- Throws:
InterruptedException
reserve
public MapOutput<K,V> reserve(TaskAttemptID mapId,
long requestedSize,
int fetcher)
throws IOException
- Description copied from interface:
MergeManager
- To reserve resources for data to be shuffled. This will be called after
a network connection is made to shuffle the data.
- Specified by:
reserve
in interface MergeManager<K,V>
- Parameters:
mapId
- mapper from which data will be shuffled.requestedSize
- size in bytes of data that will be shuffled.fetcher
- id of the map output fetcher that will shuffle the data.
- Returns:
- a MapOutput object that can be used by shuffle to shuffle data. If
required resources cannot be reserved immediately, a null can be returned.
- Throws:
IOException
closeInMemoryFile
public void closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
closeInMemoryMergedFile
public void closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
closeOnDiskFile
public void closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)
close
public RawKeyValueIterator close()
throws Throwable
- Description copied from interface:
MergeManager
- Called at the end of shuffle.
- Specified by:
close
in interface MergeManager<K,V>
- Returns:
- a key value iterator object.
- Throws:
Throwable
Copyright © 2014 Apache Software Foundation. All Rights Reserved.