Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.file

Class ProducerMergedPartitionFileReader

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.ProducerMergedPartitionFileReader

All Implemented Interfaces:

PartitionFileReader

public class ProducerMergedPartitionFileReader
extends Object
implements PartitionFileReader

The implementation of PartitionFileReader with producer-merge mode. In this mode, the shuffle data is written in the producer side, the consumer side need to read multiple producers to get its partition data.

Note that one partition file may contain the data of multiple subpartitions.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ProducerMergedPartitionFileReader.ProducerMergedReadProgress	The implementation of PartitionFileReader.ReadProgress mainly includes current reading offset, end of read offset, etc.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.PartitionFileReader

PartitionFileReader.ReadBufferResult, PartitionFileReader.ReadProgress

Method Summary

Modifier and Type	Method	Description
long	getPriority(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId, int bufferIndex, PartitionFileReader.ReadProgress readProgress)	Get the priority for reading a particular buffer from the partitioned file.
PartitionFileReader.ReadBufferResult	readBuffer(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId, int bufferIndex, org.apache.flink.core.memory.MemorySegment memorySegment, BufferRecycler recycler, PartitionFileReader.ReadProgress readProgress, CompositeBuffer partialBuffer)	Read a buffer from the partition file.
void	release()	Release the PartitionFileReader.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

readBuffer

public PartitionFileReader.ReadBufferResult readBuffer(TieredStoragePartitionId partitionId,
 TieredStorageSubpartitionId subpartitionId,
 int segmentId,
 int bufferIndex,
 org.apache.flink.core.memory.MemorySegment memorySegment,
 BufferRecycler recycler,
 @Nullable
 PartitionFileReader.ReadProgress readProgress,
 @Nullable
 CompositeBuffer partialBuffer)
                                                throws IOException

Description copied from interface: PartitionFileReader

Read a buffer from the partition file.

Specified by:

readBuffer in interface PartitionFileReader

Parameters:

partitionId - the partition id of the buffer

subpartitionId - the subpartition id of the buffer

segmentId - the segment id of the buffer

bufferIndex - the index of buffer

memorySegment - the empty buffer to store the read buffer

recycler - the buffer recycler

readProgress - the current read progress. The progress comes from the previous ReadBufferResult. Note that the read progress should be implemented and provided by Flink, and it should be directly tied to the file format. The field can be null if the current file reader has no the read progress

partialBuffer - the previous partial buffer. The partial buffer is not null only when the last read has a partial buffer, it will construct a full buffer during the read process.

Returns:

null if there is no data otherwise return a read buffer result.

Throws:

IOException - if an error happens.

getPriority

public long getPriority(TieredStoragePartitionId partitionId,
 TieredStorageSubpartitionId subpartitionId,
 int segmentId,
 int bufferIndex,
 @Nullable
 PartitionFileReader.ReadProgress readProgress)
                 throws IOException

Description copied from interface: PartitionFileReader

Get the priority for reading a particular buffer from the partitioned file. The priority is defined as, it is suggested to read buffers with higher priority (smaller value) in prior to buffers with lower priority (larger value).

Depending on the partition file implementation, following the suggestions should typically result in better performance and efficiency. This can be achieved by e.g. choosing preloaded data over others, optimizing the order of disk access to be more sequential, etc.

Note: Priorities are suggestions rather than a requirements. The caller can still read data in whichever order it wants.

Specified by:

getPriority in interface PartitionFileReader

Parameters:

partitionId - the partition id of the buffer

subpartitionId - the subpartition id of the buffer

segmentId - the segment id of the buffer

bufferIndex - the index of buffer

readProgress - the current read progress. The progress comes from the previous ReadBufferResult. Note that the read progress should be implemented and provided by Flink, and it should be directly tied to the file format. The field can be null if the current file reader has no the read progress

Returns:

the priority of the PartitionFileReader.

Throws:

IOException - if an error happens.

release

public void release()

Description copied from interface: PartitionFileReader

Release the PartitionFileReader.

Specified by:

release in interface PartitionFileReader