Package org.apache.flink.runtime.io.network.partition

Class SortBuffer

java.lang.Object
org.apache.flink.runtime.io.network.partition.SortBuffer
All Implemented Interfaces:
DataBuffer
Direct Known Subclasses:
SortBasedDataBuffer, TieredStorageSortBuffer
@NotThreadSafe public abstract class SortBuffer extends Object implements DataBuffer
A DataBuffer implementation which sorts all appended records only by subpartition index. Records of the same subpartition keep the appended order.

It maintains a list of MemorySegments as a joint buffer. Data will be appended to the joint buffer sequentially. When writing a record, an index entry will be appended first. An index entry consists of 4 fields: 4 bytes for record length, 4 bytes for Buffer.DataType and 8 bytes for address pointing to the next index entry of the same subpartition which will be used to index the next record to read when coping data from this DataBuffer. For simplicity, no index entry can span multiple segments. The corresponding record data is seated right after its index entry and different from the index entry, records have variable length thus may span multiple segments.

  • Field Details

    • INDEX_ENTRY_SIZE

      protected static final int INDEX_ENTRY_SIZE
      Size of an index entry: 4 bytes for record length, 4 bytes for data type and 8 bytes for pointer to next entry.
      See Also:

    • freeSegments

      protected final LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments
      A list of MemorySegments used to store data in memory.

    • bufferRecycler

      protected final BufferRecycler bufferRecycler
      BufferRecycler used to recycle freeSegments.

    • segments

      public final ArrayList<org.apache.flink.core.memory.MemorySegment> segments
      A segment list as a joint buffer which stores all records and index entries.

    • lastIndexEntryAddresses

      protected final long[] lastIndexEntryAddresses
      Addresses of the last record's index entry for each subpartition.

    • numTotalBytesRead

      protected long numTotalBytesRead
      Total number of bytes already read from this sort buffer.

    • isFinished

      protected boolean isFinished
      Whether this sort buffer is finished. One can only read a finished sort buffer.

    • isReleased

      protected boolean isReleased
      Whether this sort buffer is released. A released sort buffer can not be used.

    • subpartitionReadOrder

      protected final int[] subpartitionReadOrder
      Data of different subpartitions in this sort buffer will be read in this order.

    • readIndexEntryAddress

      protected long readIndexEntryAddress
      Index entry address of the current record or event to be read.

    • recordRemainingBytes

      protected int recordRemainingBytes
      Record bytes remaining after last copy, which must be read first in next copy.

    • readOrderIndex

      protected int readOrderIndex
      Used to index the current available subpartition to read data from.

  • Constructor Details

    • SortBuffer

      protected SortBuffer(LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments, BufferRecycler bufferRecycler, int numSubpartitions, int bufferSize, int numGuaranteedBuffers, @Nullable int[] customReadOrder)

  • Method Details

    • append

      public boolean append(ByteBuffer source, int targetSubpartition, Buffer.DataType dataType) throws IOException
      No partial record will be written to this SortBasedDataBuffer, which means that either all data of target record will be written or nothing will be written.
      Specified by:
      append in interface DataBuffer
      Throws:
      IOException

    • copyRecordOrEvent

      protected int copyRecordOrEvent(org.apache.flink.core.memory.MemorySegment targetSegment, int targetSegmentOffset, int sourceSegmentIndex, int sourceSegmentOffset, int recordLength)

    • updateReadSubpartitionAndIndexEntryAddress

      protected void updateReadSubpartitionAndIndexEntryAddress()

    • getSegmentIndexFromPointer

      protected int getSegmentIndexFromPointer(long value)

    • getSegmentOffsetFromPointer

      protected int getSegmentOffsetFromPointer(long value)

    • numTotalRecords

      public long numTotalRecords()
      Description copied from interface: DataBuffer
      Returns the total number of records written to this DataBuffer.
      Specified by:
      numTotalRecords in interface DataBuffer

    • numTotalBytes

      public long numTotalBytes()
      Description copied from interface: DataBuffer
      Returns the total number of bytes written to this DataBuffer.
      Specified by:
      numTotalBytes in interface DataBuffer

    • hasRemaining

      public boolean hasRemaining()
      Description copied from interface: DataBuffer
      Returns true if not all data appended to this DataBuffer is consumed.
      Specified by:
      hasRemaining in interface DataBuffer

    • finish

      public void finish()
      Description copied from interface: DataBuffer
      Finishes this DataBuffer which means no record can be appended anymore.
      Specified by:
      finish in interface DataBuffer

    • isFinished

      public boolean isFinished()
      Description copied from interface: DataBuffer
      Whether this DataBuffer is finished or not.
      Specified by:
      isFinished in interface DataBuffer

    • release

      public void release()
      Description copied from interface: DataBuffer
      Releases this DataBuffer which releases all resources.
      Specified by:
      release in interface DataBuffer

    • isReleased

      public boolean isReleased()
      Description copied from interface: DataBuffer
      Whether this DataBuffer is released or not.
      Specified by:
      isReleased in interface DataBuffer