Package org.apache.flink.runtime.operators.hash

Class HashPartition<BT,PT>

java.lang.Object
org.apache.flink.runtime.memory.AbstractPagedInputView
org.apache.flink.runtime.operators.hash.HashPartition<BT,PT>
Type Parameters:
BT - The type of the build side records.
PT - The type of the probe side records.
All Implemented Interfaces:
DataInput, org.apache.flink.core.memory.DataInputView, org.apache.flink.core.memory.SeekableDataInputView
Direct Known Subclasses:
ReOpenableHashPartition
public class HashPartition<BT,PT> extends AbstractPagedInputView implements org.apache.flink.core.memory.SeekableDataInputView

  • Field Details

    • overflowSegments

      protected org.apache.flink.core.memory.MemorySegment[] overflowSegments

    • numOverflowSegments

      protected int numOverflowSegments

    • nextOverflowBucket

      protected int nextOverflowBucket

    • partitionBuffers

      protected org.apache.flink.core.memory.MemorySegment[] partitionBuffers

    • probeSideBuffer

      protected ChannelWriterOutputView probeSideBuffer

    • probeSideRecordCounter

      protected long probeSideRecordCounter

    • recursionLevel

      protected int recursionLevel

    • probeSideChannel

      protected BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> probeSideChannel

    • furtherPartitioning

      protected boolean furtherPartitioning

  • Method Details

    • setFurtherPatitioning

      protected void setFurtherPatitioning(boolean v)

    • getPartitionNumber

      public int getPartitionNumber()
      Gets the partition number of this partition.
      Returns:
      This partition's number.

    • getRecursionLevel

      public int getRecursionLevel()
      Gets this partition's recursion level.
      Returns:
      The partition's recursion level.

    • isInMemory

      public final boolean isInMemory()
      Checks whether this partition is in memory or spilled.
      Returns:
      True, if the partition is in memory, false if it is spilled.

    • getNumOccupiedMemorySegments

      public int getNumOccupiedMemorySegments()
      Gets the number of memory segments used by this partition, which includes build side memory buffers and overflow memory segments.
      Returns:
      The number of occupied memory segments.

    • getBuildSideBlockCount

      public int getBuildSideBlockCount()

    • getProbeSideBlockCount

      public int getProbeSideBlockCount()

    • getBuildSideRecordCount

      public long getBuildSideRecordCount()

    • getProbeSideRecordCount

      public long getProbeSideRecordCount()

    • getBuildSideChannel

      public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> getBuildSideChannel()

    • getProbeSideChannel

      public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> getProbeSideChannel()

    • insertIntoBuildBuffer

      public final long insertIntoBuildBuffer(BT record) throws IOException
      Inserts the given object into the current buffer. This method returns a pointer that can be used to address the written record in this partition, if it is in-memory. The returned pointers have no expressiveness in the case where the partition is spilled.
      Parameters:
      record - The object to be written to the partition.
      Returns:
      A pointer to the object in the partition, or -1, if the partition is spilled.
      Throws:
      IOException - Thrown, when this is a spilled partition and the write failed.

    • insertIntoProbeBuffer

      public final void insertIntoProbeBuffer(PT record) throws IOException
      Inserts the given record into the probe side buffers. This method is only applicable when the partition was spilled while processing the build side.

      If this method is invoked when the partition is still being built, it has undefined behavior.

      Parameters:
      record - The record to be inserted into the probe side buffers.
      Throws:
      IOException - Thrown, if the buffer is full, needs to be spilled, and spilling causes an error.

    • spillPartition

      public int spillPartition(List<org.apache.flink.core.memory.MemorySegment> target, IOManager ioAccess, FileIOChannel.ID targetChannel, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> bufferReturnQueue) throws IOException
      Spills this partition to disk and sets it up such that it continues spilling records that are added to it. The spilling process must free at least one buffer, either in the partition's record buffers, or in the memory segments for overflow buckets. The partition immediately takes back one buffer to use it for further spilling.
      Parameters:
      target - The list to which memory segments from overflow buckets are added.
      ioAccess - The I/O manager to be used to create a writer to disk.
      targetChannel - The id of the target channel for this partition.
      Returns:
      The number of buffers that were freed by spilling this partition.
      Throws:
      IOException - Thrown, if the writing failed.

    • finalizeBuildPhase

      public void finalizeBuildPhase(IOManager ioAccess, FileIOChannel.Enumerator probeChannelEnumerator, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> bufferReturnQueue) throws IOException
      Throws:
      IOException

    • finalizeProbePhase

      public int finalizeProbePhase(List<org.apache.flink.core.memory.MemorySegment> freeMemory, List<HashPartition<BT,PT>> spilledPartitions, boolean keepUnprobedSpilledPartitions) throws IOException
      Parameters:
      keepUnprobedSpilledPartitions - If true then partitions that were spilled but received no further probe requests will be retained; used for build-side outer joins.
      Returns:
      The number of write-behind buffers reclaimable after this method call.
      Throws:
      IOException

    • clearAllMemory

      public void clearAllMemory(List<org.apache.flink.core.memory.MemorySegment> target)

    • prepareProbePhase

      public void prepareProbePhase(IOManager ioAccess, FileIOChannel.Enumerator probeChannelEnumerator, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> bufferReturnQueue) throws IOException
      Throws:
      IOException

    • setReadPosition

      public void setReadPosition(long pointer)
      Specified by:
      setReadPosition in interface org.apache.flink.core.memory.SeekableDataInputView

    • nextSegment

      protected org.apache.flink.core.memory.MemorySegment nextSegment(org.apache.flink.core.memory.MemorySegment current) throws IOException
      Description copied from class: AbstractPagedInputView
      The method by which concrete subclasses realize page crossing. This method is invoked when the current page is exhausted and a new page is required to continue the reading. If no further page is available, this method must throw an EOFException.
      Specified by:
      nextSegment in class AbstractPagedInputView
      Parameters:
      current - The current page that was read to its limit. May be null, if this method is invoked for the first time.
      Returns:
      The next page from which the reading should continue. May not be null. If the input is exhausted, an EOFException must be thrown instead.
      Throws:
      EOFException - Thrown, if no further segment is available.
      IOException - Thrown, if the method cannot provide the next page due to an I/O related problem.

    • getLimitForSegment

      protected int getLimitForSegment(org.apache.flink.core.memory.MemorySegment segment)
      Description copied from class: AbstractPagedInputView
      Gets the limit for reading bytes from the given memory segment. This method must return the position of the byte after the last valid byte in the given memory segment. When the position returned by this method is reached, the view will attempt to switch to the next memory segment.
      Specified by:
      getLimitForSegment in class AbstractPagedInputView
      Parameters:
      segment - The segment to determine the limit for.
      Returns:
      The limit for the given memory segment.