Package org.apache.flink.runtime.io.network.partition

Class ResultPartition

java.lang.Object
org.apache.flink.runtime.io.network.partition.ResultPartition
All Implemented Interfaces:
AutoCloseable, AvailabilityProvider, ResultPartitionWriter
Direct Known Subclasses:
BufferWritingResultPartition, SortMergeResultPartition, TieredResultPartition
public abstract class ResultPartition extends Object implements ResultPartitionWriter
A result partition for data produced by a single task.

This class is the runtime part of a logical IntermediateResultPartition. Essentially, a result partition is a collection of Buffer instances. The buffers are organized in one or more ResultSubpartition instances or in a joint structure which further partition the data depending on the number of consuming tasks and the data DistributionPattern.

Tasks, which consume a result partition have to request one of its subpartitions. The request happens either remotely (see RemoteInputChannel) or locally (see LocalInputChannel)

Life-cycle

The life-cycle of each result partition has three (possibly overlapping) phases:

  1. Produce:
  2. Consume:
  3. Release:

Buffer management

State management

  • Field Details

    • LOG

      protected static final org.slf4j.Logger LOG

    • partitionId

      protected final ResultPartitionID partitionId

    • partitionType

      protected final ResultPartitionType partitionType
      Type of this partition. Defines the concrete subpartition implementation to use.

    • partitionManager

      protected final ResultPartitionManager partitionManager

    • numSubpartitions

      protected final int numSubpartitions

    • bufferPool

      protected BufferPool bufferPool

    • bufferCompressor

      @Nullable protected final BufferCompressor bufferCompressor
      Used to compress buffer to reduce IO.

    • numBytesOut

      protected org.apache.flink.metrics.Counter numBytesOut

    • numBuffersOut

      protected org.apache.flink.metrics.Counter numBuffersOut

    • resultPartitionBytes

      protected ResultPartitionBytesCounter resultPartitionBytes

  • Constructor Details

  • Method Details

    • setup

      public void setup() throws IOException
      Registers a buffer pool with this result partition.

      There is one pool for each result partition, which is shared by all its sub partitions.

      The pool is registered with the partition *after* it as been constructed in order to conform to the life-cycle of task registrations in the TaskExecutor.

      Specified by:
      setup in interface ResultPartitionWriter
      Throws:
      IOException

    • setupInternal

      protected abstract void setupInternal() throws IOException
      Do the subclass's own setup operation.
      Throws:
      IOException

    • getOwningTaskName

      public String getOwningTaskName()

    • getPartitionId

      public ResultPartitionID getPartitionId()
      Specified by:
      getPartitionId in interface ResultPartitionWriter

    • getPartitionIndex

      public int getPartitionIndex()

    • getNumberOfSubpartitions

      public int getNumberOfSubpartitions()
      Specified by:
      getNumberOfSubpartitions in interface ResultPartitionWriter

    • getBufferPool

      public BufferPool getBufferPool()

    • isNumberOfPartitionConsumerUndefined

      public void isNumberOfPartitionConsumerUndefined(boolean isNumberOfPartitionConsumerUndefined)

    • isNumberOfPartitionConsumerUndefined

      public boolean isNumberOfPartitionConsumerUndefined()

    • getNumberOfQueuedBuffers

      public abstract int getNumberOfQueuedBuffers()
      Returns the total number of queued buffers of all subpartitions.

    • getSizeOfQueuedBuffersUnsafe

      public abstract long getSizeOfQueuedBuffersUnsafe()
      Returns the total size in bytes of queued buffers of all subpartitions.

    • getNumberOfQueuedBuffers

      public abstract int getNumberOfQueuedBuffers(int targetSubpartition)
      Returns the number of queued buffers of the given target subpartition.

    • setMaxOverdraftBuffersPerGate

      public void setMaxOverdraftBuffersPerGate(int maxOverdraftBuffersPerGate)
      Description copied from interface: ResultPartitionWriter
      Sets the max overdraft buffer size of per gate.
      Specified by:
      setMaxOverdraftBuffersPerGate in interface ResultPartitionWriter

    • getPartitionType

      public ResultPartitionType getPartitionType()
      Returns the type of this result partition.
      Returns:
      result partition type

    • getResultPartitionBytes

      public ResultPartitionBytesCounter getResultPartitionBytes()

    • notifyEndOfData

      public void notifyEndOfData(StopMode mode) throws IOException
      Description copied from interface: ResultPartitionWriter
      Notifies the downstream tasks that this ResultPartitionWriter have emitted all the user records.
      Specified by:
      notifyEndOfData in interface ResultPartitionWriter
      Parameters:
      mode - tells if we should flush all records or not (it is false in case of stop-with-savepoint (--no-drain))
      Throws:
      IOException

    • getAllDataProcessedFuture

      public CompletableFuture<Void> getAllDataProcessedFuture()
      Description copied from interface: ResultPartitionWriter
      Gets the future indicating whether all the records has been processed by the downstream tasks.
      Specified by:
      getAllDataProcessedFuture in interface ResultPartitionWriter

    • onSubpartitionAllDataProcessed

      public void onSubpartitionAllDataProcessed(int subpartition)
      The subpartition notifies that the corresponding downstream task have processed all the user records.
      Parameters:
      subpartition - The index of the subpartition sending the notification.
      See Also:

    • finish

      public void finish() throws IOException
      Finishes the result partition.

      After this operation, it is not possible to add further data to the result partition.

      For BLOCKING results, this will trigger the deployment of consuming tasks.

      Specified by:
      finish in interface ResultPartitionWriter
      Throws:
      IOException

    • isFinished

      public boolean isFinished()
      Specified by:
      isFinished in interface ResultPartitionWriter

    • release

      public void release()

    • release

      public void release(Throwable cause)
      Description copied from interface: ResultPartitionWriter
      Releases the partition writer which releases the produced data and no reader can consume the partition any more.
      Specified by:
      release in interface ResultPartitionWriter

    • releaseInternal

      protected abstract void releaseInternal()
      Releases all produced data including both those stored in memory and persisted on disk.

    • close

      public void close()
      Description copied from interface: ResultPartitionWriter
      Closes the partition writer which releases the allocated resource, for example the buffer pool.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface ResultPartitionWriter

    • fail

      public void fail(@Nullable Throwable throwable)
      Description copied from interface: ResultPartitionWriter
      Fail the production of the partition.

      This method propagates non-null failure causes to consumers on a best-effort basis. This call also leads to the release of all resources associated with the partition. Closing of the partition is still needed afterwards if it has not been done before.

      Specified by:
      fail in interface ResultPartitionWriter
      Parameters:
      throwable - failure cause

    • getFailureCause

      public Throwable getFailureCause()

    • getNumTargetKeyGroups

      public int getNumTargetKeyGroups()
      Specified by:
      getNumTargetKeyGroups in interface ResultPartitionWriter

    • setMetricGroup

      public void setMetricGroup(TaskIOMetricGroup metrics)
      Description copied from interface: ResultPartitionWriter
      Sets the metric group for the ResultPartitionWriter.
      Specified by:
      setMetricGroup in interface ResultPartitionWriter

    • createSubpartitionView

      public ResultSubpartitionView createSubpartitionView(ResultSubpartitionIndexSet indexSet, BufferAvailabilityListener availabilityListener) throws IOException
      Description copied from interface: ResultPartitionWriter
      Returns a reader for the subpartition with the given index range.
      Specified by:
      createSubpartitionView in interface ResultPartitionWriter
      Throws:
      IOException

    • createSubpartitionView

      protected abstract ResultSubpartitionView createSubpartitionView(int index, BufferAvailabilityListener availabilityListener) throws IOException
      Returns a reader for the subpartition with the given index.

      Given that the function to merge outputs from multiple subpartition views is supported uniformly in UnionResultSubpartitionView, subclasses of ResultPartition only needs to take care of creating subpartition view for a single subpartition.

      Throws:
      IOException

    • isReleased

      public boolean isReleased()
      Whether this partition is released.

      A partition is released when each subpartition is either consumed and communication is closed by consumer or failed. A partition is also released if task is cancelled.

      Specified by:
      isReleased in interface ResultPartitionWriter

    • getAvailableFuture

      public CompletableFuture<?> getAvailableFuture()
      Specified by:
      getAvailableFuture in interface AvailabilityProvider
      Returns:
      a future that is completed if the respective provider is available.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • checkInProduceState

      protected void checkInProduceState() throws IllegalStateException
      Throws:
      IllegalStateException

    • getPartitionManager

      @VisibleForTesting public ResultPartitionManager getPartitionManager()

    • canBeCompressed

      protected boolean canBeCompressed(Buffer buffer)
      Whether the buffer can be compressed or not. Note that event is not compressed because it is usually small and the size can become even larger after compression.