Package org.apache.flink.runtime.scheduler.adaptivebatch

Class AllToAllBlockingResultInfo

java.lang.Object
org.apache.flink.runtime.scheduler.adaptivebatch.AllToAllBlockingResultInfo
All Implemented Interfaces:
IntermediateResultInfo, BlockingResultInfo
public class AllToAllBlockingResultInfo extends Object
Information of All-To-All result.

  • Field Details

    • aggregatedSubpartitionBytes

      @Nullable protected List<Long> aggregatedSubpartitionBytes
      Aggregated subpartition bytes, which aggregates the subpartition bytes with the same subpartition index in different partitions. Note that We can aggregate them because they will be consumed by the same downstream task.

    • numOfPartitions

      protected final int numOfPartitions

    • numOfSubpartitions

      protected final int numOfSubpartitions

    • subpartitionBytesByPartitionIndex

      protected final Map<Integer,long[]> subpartitionBytesByPartitionIndex
      The subpartition bytes map. The key is the partition index, value is a subpartition bytes list.

  • Method Details

    • isBroadcast

      public boolean isBroadcast()
      Description copied from interface: IntermediateResultInfo
      Determines whether the associated intermediate data set uses a broadcast distribution pattern.

      A broadcast distribution pattern indicates that all data produced by this intermediate data set should be broadcast to every downstream consumer.

      Returns:
      true if the intermediate data set is using a broadcast distribution pattern; false otherwise.

    • isSingleSubpartitionContainsAllData

      public boolean isSingleSubpartitionContainsAllData()
      Description copied from interface: IntermediateResultInfo
      Checks whether there is a single subpartition that contains all the produced data.
      Returns:
      true if one subpartition that contains all the data; false otherwise.

    • isPointwise

      public boolean isPointwise()
      Description copied from interface: IntermediateResultInfo
      Whether it is a pointwise result.
      Returns:
      whether it is a pointwise result

    • getNumPartitions

      public int getNumPartitions()
      Description copied from interface: IntermediateResultInfo
      Get number of partitions for this result.
      Returns:
      the number of partitions in this result

    • getNumSubpartitions

      public int getNumSubpartitions(int partitionIndex)
      Description copied from interface: IntermediateResultInfo
      Get number of subpartitions for the given partition.
      Parameters:
      partitionIndex - the partition index
      Returns:
      the number of subpartitions of the partition

    • getNumBytesProduced

      public long getNumBytesProduced()
      Description copied from interface: BlockingResultInfo
      Return the num of bytes produced(numBytesProduced) by the producer.

      The difference between numBytesProduced and numBytesOut : numBytesProduced represents the number of bytes actually produced, and numBytesOut represents the number of bytes sent to downstream tasks. In unicast scenarios, these two values should be equal. In broadcast scenarios, numBytesOut should be (N * numBytesProduced), where N refers to the number of subpartitions.

      Returns:
      the num of bytes produced by the producer

    • recordPartitionInfo

      public void recordPartitionInfo(int partitionIndex, ResultPartitionBytes partitionBytes)
      Description copied from interface: BlockingResultInfo
      Record the information of the result partition.
      Specified by:
      recordPartitionInfo in interface BlockingResultInfo
      Parameters:
      partitionIndex - the intermediate result partition index
      partitionBytes - the ResultPartitionBytes of the partition

    • onFineGrainedSubpartitionBytesNotNeeded

      protected void onFineGrainedSubpartitionBytesNotNeeded()
      This method should be called when fine-grained information is no longer needed. It will aggregate and clears the fine-grained subpartition bytes to reduce space usage.

      Once all partitions are finished and all consumer jobVertices are initialized, we can convert the subpartition bytes to aggregated value to reduce the space usage, because the distribution of source splits does not affect the distribution of data consumed by downstream tasks of ALL_TO_ALL edges(Hashing or Rebalancing, we do not consider rare cases such as custom partitions here).

    • resetPartitionInfo

      public void resetPartitionInfo(int partitionIndex)
      Description copied from interface: BlockingResultInfo
      Reset the information of the result partition.
      Specified by:
      resetPartitionInfo in interface BlockingResultInfo
      Parameters:
      partitionIndex - the intermediate result partition index

    • getNumBytesProduced

      public long getNumBytesProduced(IndexRange partitionIndexRange, IndexRange subpartitionIndexRange)
      Description copied from interface: BlockingResultInfo
      Return the aggregated num of bytes according to the index range for partition and subpartition.
      Specified by:
      getNumBytesProduced in interface BlockingResultInfo
      Parameters:
      partitionIndexRange - range of the index of the consumed partition.
      subpartitionIndexRange - range of the index of the consumed subpartition.
      Returns:
      aggregated bytes according to the index ranges.

    • getAggregatedSubpartitionBytes

      public List<Long> getAggregatedSubpartitionBytes()

    • getResultId

      public IntermediateDataSetID getResultId()
      Description copied from interface: IntermediateResultInfo
      Get the intermediate result id.
      Specified by:
      getResultId in interface IntermediateResultInfo
      Returns:
      the intermediate result id

    • getSubpartitionBytesByPartitionIndex

      public Map<Integer,long[]> getSubpartitionBytesByPartitionIndex()
      Description copied from interface: BlockingResultInfo
      Gets subpartition bytes by partition index.
      Specified by:
      getSubpartitionBytesByPartitionIndex in interface BlockingResultInfo
      Returns:
      a map with integer keys representing partition indices and long array values representing subpartition bytes.