Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.common

Class TieredStorageUtils

java.lang.Object
org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils
public class TieredStorageUtils extends Object
Utils for reading from or writing to tiered storage.

  • Constructor Details

    • TieredStorageUtils

      public TieredStorageUtils()

  • Method Details

    • getNumBuffersTriggerFlushRatio

      public static float getNumBuffersTriggerFlushRatio()
      When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.
      Returns:
      flush ratio.

    • getAccumulatorExclusiveBuffers

      public static int getAccumulatorExclusiveBuffers()
      Get exclusive buffer number of accumulator.

      The buffer number is used to compare with the subpartition number to determine the type of BufferAccumulator.

      If the exclusive buffer number is larger than (subpartitionNum + 1), the accumulator will use HashBufferAccumulator. If the exclusive buffer number is equal to or smaller than (subpartitionNum + 1), the accumulator will use SortBufferAccumulator

      Returns:
      the buffer number.

    • getPoolSizeCheckInterval

      public static long getPoolSizeCheckInterval()
      Get the pool size check interval.

    • getMinBuffersPerGate

      public static int getMinBuffersPerGate()
      Get the number of minimum buffers per input gate. It is only used when taskmanager.network.hybrid-shuffle.memory-decoupling.enabled is set to true.
      Returns:
      the buffer number.

    • getMinBuffersPerResultPartition

      public static int getMinBuffersPerResultPartition()
      *

      Get the number of minimum buffers per result partition.

      Returns:
      the buffer number.

    • getMemoryTierName

      public static String getMemoryTierName()

    • getDiskTierName

      public static String getDiskTierName()

    • getRemoteTierName

      public static String getRemoteTierName()

    • generateBufferWithHeaders

      public static ByteBuffer[] generateBufferWithHeaders(List<org.apache.flink.api.java.tuple.Tuple2<Buffer,Integer>> bufferWithIndexes)

    • compressBufferIfPossible

      public static Buffer compressBufferIfPossible(Buffer buffer, BufferCompressor bufferCompressor)
      Try compress buffer if possible.

    • canBeCompressed

      public static boolean canBeCompressed(Buffer buffer, BufferCompressor bufferCompressor)
      Whether the buffer can be compressed or not. Note that event is not compressed because it is usually small and the size can become even larger after compression.