Package org.apache.flink.runtime.io.network.buffer

Class NetworkBufferPool

java.lang.Object
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool
All Implemented Interfaces:
org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider, BufferPoolFactory
public class NetworkBufferPool extends Object implements BufferPoolFactory, org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider
The NetworkBufferPool is a fixed size pool of MemorySegment instances for the network stack.

The NetworkBufferPool creates LocalBufferPools from which the individual tasks draw the buffers for the network data transfer. When new local buffer pools are created, the NetworkBufferPool dynamically redistributes the buffers between the pools.

  • Field Details

  • Constructor Details

    • NetworkBufferPool

      @VisibleForTesting public NetworkBufferPool(int numberOfSegmentsToAllocate, int segmentSize)

    • NetworkBufferPool

      public NetworkBufferPool(int numberOfSegmentsToAllocate, int segmentSize, Duration requestSegmentsTimeout)
      Allocates all MemorySegment instances managed by this pool.

  • Method Details

    • requestPooledMemorySegment

      @Nullable public org.apache.flink.core.memory.MemorySegment requestPooledMemorySegment()
      Different from requestUnpooledMemorySegments(int) for unpooled segments allocation. This method and the below requestPooledMemorySegmentsBlocking(int) method are designed to be used from LocalBufferPool for pooled memory segments allocation. Note that these methods for pooled memory segments requesting and recycling are prohibited from acquiring the factoryLock to avoid deadlock.

    • requestPooledMemorySegmentsBlocking

      public List<org.apache.flink.core.memory.MemorySegment> requestPooledMemorySegmentsBlocking(int numberOfSegmentsToRequest) throws IOException
      Throws:
      IOException

    • recyclePooledMemorySegment

      public void recyclePooledMemorySegment(org.apache.flink.core.memory.MemorySegment segment)
      Corresponding to requestPooledMemorySegmentsBlocking(int) and requestPooledMemorySegment(), this method is for pooled memory segments recycling.

    • requestUnpooledMemorySegments

      public List<org.apache.flink.core.memory.MemorySegment> requestUnpooledMemorySegments(int numberOfSegmentsToRequest) throws IOException
      Unpooled memory segments are requested directly from NetworkBufferPool, as opposed to pooled segments, that are requested through BufferPool that was created from this NetworkBufferPool (see createBufferPool(int, int)). They are used for example for exclusive RemoteInputChannel credits, that are permanently assigned to that channel, and never returned to any BufferPool. As opposed to pooled segments, when requested, unpooled segments needs to be accounted against numTotalRequiredBuffers, which might require redistribution of the segments.
      Specified by:
      requestUnpooledMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider
      Throws:
      IOException

    • recycleUnpooledMemorySegments

      public void recycleUnpooledMemorySegments(Collection<org.apache.flink.core.memory.MemorySegment> segments)
      Corresponding to requestUnpooledMemorySegments(int), this method is for unpooled memory segments recycling.
      Specified by:
      recycleUnpooledMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider

    • destroy

      public void destroy()

    • isDestroyed

      public boolean isDestroyed()

    • getTotalNumberOfMemorySegments

      public int getTotalNumberOfMemorySegments()

    • getTotalMemory

      public long getTotalMemory()

    • getNumberOfAvailableMemorySegments

      public int getNumberOfAvailableMemorySegments()

    • getAvailableMemory

      public long getAvailableMemory()

    • getNumberOfUsedMemorySegments

      public int getNumberOfUsedMemorySegments()

    • getUsedMemory

      public long getUsedMemory()

    • getNumberOfRegisteredBufferPools

      public int getNumberOfRegisteredBufferPools()

    • getEstimatedNumberOfRequestedMemorySegments

      public long getEstimatedNumberOfRequestedMemorySegments()

    • getEstimatedRequestedMemory

      public long getEstimatedRequestedMemory()

    • getEstimatedRequestedSegmentsUsage

      public int getEstimatedRequestedSegmentsUsage()

    • maybeLogUsageWarning

      public void maybeLogUsageWarning()

    • countBuffers

      public int countBuffers()

    • getAvailableFuture

      public CompletableFuture<?> getAvailableFuture()
      Returns a future that is completed when there are free segments in this pool.
      Specified by:
      getAvailableFuture in interface AvailabilityProvider
      Returns:
      a future that is completed if the respective provider is available.

    • createBufferPool

      public BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException
      Description copied from interface: BufferPoolFactory
      Tries to create a buffer pool, which is guaranteed to provide at least the number of required buffers.

      The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

      Specified by:
      createBufferPool in interface BufferPoolFactory
      Parameters:
      numRequiredBuffers - minimum number of network buffers in this pool
      maxUsedBuffers - maximum number of network buffers this pool offers
      Throws:
      IOException

    • createBufferPool

      public BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers, int numSubpartitions, int maxBuffersPerChannel, int maxOverdraftBuffersPerGate) throws IOException
      Description copied from interface: BufferPoolFactory
      Tries to create a buffer pool with an owner, which is guaranteed to provide at least the number of required buffers.

      The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

      Specified by:
      createBufferPool in interface BufferPoolFactory
      Parameters:
      numRequiredBuffers - minimum number of network buffers in this pool
      maxUsedBuffers - maximum number of network buffers this pool offers
      numSubpartitions - number of subpartitions in this pool
      maxBuffersPerChannel - maximum number of buffers to use for each channel
      maxOverdraftBuffersPerGate - maximum number of overdraft buffers to use for each gate
      Throws:
      IOException

    • destroyBufferPool

      public void destroyBufferPool(BufferPool bufferPool)
      Description copied from interface: BufferPoolFactory
      Destroy callback for updating factory book keeping.
      Specified by:
      destroyBufferPool in interface BufferPoolFactory

    • destroyAllBufferPools

      public void destroyAllBufferPools()
      Destroys all buffer pools that allocate their buffers from this buffer pool (created via createBufferPool(int, int)).