Package org.apache.flink.runtime.io.disk.iomanager

Class IOManagerAsync

java.lang.Object
org.apache.flink.runtime.io.disk.iomanager.IOManager
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync
All Implemented Interfaces:
AutoCloseable, Thread.UncaughtExceptionHandler
public class IOManagerAsync extends IOManager implements Thread.UncaughtExceptionHandler
A version of the IOManager that uses asynchronous I/O.

  • Constructor Details

    • IOManagerAsync

      @VisibleForTesting public IOManagerAsync()
      Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.

    • IOManagerAsync

      @VisibleForTesting public IOManagerAsync(ExecutorService executorService)
      Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.

    • IOManagerAsync

      @VisibleForTesting public IOManagerAsync(String tempDir)
      Constructs a new asynchronous I/O manager, writing file to the given directory.
      Parameters:
      tempDir - The directory to write temporary files to.

    • IOManagerAsync

      @VisibleForTesting public IOManagerAsync(String tempDir, ExecutorService executorService)
      Constructs a new asynchronous I/O manager, writing file to the given directory.
      Parameters:
      tempDir - The directory to write temporary files to.

    • IOManagerAsync

      public IOManagerAsync(String[] tempDirs, ExecutorService executorService)
      Constructs a new asynchronous I/O manager, writing file round robin across the given directories.
      Parameters:
      tempDirs - The directories to write temporary files to.

  • Method Details

    • close

      public void close() throws Exception
      Close method. Shuts down the reader and writer threads immediately, not waiting for their pending requests to be served. This method waits until the threads have actually ceased their operation.
      Specified by:
      close in interface AutoCloseable
      Overrides:
      close in class IOManager
      Throws:
      Exception

    • uncaughtException

      public void uncaughtException(Thread t, Throwable e)
      Specified by:
      uncaughtException in interface Thread.UncaughtExceptionHandler

    • createBlockChannelWriter

      public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue) throws IOException
      Description copied from class: IOManager
      Creates a block channel writer that writes to the given channel. The writer adds the written segment to the given queue (to allow for asynchronous implementations).
      Specified by:
      createBlockChannelWriter in class IOManager
      Parameters:
      channelID - The descriptor for the channel to write to.
      returnQueue - The queue to put the written buffers into.
      Returns:
      A block channel writer that writes to the given channel.
      Throws:
      IOException - Thrown, if the channel for the writer could not be opened.

    • createBlockChannelWriter

      public BlockChannelWriterWithCallback<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID, RequestDoneCallback<org.apache.flink.core.memory.MemorySegment> callback) throws IOException
      Description copied from class: IOManager
      Creates a block channel writer that writes to the given channel. The writer calls the given callback after the I/O operation has been performed (successfully or unsuccessfully), to allow for asynchronous implementations.
      Specified by:
      createBlockChannelWriter in class IOManager
      Parameters:
      channelID - The descriptor for the channel to write to.
      callback - The callback to be called for
      Returns:
      A block channel writer that writes to the given channel.
      Throws:
      IOException - Thrown, if the channel for the writer could not be opened.

    • createBlockChannelReader

      public BlockChannelReader<org.apache.flink.core.memory.MemorySegment> createBlockChannelReader(FileIOChannel.ID channelID, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue) throws IOException
      Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the given queue.
      Specified by:
      createBlockChannelReader in class IOManager
      Parameters:
      channelID - The descriptor for the channel to write to.
      returnQueue - The queue to put the full buffers into.
      Returns:
      A block channel reader that reads from the given channel.
      Throws:
      IOException - Thrown, if the channel for the reader could not be opened.

    • createBufferFileWriter

      public BufferFileWriter createBufferFileWriter(FileIOChannel.ID channelID) throws IOException
      Specified by:
      createBufferFileWriter in class IOManager
      Throws:
      IOException

    • createBufferFileReader

      public BufferFileReader createBufferFileReader(FileIOChannel.ID channelID, RequestDoneCallback<Buffer> callback) throws IOException
      Specified by:
      createBufferFileReader in class IOManager
      Throws:
      IOException

    • createBufferFileSegmentReader

      public BufferFileSegmentReader createBufferFileSegmentReader(FileIOChannel.ID channelID, RequestDoneCallback<FileSegment> callback) throws IOException
      Specified by:
      createBufferFileSegmentReader in class IOManager
      Throws:
      IOException

    • createBulkBlockChannelReader

      public BulkBlockChannelReader createBulkBlockChannelReader(FileIOChannel.ID channelID, List<org.apache.flink.core.memory.MemorySegment> targetSegments, int numBlocks) throws IOException
      Creates a block channel reader that reads all blocks from the given channel directly in one bulk. The reader draws segments to read the blocks into from a supplied list, which must contain as many segments as the channel has blocks. After the reader is done, the list with the full segments can be obtained from the reader.

      If a channel is not to be read in one bulk, but in multiple smaller batches, a BlockChannelReader should be used.

      Specified by:
      createBulkBlockChannelReader in class IOManager
      Parameters:
      channelID - The descriptor for the channel to write to.
      targetSegments - The list to take the segments from into which to read the data.
      numBlocks - The number of blocks in the channel to read.
      Returns:
      A block channel reader that reads from the given channel.
      Throws:
      IOException - Thrown, if the channel for the reader could not be opened.