Package org.apache.flink.runtime.state.filesystem

Class FsCheckpointStorageAccess

java.lang.Object
org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorageAccess
org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess
All Implemented Interfaces:
CheckpointStorageAccess, CheckpointStorageCoordinatorView, CheckpointStorageWorkerView
Direct Known Subclasses:
FsMergingCheckpointStorageAccess
public class FsCheckpointStorageAccess extends AbstractFsCheckpointStorageAccess
An implementation of durable checkpoint storage to file systems.

  • Field Details

    • fileSystem

      protected final org.apache.flink.core.fs.FileSystem fileSystem

    • checkpointsDirectory

      protected final org.apache.flink.core.fs.Path checkpointsDirectory

    • sharedStateDirectory

      protected final org.apache.flink.core.fs.Path sharedStateDirectory

    • taskOwnedStateDirectory

      protected final org.apache.flink.core.fs.Path taskOwnedStateDirectory

    • fileSizeThreshold

      protected final int fileSizeThreshold

    • writeBufferSize

      protected final int writeBufferSize

  • Constructor Details

    • FsCheckpointStorageAccess

      public FsCheckpointStorageAccess(org.apache.flink.core.fs.Path checkpointBaseDirectory, @Nullable org.apache.flink.core.fs.Path defaultSavepointDirectory, org.apache.flink.api.common.JobID jobId, int fileSizeThreshold, int writeBufferSize) throws IOException
      Throws:
      IOException

    • FsCheckpointStorageAccess

      public FsCheckpointStorageAccess(org.apache.flink.core.fs.Path checkpointBaseDirectory, @Nullable org.apache.flink.core.fs.Path defaultSavepointDirectory, boolean createCheckpointSubDirs, org.apache.flink.api.common.JobID jobId, int fileSizeThreshold, int writeBufferSize) throws IOException
      Throws:
      IOException

    • FsCheckpointStorageAccess

      public FsCheckpointStorageAccess(org.apache.flink.core.fs.FileSystem fs, org.apache.flink.core.fs.Path checkpointBaseDirectory, @Nullable org.apache.flink.core.fs.Path defaultSavepointDirectory, boolean createCheckpointSubDirs, org.apache.flink.api.common.JobID jobId, int fileSizeThreshold, int writeBufferSize) throws IOException
      Throws:
      IOException

  • Method Details

    • getCheckpointsDirectory

      public org.apache.flink.core.fs.Path getCheckpointsDirectory()

    • getSharedStateDirectory

      public org.apache.flink.core.fs.Path getSharedStateDirectory()

    • getFileSystem

      public org.apache.flink.core.fs.FileSystem getFileSystem()

    • supportsHighlyAvailableStorage

      public boolean supportsHighlyAvailableStorage()
      Description copied from interface: CheckpointStorageCoordinatorView
      Checks whether this backend supports highly available storage of data.

      Some state backends may not support highly-available durable storage, with default settings, which makes them suitable for zero-config prototyping, but not for actual production setups.

    • initializeBaseLocationsForCheckpoint

      public void initializeBaseLocationsForCheckpoint() throws IOException
      Description copied from interface: CheckpointStorageCoordinatorView
      Initializes the necessary prerequisites for storage locations of checkpoints.

      For file-based checkpoint storage, this method would initialize essential base checkpoint directories on checkpoint coordinator side and should be executed before calling CheckpointStorageCoordinatorView.initializeLocationForCheckpoint(long).

      Throws:
      IOException - Thrown, if these base storage locations cannot be initialized due to an I/O exception.

    • initializeLocationForCheckpoint

      public CheckpointStorageLocation initializeLocationForCheckpoint(long checkpointId) throws IOException
      Description copied from interface: CheckpointStorageCoordinatorView
      Initializes a storage location for new checkpoint with the given ID.

      The returned storage location can be used to write the checkpoint data and metadata to and to obtain the pointers for the location(s) where the actual checkpoint data should be stored.

      Parameters:
      checkpointId - The ID (logical timestamp) of the checkpoint that should be persisted.
      Returns:
      A storage location for the data and metadata of the given checkpoint.
      Throws:
      IOException - Thrown if the storage location cannot be initialized due to an I/O exception.

    • resolveCheckpointStorageLocation

      public CheckpointStreamFactory resolveCheckpointStorageLocation(long checkpointId, CheckpointStorageLocationReference reference) throws IOException
      Description copied from interface: CheckpointStorageWorkerView
      Resolves a storage location reference into a CheckpointStreamFactory.

      The reference may be the default reference, in which case the method should return the default location, taking existing configuration and checkpoint ID into account.

      Parameters:
      checkpointId - The ID of the checkpoint that the location is initialized for.
      reference - The checkpoint location reference.
      Returns:
      A checkpoint storage location reflecting the reference and checkpoint ID.
      Throws:
      IOException - Thrown, if the storage location cannot be initialized from the reference.

    • createTaskOwnedStateStream

      public CheckpointStateOutputStream createTaskOwnedStateStream()
      Description copied from interface: CheckpointStorageWorkerView
      Opens a stream to persist checkpoint state data that is owned strictly by tasks and not attached to the life cycle of a specific checkpoint.

      This method should be used when the persisted data cannot be immediately dropped once the checkpoint that created it is dropped. Examples are write-ahead-logs. For those, the state can only be dropped once the data has been moved to the target system, which may sometimes take longer than one checkpoint (if the target system is temporarily unable to keep up).

      The fact that the job manager does not own the life cycle of this type of state means also that it is strictly the responsibility of the tasks to handle the cleanup of this data.

      Developer note: In the future, we may be able to make this a special case of "shared state", where the task re-emits the shared state reference as long as it needs to hold onto the persisted state data.

      Returns:
      A checkpoint state stream to the location for state owned by tasks.

    • createTaskOwnedCheckpointStateToolset

      public CheckpointStateToolset createTaskOwnedCheckpointStateToolset()
      Description copied from interface: CheckpointStorageWorkerView
      A complementary method to CheckpointStorageWorkerView.createTaskOwnedStateStream(). Creates a toolset that gives access to additional operations that can be performed in the task owned state location.
      Returns:
      A toolset for additional operations for state owned by tasks.

    • createSavepointLocation

      protected CheckpointStorageLocation createSavepointLocation(org.apache.flink.core.fs.FileSystem fs, org.apache.flink.core.fs.Path location)
      Specified by:
      createSavepointLocation in class AbstractFsCheckpointStorageAccess

    • toFileMergingStorage

      public FsMergingCheckpointStorageAccess toFileMergingStorage(FileMergingSnapshotManager mergingSnapshotManager, Environment environment) throws IOException
      Description copied from interface: CheckpointStorageWorkerView
      Return FsMergingCheckpointStorageAccess if file merging is enabled. Otherwise, return itself. File merging is supported by subclasses of AbstractFsCheckpointStorageAccess.
      Throws:
      IOException