Package org.apache.flink.runtime.blob

Class AbstractBlobCache

java.lang.Object
org.apache.flink.runtime.blob.AbstractBlobCache
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
PermanentBlobCache, TransientBlobCache
public abstract class AbstractBlobCache extends Object implements Closeable
Abstract base class for permanent and transient BLOB files.

  • Field Details

    • log

      protected final org.slf4j.Logger log
      The log object used for debugging.

    • tempFileCounter

      protected final AtomicLong tempFileCounter
      Counter to generate unique names for temporary files.

    • storageDir

      protected final org.apache.flink.util.Reference<File> storageDir
      Root directory for local file storage.

    • blobView

      protected final BlobView blobView
      Blob store for distributed file storage, e.g. in HA.

    • shutdownRequested

      protected final AtomicBoolean shutdownRequested

    • shutdownHook

      protected final Thread shutdownHook
      Shutdown hook thread to ensure deletion of the local storage directory.

    • numFetchRetries

      protected final int numFetchRetries
      The number of retries when the transfer fails.

    • blobClientConfig

      protected final org.apache.flink.configuration.Configuration blobClientConfig
      Configuration for the blob client like ssl parameters required to connect to the blob server.

    • readWriteLock

      protected final ReadWriteLock readWriteLock
      Lock guarding concurrent file accesses.

    • serverAddress

      @Nullable protected volatile InetSocketAddress serverAddress

  • Constructor Details

    • AbstractBlobCache

      public AbstractBlobCache(org.apache.flink.configuration.Configuration blobClientConfig, org.apache.flink.util.Reference<File> storageDir, BlobView blobView, org.slf4j.Logger logger, @Nullable InetSocketAddress serverAddress) throws IOException
      Throws:
      IOException

  • Method Details

    • getStorageDir

      public File getStorageDir()

    • getFileInternal

      protected File getFileInternal(@Nullable org.apache.flink.api.common.JobID jobId, BlobKey blobKey) throws IOException
      Returns local copy of the file for the BLOB with the given key.

      The method will first attempt to serve the BLOB from its local cache. If the BLOB is not in the cache, the method will try to download it from this cache's BLOB server via a distributed BLOB store (if available) or direct end-to-end download.

      Parameters:
      jobId - ID of the job this blob belongs to (or null if job-unrelated)
      blobKey - The key of the desired BLOB.
      Returns:
      file referring to the local storage location of the BLOB.
      Throws:
      IOException - Thrown if an I/O error occurs while downloading the BLOBs from the BLOB server.

    • getPort

      public int getPort()
      Returns the port the BLOB server is listening on.
      Returns:
      BLOB server port or -1 if no server address

    • setBlobServerAddress

      public void setBlobServerAddress(InetSocketAddress blobServerAddress)
      Sets the address of the BlobServer.
      Parameters:
      blobServerAddress - address of the BlobServer.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • cancelCleanupTask

      protected abstract void cancelCleanupTask()
      Cancels any cleanup task that subclasses may be executing.

      This is called during close().