Package org.apache.flink.state.forst.fs.cache

Class FileBasedCache

java.lang.Object
org.apache.flink.state.forst.fs.cache.DoubleListLru<String,FileCacheEntry>
org.apache.flink.state.forst.fs.cache.FileBasedCache
All Implemented Interfaces:
Closeable, AutoCloseable, Iterable<org.apache.flink.api.java.tuple.Tuple2<String,FileCacheEntry>>
public final class FileBasedCache extends DoubleListLru<String,FileCacheEntry> implements Closeable
A file-granularity LRU cache. Only newly generated SSTs are written to the cache, the file reading from the remote will not. Newly generated SSTs are written to the original file system and cache simultaneously, so, the cached file can be directly deleted with persisting when evicting. The FileBasedCache, FileCacheEntry, CachedDataInputStream, and CachedDataOutputStream classes work together to implement a file-based caching mechanism in ForSt State Backend.
  • FileBasedCache manages multiple FileCacheEntry instances.
  • Each FileCacheEntry represents a cached file and can open CachedDataInputStream for reading the file.
  • CachedDataInputStream instances are created by FileCacheEntry and can read data from either the cached file or the original file, depending on the cache entry's state. It has internal stream status to indicate the current reading source.
  • CachedDataOutputStream instances are created by FileBasedCache and write data to both the original and cached files, creating a new cache entry in the cache when the writing is finished.

    • Constructor Details

      • FileBasedCache

        public FileBasedCache(org.apache.flink.configuration.ReadableConfig configuration, CacheLimitPolicy cacheLimitPolicy, org.apache.flink.core.fs.FileSystem cacheFs, org.apache.flink.core.fs.Path basePath, org.apache.flink.metrics.MetricGroup metricGroup)

    • Method Details

      • setFlinkThread

        public static void setFlinkThread()
        Sets the current thread as a Flink thread. This method is used to mark the thread as a Flink thread, which can be used to determine whether the file access would affect the LRU cache order, or metrics updates.

      • isFlinkThread

        public static boolean isFlinkThread()
        Checks if the current thread is a Flink thread. This method returns a boolean indicating whether the current thread has been marked as a Flink thread using the setFlinkThread() method.
        Returns:
        true if the current thread is a Flink thread, false otherwise.

      • incHitCounter

        public void incHitCounter()

      • incMissCounter

        public void incMissCounter()

      • open

        public CachedDataInputStream open(org.apache.flink.core.fs.Path path, org.apache.flink.core.fs.FSDataInputStream originalStream) throws IOException
        Throws:
        IOException

      • create

        public CachedDataOutputStream create(org.apache.flink.core.fs.FSDataOutputStream originalOutputStream, org.apache.flink.core.fs.Path path) throws IOException
        Throws:
        IOException

      • delete

        public void delete(org.apache.flink.core.fs.Path path)

      • get

        public FileCacheEntry get(String key, boolean affectOrder)
        Description copied from class: DoubleListLru
        Retrieves the value associated with the specified key. Optionally affects the order of the entries in the cache.
        Overrides:
        get in class DoubleListLru<String,FileCacheEntry>
        Parameters:
        key - the key of the entry
        affectOrder - true if the order of the entries should be affected, false otherwise
        Returns:
        the value associated with the key, or null if the key is not found

      • addFirst

        public void addFirst(String key, FileCacheEntry value)
        Description copied from class: DoubleListLru
        Adds a new entry to the front of the cache.
        Overrides:
        addFirst in class DoubleListLru<String,FileCacheEntry>
        Parameters:
        key - the key of the entry
        value - the value of the entry

      • addSecond

        public void addSecond(String key, FileCacheEntry value)
        Description copied from class: DoubleListLru
        Inserts a new entry at the middle of the cache.
        Overrides:
        addSecond in class DoubleListLru<String,FileCacheEntry>
        Parameters:
        key - the key of the entry
        value - the value of the entry

      • remove

        public FileCacheEntry remove(String key)
        Description copied from class: DoubleListLru
        Removes the entry associated with the specified key from the cache.
        Overrides:
        remove in class DoubleListLru<String,FileCacheEntry>
        Parameters:
        key - the key of the entry to be removed
        Returns:
        the value of the removed entry, or null if the key is not found

      • registerInCache

        public void registerInCache(org.apache.flink.core.fs.Path originalPath, long size)
        Directly insert in cache when restoring.

      • close

        public void close() throws IOException
        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface Closeable
        Throws:
        IOException