Package org.apache.flink.state.rocksdb

Class RocksDBIncrementalCheckpointUtils

java.lang.Object
org.apache.flink.state.rocksdb.RocksDBIncrementalCheckpointUtils
public class RocksDBIncrementalCheckpointUtils extends Object
Utils for RocksDB Incremental Checkpoint.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final class 
    RocksDBIncrementalCheckpointUtils.RangeCheckResult
    Helper class that represents the result of a range check of the actual keys in a RocksDB instance against the proclaimed key-group range of the instance.

  • Constructor Summary

    Constructors
    Constructor
    Description
    RocksDBIncrementalCheckpointUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    beforeThePrefixBytes(byte[] bytes, byte[] prefixBytes)
    check whether the bytes is before prefixBytes in the character order.
    static RocksDBIncrementalCheckpointUtils.RangeCheckResult
    checkSstDataAgainstKeyGroupRange(org.rocksdb.RocksDB db, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
    Checks data in the SST files of the given DB for keys that exceed either the lower and upper bound of the proclaimed key-groups range of the DB.
    static <T extends org.apache.flink.runtime.state.KeyedStateHandle>
    T
    chooseTheBestStateHandleForInitial(List<T> restoreStateHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
    Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db.
    static void
    clipDBWithKeyGroupRange(org.rocksdb.RocksDB db, List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, org.apache.flink.runtime.state.KeyGroupRange currentKeyGroupRange, int keyGroupPrefixBytes, boolean useDeleteFilesInRange)
    The method to clip the db instance according to the target key group range using the RocksDB.delete(ColumnFamilyHandle, byte[]).
    static org.apache.flink.util.function.RunnableWithException
    createAsyncRangeCompactionTask(org.rocksdb.RocksDB db, Collection<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange, org.apache.flink.util.ResourceGuard rocksDBResourceGuard, org.apache.flink.core.fs.CloseableRegistry closeableRegistry)
    Returns a range compaction task as runnable if any data in the SST files of the given DB exceeds the proclaimed key-group range.
    static void
    exportColumnFamilies(org.rocksdb.RocksDB db, List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases, Path exportBasePath, Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.rocksdb.ExportImportFilesMetaData>> resultOutput)
    Exports the data of the given column families in the given DB.
    static <T extends org.apache.flink.runtime.state.KeyedStateHandle>
    int
    findTheBestStateHandleForInitial(List<T> restoreStateHandles, org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
    Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db from the given lists and returns its index.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • RocksDBIncrementalCheckpointUtils

      public RocksDBIncrementalCheckpointUtils()

  • Method Details

    • clipDBWithKeyGroupRange

      public static void clipDBWithKeyGroupRange(@Nonnull org.rocksdb.RocksDB db, @Nonnull List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, @Nonnull org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, @Nonnull org.apache.flink.runtime.state.KeyGroupRange currentKeyGroupRange, @Nonnegative int keyGroupPrefixBytes, boolean useDeleteFilesInRange) throws org.rocksdb.RocksDBException
      The method to clip the db instance according to the target key group range using the RocksDB.delete(ColumnFamilyHandle, byte[]).
      Parameters:
      db - the RocksDB instance to be clipped.
      columnFamilyHandles - the column families in the db instance.
      targetKeyGroupRange - the target key group range.
      currentKeyGroupRange - the key group range of the db instance.
      keyGroupPrefixBytes - Number of bytes required to prefix the key groups.
      useDeleteFilesInRange - whether to call db.deleteFilesInRanges for the deleted ranges.
      Throws:
      org.rocksdb.RocksDBException

    • createAsyncRangeCompactionTask

      public static org.apache.flink.util.function.RunnableWithException createAsyncRangeCompactionTask(org.rocksdb.RocksDB db, Collection<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange, org.apache.flink.util.ResourceGuard rocksDBResourceGuard, org.apache.flink.core.fs.CloseableRegistry closeableRegistry)
      Returns a range compaction task as runnable if any data in the SST files of the given DB exceeds the proclaimed key-group range.
      Parameters:
      db - the DB to check and compact if needed.
      columnFamilyHandles - list of column families to check.
      keyGroupPrefixBytes - the number of bytes used to serialize the key-group prefix of keys in the DB.
      dbExpectedKeyGroupRange - the expected key-groups range of the DB.
      rocksDBResourceGuard - the resource guard for the given db instance.
      Returns:
      runnable that performs compaction upon execution if the key-groups range is exceeded. Otherwise, empty optional is returned.

    • checkSstDataAgainstKeyGroupRange

      public static RocksDBIncrementalCheckpointUtils.RangeCheckResult checkSstDataAgainstKeyGroupRange(org.rocksdb.RocksDB db, int keyGroupPrefixBytes, org.apache.flink.runtime.state.KeyGroupRange dbExpectedKeyGroupRange)
      Checks data in the SST files of the given DB for keys that exceed either the lower and upper bound of the proclaimed key-groups range of the DB.
      Parameters:
      db - the DB to check.
      keyGroupPrefixBytes - the number of bytes used to serialize the key-group prefix of keys in the DB.
      dbExpectedKeyGroupRange - the expected key-groups range of the DB.
      Returns:
      the check result with detailed info about lower and upper bound violations.

    • exportColumnFamilies

      public static void exportColumnFamilies(org.rocksdb.RocksDB db, List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles, List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases, Path exportBasePath, Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.rocksdb.ExportImportFilesMetaData>> resultOutput) throws org.rocksdb.RocksDBException
      Exports the data of the given column families in the given DB.
      Parameters:
      db - the DB to export from.
      columnFamilyHandles - the column families to export.
      registeredStateMetaInfoBases - meta information about the registered states in the DB.
      exportBasePath - the path to which the export files go.
      resultOutput - output parameter for the metadata of the export.
      Throws:
      org.rocksdb.RocksDBException - on problems inside RocksDB.

    • beforeThePrefixBytes

      public static boolean beforeThePrefixBytes(@Nonnull byte[] bytes, @Nonnull byte[] prefixBytes)
      check whether the bytes is before prefixBytes in the character order.

    • chooseTheBestStateHandleForInitial

      @Nullable public static <T extends org.apache.flink.runtime.state.KeyedStateHandle> T chooseTheBestStateHandleForInitial(@Nonnull List<T> restoreStateHandles, @Nonnull org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
      Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db.
      Type Parameters:
      T - the generic parameter type of the state handles.
      Parameters:
      restoreStateHandles - The candidate state handles.
      targetKeyGroupRange - The target key group range.
      overlapFractionThreshold - configured threshold for overlap.
      Returns:
      The best candidate or null if no candidate was a good fit.

    • findTheBestStateHandleForInitial

      public static <T extends org.apache.flink.runtime.state.KeyedStateHandle> int findTheBestStateHandleForInitial(@Nonnull List<T> restoreStateHandles, @Nonnull org.apache.flink.runtime.state.KeyGroupRange targetKeyGroupRange, double overlapFractionThreshold)
      Choose the best state handle according to the stateHandleEvaluator(KeyedStateHandle, KeyGroupRange, double) to init the initial db from the given lists and returns its index.
      Type Parameters:
      T - the generic parameter type of the state handles.
      Parameters:
      restoreStateHandles - The candidate state handles.
      targetKeyGroupRange - The target key group range.
      overlapFractionThreshold - configured threshold for overlap.
      Returns:
      the index of the best candidate handle in the list or -1 if the list was empty.