Package org.apache.flink.state.forst.restore

Class ForStIncrementalRestoreOperation<K>

java.lang.Object
org.apache.flink.state.forst.restore.ForStIncrementalRestoreOperation<K>
All Implemented Interfaces:
AutoCloseable, org.apache.flink.runtime.state.RestoreOperation<ForStRestoreResult>, ForStRestoreOperation
public class ForStIncrementalRestoreOperation<K> extends Object implements ForStRestoreOperation
Encapsulates the process of restoring a ForSt instance from an incremental snapshot.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    ForStIncrementalRestoreOperation.RestoredDBInstance
    Entity to hold the temporary RocksDB instance created for restore.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ForStIncrementalRestoreOperation(String operatorIdentifier, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, ClassLoader userCodeClassLoader, Map<String,ForStOperationUtils.ForStKvStateInfo> kvStateInformation, org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider, ForStResourceContainer optionsContainer, org.apache.flink.core.fs.Path forstBasePath, org.apache.flink.core.fs.Path instanceRocksDBPath, org.forstdb.DBOptions dbOptions, Function<String,org.forstdb.ColumnFamilyOptions> columnFamilyOptionsFactory, ForStNativeMetricOptions nativeMetricOptions, org.apache.flink.metrics.MetricGroup metricGroup, ForStDBTtlCompactFiltersManager ttlCompactFiltersManager, long writeBatchSize, Long writeBufferManagerCapacity, org.apache.flink.runtime.state.StateBackend.CustomInitializationMetrics customInitializationMetrics, Collection<org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle> restoreStateHandles, double overlapFractionThreshold, boolean useIngestDbRestoreMode, boolean useDeleteFilesInRange, org.apache.flink.core.execution.RecoveryClaimMode recoveryClaimMode)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    void
    exportColumnFamilies(org.forstdb.RocksDB db, List<org.forstdb.ColumnFamilyHandle> columnFamilyHandles, List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases, org.apache.flink.core.fs.Path exportBasePath, Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.forstdb.ExportImportFilesMetaData>> resultOutput)
    Exports the data of the given column families in the given DB.
    void
    mergeStateHandlesWithClipAndIngest(List<StateHandleTransferSpec> keyedStateHandles, byte[] startKeyGroupPrefixBytes, byte[] stopKeyGroupPrefixBytes)
    Restores the base DB by merging multiple state handles into one.
    ForStRestoreResult
    restore()
    Root method that branches for different implementations of IncrementalKeyedStateHandle.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ForStIncrementalRestoreOperation

      public ForStIncrementalRestoreOperation(String operatorIdentifier, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, ClassLoader userCodeClassLoader, Map<String,ForStOperationUtils.ForStKvStateInfo> kvStateInformation, org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider, ForStResourceContainer optionsContainer, org.apache.flink.core.fs.Path forstBasePath, org.apache.flink.core.fs.Path instanceRocksDBPath, org.forstdb.DBOptions dbOptions, Function<String,org.forstdb.ColumnFamilyOptions> columnFamilyOptionsFactory, ForStNativeMetricOptions nativeMetricOptions, org.apache.flink.metrics.MetricGroup metricGroup, @Nonnull ForStDBTtlCompactFiltersManager ttlCompactFiltersManager, @Nonnegative long writeBatchSize, Long writeBufferManagerCapacity, org.apache.flink.runtime.state.StateBackend.CustomInitializationMetrics customInitializationMetrics, @Nonnull Collection<org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle> restoreStateHandles, double overlapFractionThreshold, boolean useIngestDbRestoreMode, boolean useDeleteFilesInRange, org.apache.flink.core.execution.RecoveryClaimMode recoveryClaimMode)

  • Method Details

    • restore

      public ForStRestoreResult restore() throws Exception
      Root method that branches for different implementations of IncrementalKeyedStateHandle.
      Specified by:
      restore in interface ForStRestoreOperation
      Specified by:
      restore in interface org.apache.flink.runtime.state.RestoreOperation<K>
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface AutoCloseable
      Throws:
      Exception

    • mergeStateHandlesWithClipAndIngest

      public void mergeStateHandlesWithClipAndIngest(List<StateHandleTransferSpec> keyedStateHandles, byte[] startKeyGroupPrefixBytes, byte[] stopKeyGroupPrefixBytes) throws Exception
      Restores the base DB by merging multiple state handles into one. This method first checks if all data to import is in the expected key-groups range and then uses import/export. Otherwise, this method falls back to copying the data using a temporary DB.
      Parameters:
      keyedStateHandles - the list of state handles to restore the base DB from.
      startKeyGroupPrefixBytes - the min/start key of the key groups range as bytes.
      stopKeyGroupPrefixBytes - the max+1/end key of the key groups range as bytes.
      Throws:
      Exception - on any restore error.

    • exportColumnFamilies

      public void exportColumnFamilies(org.forstdb.RocksDB db, List<org.forstdb.ColumnFamilyHandle> columnFamilyHandles, List<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase> registeredStateMetaInfoBases, org.apache.flink.core.fs.Path exportBasePath, Map<org.apache.flink.runtime.state.RegisteredStateMetaInfoBase.Key,List<org.forstdb.ExportImportFilesMetaData>> resultOutput) throws org.forstdb.RocksDBException, IOException
      Exports the data of the given column families in the given DB.
      Parameters:
      db - the DB to export from.
      columnFamilyHandles - the column families to export.
      registeredStateMetaInfoBases - meta information about the registered states in the DB.
      exportBasePath - the path to which the export files go.
      resultOutput - output parameter for the metadata of the export.
      Throws:
      org.forstdb.RocksDBException - on problems inside RocksDB.
      IOException