Package org.apache.flink.changelog.fs

Class DuplicatingStateChangeFsUploader

java.lang.Object

org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

org.apache.flink.changelog.fs.DuplicatingStateChangeFsUploader

All Implemented Interfaces:

AutoCloseable, StateChangeUploader

public class DuplicatingStateChangeFsUploader
extends AbstractStateChangeFsUploader

A StateChangeFsUploader implementation that writes the changes to remote and local.

Local dstl files are only managed by TM side, LocalChangelogRegistry, TaskChangelogRegistry and ChangelogTaskLocalStateStore are responsible for managing them.

Remote dstl files are managed by TM side and JM side, TaskChangelogRegistry is responsible for TM side, and SharedStateRegistry is responsible for JM side.

The total discard logic of local dstl files is:

1. Register files to TaskChangelogRegistry.startTracking(org.apache.flink.runtime.state.StreamStateHandle, long) on AbstractStateChangeFsUploader.upload(java.util.Collection<org.apache.flink.changelog.fs.StateChangeUploadScheduler.UploadTask>).
2. Store the meta of files into ChangelogTaskLocalStateStore by AsyncCheckpointRunnable#reportCompletedSnapshotStates().
3. Pass control of the file to LocalChangelogRegistry.register(org.apache.flink.runtime.state.StreamStateHandle, long) when FsStateChangelogWriter#persist , files of the previous checkpoint will be deleted by LocalChangelogRegistry.discardUpToCheckpoint(long) when the checkpoint is confirmed.
4. When ChangelogTruncateHelper#materialized() or ChangelogTruncateHelper#checkpointSubsumed() is called, TaskChangelogRegistry.release(org.apache.flink.runtime.state.StreamStateHandle) is responsible for deleting local files.
5. When one checkpoint is aborted, all accumulated local dstl files will be deleted at once.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader

StateChangeUploader.UploadTasksResult

Field Summary

Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

bufferSize, compression, metrics

Constructor Summary

Constructors

Constructor	Description
DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID, org.apache.flink.core.fs.Path basePath, org.apache.flink.core.fs.FileSystem fileSystem, boolean compression, int bufferSize, ChangelogStorageMetricGroup metrics, TaskChangelogRegistry changelogRegistry, org.apache.flink.runtime.state.LocalSnapshotDirectoryProvider localSnapshotDirectoryProvider)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.flink.changelog.fs.OutputStreamWithPos	prepareStream()

Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

generateFileName, upload

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DuplicatingStateChangeFsUploader

public DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID,
 org.apache.flink.core.fs.Path basePath,
 org.apache.flink.core.fs.FileSystem fileSystem,
 boolean compression,
 int bufferSize,
 ChangelogStorageMetricGroup metrics,
 TaskChangelogRegistry changelogRegistry,
 org.apache.flink.runtime.state.LocalSnapshotDirectoryProvider localSnapshotDirectoryProvider)

Method Details

prepareStream

public org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
                                                                throws IOException

Throws:

IOException

close

public void close()
           throws Exception

Throws:

Exception