Package org.apache.flink.streaming.api.connector.sink2

Interface SupportsPreCommitTopology<WriterResultT,CommittableT>

@Experimental public interface SupportsPreCommitTopology<WriterResultT,CommittableT>
Allows expert users to implement a custom topology after SinkWriter and before Committer.

It is recommended to use immutable committables because mutating committables can have unexpected side-effects.

It's important that all CommittableMessages are modified appropriately, such that all messages with the same subtask id will also be processed by the same Committer subtask and the CommittableSummary matches the respective count. If committables are combined or split in any way, the summary needs to be adjusted.

There is also no requirement to keep the subtask ids of the writer, they can be changed as long as there are no two summaries with the same subtask ids (and corresponding CommittableWithLineage). Subtask ids don't need to be consecutive or small. The global committer will use CommittableSummary.getNumberOfSubtasks() to determine if all committables have been received, so that number needs to correctly reflect the number of distinct subtask ids. The easiest way to guarantee all of this is to use RuntimeContext.getTaskInfo().

  • Method Details

    • addPreCommitTopology

      DataStream<CommittableMessage<CommittableT>> addPreCommitTopology(DataStream<CommittableMessage<WriterResultT>> committables)
      Intercepts and modifies the committables sent on checkpoint or at end of input. Implementers need to ensure to modify all CommittableMessages appropriately.
      Parameters:
      committables - the stream of committables.
      Returns:
      the custom topology before Committer.

    • getWriteResultSerializer

      org.apache.flink.core.io.SimpleVersionedSerializer<WriterResultT> getWriteResultSerializer()
      Returns the serializer of the WriteResult type.