Package org.apache.flink.streaming.api.functions.sink.legacy

Class TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>

java.lang.Object
org.apache.flink.api.common.functions.AbstractRichFunction
org.apache.flink.streaming.api.functions.sink.legacy.RichSinkFunction<IN>
org.apache.flink.streaming.api.functions.sink.legacy.TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>
Type Parameters:
IN - Input type for SinkFunction.
TXN - Transaction to store all of the information required to handle a transaction.
CONTEXT - Context that will be shared across all invocations for the given TwoPhaseCommitSinkFunction instance. Context is created once
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN>
@Internal public abstract class TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> extends org.apache.flink.streaming.api.functions.sink.legacy.RichSinkFunction<IN> implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.api.common.state.CheckpointListener
Deprecated.
This interface will be removed in future versions. Use the new Sink interface instead.
This is a recommended base class for all of the SinkFunction that intend to implement exactly-once semantic. It does that by implementing two phase commit algorithm on top of the CheckpointedFunction and CheckpointListener. User should provide custom TXN (transaction handle) and implement abstract methods handling this transaction handle.
See Also:

  • Field Details

  • Constructor Details

    • TwoPhaseCommitSinkFunction

      public TwoPhaseCommitSinkFunction(org.apache.flink.api.common.typeutils.TypeSerializer<TXN> transactionSerializer, org.apache.flink.api.common.typeutils.TypeSerializer<CONTEXT> contextSerializer)
      Deprecated.
      Use default ListStateDescriptor for internal state serialization. Helpful utilities for using this constructor are TypeInformation.of(Class), TypeHint and TypeInformation.of(TypeHint). Example: 
      
 TwoPhaseCommitSinkFunction(TypeInformation.of(new TypeHint<State<TXN, CONTEXT>>() {}));
      Parameters:
      transactionSerializer - TypeSerializer for the transaction type of this sink
      contextSerializer - TypeSerializer for the context type of this sink

  • Method Details

    • initializeUserContext

      protected Optional<CONTEXT> initializeUserContext()
      Deprecated.

    • getUserContext

      protected Optional<CONTEXT> getUserContext()
      Deprecated.

    • currentTransaction

      @Nullable protected TXN currentTransaction()
      Deprecated.

    • pendingTransactions

      @Nonnull protected Stream<Map.Entry<Long,TXN>> pendingTransactions()
      Deprecated.

    • invoke

      protected abstract void invoke(TXN transaction, IN value, org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction.Context context) throws Exception
      Deprecated.
      Write value within a transaction.
      Throws:
      Exception

    • beginTransaction

      protected abstract TXN beginTransaction() throws Exception
      Deprecated.
      Method that starts a new transaction.
      Returns:
      newly created transaction.
      Throws:
      Exception

    • preCommit

      protected abstract void preCommit(TXN transaction) throws Exception
      Deprecated.
      Pre commit previously created transaction. Pre commit must make all of the necessary steps to prepare the transaction for a commit that might happen in the future. After this point the transaction might still be aborted, but underlying implementation must ensure that commit calls on already pre committed transactions will always succeed.

      Usually implementation involves flushing the data.

      Throws:
      Exception

    • commit

      protected abstract void commit(TXN transaction)
      Deprecated.
      Commit a pre-committed transaction. If this method fail, Flink application will be restarted and recoverAndCommit(Object) will be called again for the same transaction.

    • recoverAndCommit

      protected void recoverAndCommit(TXN transaction)
      Deprecated.
      Invoked on recovered transactions after a failure. User implementation must ensure that this call will eventually succeed. If it fails, Flink application will be restarted and it will be invoked again. If it does not succeed eventually, a data loss will occur. Transactions will be recovered in an order in which they were created.

    • abort

      protected abstract void abort(TXN transaction)
      Deprecated.
      Abort a transaction.

    • recoverAndAbort

      protected void recoverAndAbort(TXN transaction)
      Deprecated.
      Abort a transaction that was rejected by a coordinator after a failure.

    • finishRecoveringContext

      protected void finishRecoveringContext(Collection<TXN> handledTransactions)
      Deprecated.
      Callback for subclasses which is called after restoring (each) user context.
      Parameters:
      handledTransactions - transactions which were already committed or aborted and do not need further handling

    • finishProcessing

      protected void finishProcessing(@Nullable TXN transaction)
      Deprecated.
      This method is called at the end of data processing.

      The method is expected to flush all remaining buffered data. Exceptions will cause the pipeline to be recognized as failed, because the last data items are not processed properly. You may use this method to flush remaining buffered elements in the state into the current transaction which will be committed in the last checkpoint.

    • invoke

      public final void invoke(IN value) throws Exception
      Deprecated.
      This should not be implemented by subclasses.
      Specified by:
      invoke in interface org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN>
      Throws:
      Exception

    • invoke

      public final void invoke(IN value, org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction.Context context) throws Exception
      Deprecated.
      Specified by:
      invoke in interface org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN>
      Throws:
      Exception

    • finish

      public final void finish() throws Exception
      Deprecated.
      Specified by:
      finish in interface org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN>
      Throws:
      Exception

    • notifyCheckpointComplete

      public final void notifyCheckpointComplete(long checkpointId) throws Exception
      Deprecated.
      Specified by:
      notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
      Throws:
      Exception

    • notifyCheckpointAborted

      public void notifyCheckpointAborted(long checkpointId)
      Deprecated.
      Specified by:
      notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) throws Exception
      Deprecated.
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) throws Exception
      Deprecated.
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • close

      public void close() throws Exception
      Deprecated.
      Specified by:
      close in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      close in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      Exception

    • setTransactionTimeout

      protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> setTransactionTimeout(long transactionTimeout)
      Deprecated.
      Sets the transaction timeout. Setting only the transaction timeout has no effect in itself.
      Parameters:
      transactionTimeout - The transaction timeout in ms.
      See Also:

    • ignoreFailuresAfterTransactionTimeout

      protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> ignoreFailuresAfterTransactionTimeout()
      Deprecated.
      If called, the sink will only log but not propagate exceptions thrown in recoverAndCommit(Object) if the transaction is older than a specified transaction timeout. The start time of an transaction is determined by System.currentTimeMillis(). By default, failures are propagated.

    • enableTransactionTimeoutWarnings

      protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> enableTransactionTimeoutWarnings(double warningRatio)
      Deprecated.
      Enables logging of warnings if a transaction's elapsed time reaches a specified ratio of the transactionTimeout. If warningRatio is 0, a warning will be always logged when committing the transaction.
      Parameters:
      warningRatio - A value in the range [0,1].
      Returns: