Package org.apache.flink.runtime.asyncprocessing

Class AsyncExecutionController<K>

java.lang.Object
org.apache.flink.runtime.asyncprocessing.AsyncExecutionController<K>
Type Parameters:
K - the type of the key
All Implemented Interfaces:
Closeable, AutoCloseable, StateRequestHandler
public class AsyncExecutionController<K> extends Object implements StateRequestHandler, Closeable
The Async Execution Controller (AEC) receives processing requests from operators, and put them into execution according to some strategies.

It is responsible for:

  • Preserving the sequence of elements bearing the same key by delaying subsequent requests until the processing of preceding ones is finalized.
  • Tracking the in-flight data(records) and blocking the input if too much data in flight (back-pressure). It invokes MailboxExecutor.yield() to pause current operations, allowing for the execution of callbacks (mails in Mailbox).

    • Constructor Details

      • AsyncExecutionController

        public AsyncExecutionController(org.apache.flink.api.common.operators.MailboxExecutor mailboxExecutor, org.apache.flink.core.state.StateFutureImpl.AsyncFrameworkExceptionHandler exceptionHandler, StateExecutor stateExecutor, DeclarationManager declarationManager, int maxParallelism, int batchSize, long bufferTimeout, int maxInFlightRecords, @Nullable AsyncExecutionController.SwitchContextListener<K> switchContextListener, @Nullable org.apache.flink.metrics.MetricGroup metricGroup)

    • Method Details

      • buildContext

        public RecordContext<K> buildContext(Object record, K key)
        Build a new context based on record and key. Also wired with internal KeyAccountingUnit.
        Parameters:
        record - the given record.
        key - the given key.
        Returns:
        the built record context.

      • buildContext

        public RecordContext<K> buildContext(Object record, K key, boolean inherit)
        Build a new context based on record and key. Also wired with internal KeyAccountingUnit.
        Parameters:
        record - the given record.
        key - the given key.
        inherit - whether to inherit epoch and variables from the current context. Or otherwise create new ones.
        Returns:
        the built record context.

      • setCurrentContext

        public void setCurrentContext(RecordContext<K> switchingContext)
        Each time before a code segment (callback) is about to run in mailbox (task thread), this method should be called to switch a context in AEC.
        Parameters:
        switchingContext - the context to switch.

      • getCurrentContext

        public RecordContext<K> getCurrentContext()

      • handleRequest

        public <IN, OUT> org.apache.flink.core.state.InternalStateFuture<OUT> handleRequest(@Nullable org.apache.flink.api.common.state.v2.State state, StateRequestType type, @Nullable IN payload)
        Submit a StateRequest to this AsyncExecutionController and trigger it if needed.
        Specified by:
        handleRequest in interface StateRequestHandler
        Parameters:
        state - the state to request. Could be null if the type is StateRequestType.SYNC_POINT.
        type - the type of this request.
        payload - the payload input for this request.
        Returns:
        the state future.

      • handleRequest

        public <IN, OUT> org.apache.flink.core.state.InternalStateFuture<OUT> handleRequest(@Nullable org.apache.flink.api.common.state.v2.State state, StateRequestType type, boolean sync, @Nullable IN payload, boolean allowOverdraft)
        Submit a StateRequest to this AsyncExecutionController and trigger it if needed.
        Parameters:
        state - the state to request. Could be null if the type is StateRequestType.SYNC_POINT.
        type - the type of this request.
        sync - whether to trigger the request synchronously once it's ready.
        payload - the payload input for this request.
        allowOverdraft - whether to allow overdraft.
        Returns:
        the state future.

      • handleRequestSync

        public <IN, OUT> OUT handleRequestSync(org.apache.flink.api.common.state.v2.State state, StateRequestType type, @Nullable IN payload)
        Description copied from interface: StateRequestHandler
        Submit a StateRequest to this StateRequestHandler, and wait for the response synchronously.
        Specified by:
        handleRequestSync in interface StateRequestHandler
        Parameters:
        state - the state to request.
        type - the type of this request.
        payload - the payload input for this request.
        Returns:
        the state future.

      • setCurrentNamespaceForState

        public <N> void setCurrentNamespaceForState(@Nonnull InternalPartitionedState<N> state, N namespace)
        Description copied from interface: StateRequestHandler
        Set current namespace for a state. See InternalPartitionedState.setCurrentNamespace(Object).
        Specified by:
        setCurrentNamespaceForState in interface StateRequestHandler

      • triggerIfNeeded

        public boolean triggerIfNeeded(boolean force)
        Trigger a batch of requests.
        Parameters:
        force - whether to trigger requests in force.

      • syncPointRequestWithCallback

        public org.apache.flink.api.common.state.v2.StateFuture<Void> syncPointRequestWithCallback(org.apache.flink.util.function.ThrowingRunnable<Exception> callback, boolean allowOverdraft)
        A helper to request a StateRequestType.SYNC_POINT and run a callback if it finishes (once the record is not blocked).
        Parameters:
        callback - the callback to run if it finishes (once the record is not blocked).
        allowOverdraft - whether to overdraft the in-flight buffer.

      • drainInflightRecords

        public void drainInflightRecords(int targetNum)
        A helper function to drain in-flight records util inFlightRecordNum within the limit of given targetNum.
        Parameters:
        targetNum - the target inFlightRecordNum to achieve.

      • processNonRecord

        public void processNonRecord(@Nullable org.apache.flink.util.function.ThrowingRunnable<? extends Exception> triggerAction, @Nullable org.apache.flink.util.function.ThrowingRunnable<? extends Exception> finalAction)

      • getStateExecutor

        @VisibleForTesting public StateExecutor getStateExecutor()

      • getInFlightRecordNum

        @VisibleForTesting public int getInFlightRecordNum()

      • close

        public void close() throws IOException
        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface Closeable
        Throws:
        IOException