Package org.apache.flink.streaming.api.operators.python

Class AbstractPythonFunctionOperator<OUT>

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>
Direct Known Subclasses:
AbstractEmbeddedPythonFunctionOperator, AbstractExternalPythonFunctionOperator, DelegateOperatorTransformation.DelegateOperator
@Internal public abstract class AbstractPythonFunctionOperator<OUT> extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
Base class for all stream operators to execute Python functions.
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected Runnable
    bundleFinishedCallback
    Callback to be executed after the current bundle was finished.
    protected final org.apache.flink.configuration.Configuration
    config
     
    protected int
    elementCount
    Number of processed elements in the current bundle.
    protected long
    lastFinishBundleTime
    Time that the last bundle was finished.
    protected int
    maxBundleSize
    Max number of elements to include in a bundle.
    protected boolean
    systemEnvEnabled
     

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    AbstractPythonFunctionOperator(org.apache.flink.configuration.Configuration config)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected void
    checkInvokeFinishBundleByCount()
    Checks whether to invoke finishBundle by elements count.
    void
    close()
     
    protected abstract PythonEnvironmentManager
    createPythonEnvironmentManager()
     
    protected void
    drainUnregisteredTimers()
     
    void
    finish()
     
    org.apache.flink.configuration.Configuration
    getConfiguration()
    Returns the Configuration.
    protected FlinkMetricContainer
    getFlinkMetricContainer()
     
    protected abstract void
    invokeFinishBundle()
     
    boolean
    isBundleFinished()
    Returns whether the bundle is finished.
    void
    open()
     
    void
    prepareSnapshotPreBarrier(long checkpointId)
     
    void
    processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
     
    void
    setCurrentKey(Object key)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    getOperatorAttributes

  • Field Details

    • config

      protected final org.apache.flink.configuration.Configuration config

    • systemEnvEnabled

      protected transient boolean systemEnvEnabled

    • maxBundleSize

      protected transient int maxBundleSize
      Max number of elements to include in a bundle.

    • elementCount

      protected transient int elementCount
      Number of processed elements in the current bundle.

    • lastFinishBundleTime

      protected transient long lastFinishBundleTime
      Time that the last bundle was finished.

    • bundleFinishedCallback

      protected transient Runnable bundleFinishedCallback
      Callback to be executed after the current bundle was finished.

  • Constructor Details

    • AbstractPythonFunctionOperator

      public AbstractPythonFunctionOperator(org.apache.flink.configuration.Configuration config)

  • Method Details

    • open

      public void open() throws Exception
      Specified by:
      open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
      Overrides:
      open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • finish

      public void finish() throws Exception
      Specified by:
      finish in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
      Overrides:
      finish in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
      Overrides:
      close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • prepareSnapshotPreBarrier

      public void prepareSnapshotPreBarrier(long checkpointId) throws Exception
      Specified by:
      prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>
      Overrides:
      prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • processWatermark

      public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark) throws Exception
      Overrides:
      processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
      Throws:
      Exception

    • setCurrentKey

      public void setCurrentKey(Object key)
      Specified by:
      setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext
      Overrides:
      setCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

    • isBundleFinished

      public boolean isBundleFinished()
      Returns whether the bundle is finished.

    • getConfiguration

      public org.apache.flink.configuration.Configuration getConfiguration()
      Returns the Configuration.

    • invokeFinishBundle

      protected abstract void invokeFinishBundle() throws Exception
      Throws:
      Exception

    • createPythonEnvironmentManager

      protected abstract PythonEnvironmentManager createPythonEnvironmentManager()

    • drainUnregisteredTimers

      protected void drainUnregisteredTimers()

    • checkInvokeFinishBundleByCount

      protected void checkInvokeFinishBundleByCount() throws Exception
      Checks whether to invoke finishBundle by elements count. Called in processElement.
      Throws:
      Exception

    • getFlinkMetricContainer

      protected FlinkMetricContainer getFlinkMetricContainer()