Package org.apache.flink.python.chain

Class PythonOperatorChainingOptimizer

java.lang.Object
org.apache.flink.python.chain.PythonOperatorChainingOptimizer
public class PythonOperatorChainingOptimizer extends Object
A util class which attempts to chain all available Python functions.

An operator could be chained to it's predecessor if all of the following conditions are met:

  • Both of them are Python operators
  • The parallelism, the maximum parallelism and the slot sharing group are all the same
  • The chaining strategy is ChainingStrategy.ALWAYS and the chaining strategy of the predecessor isn't ChainingStrategy.NEVER
  • This partitioner between them is ForwardPartitioner

The properties of the generated chained operator are as following:

  • The name is the concatenation of all the names of the chained operators
  • The parallelism, the maximum parallelism and the slot sharing group are from one of the chained operators as all of them are the same between the chained operators
  • The chaining strategy is the same as the head operator
  • The uid and the uidHash are the same as the head operator

  • Constructor Summary

    Constructors
    Constructor
    Description
    PythonOperatorChainingOptimizer()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env)
    Perform chaining optimization.
    static org.apache.flink.api.dag.Transformation<?>
    apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, org.apache.flink.api.dag.Transformation<?> transformation)
    Perform chaining optimization.
    static List<org.apache.flink.api.dag.Transformation<?>>
    optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations)
    Perform chaining optimization.
    static org.apache.flink.api.java.tuple.Tuple2<List<org.apache.flink.api.dag.Transformation<?>>,org.apache.flink.api.dag.Transformation<?>>
    optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations, org.apache.flink.api.dag.Transformation<?> targetTransformation)
    Perform chaining optimization.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • PythonOperatorChainingOptimizer

      public PythonOperatorChainingOptimizer()

  • Method Details

    • apply

      public static void apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env) throws Exception
      Perform chaining optimization. It will iterate the transformations defined in the given StreamExecutionEnvironment and update them with the chained transformations.
      Throws:
      Exception

    • apply

      public static org.apache.flink.api.dag.Transformation<?> apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, org.apache.flink.api.dag.Transformation<?> transformation) throws Exception
      Perform chaining optimization. It will iterate the transformations defined in the given StreamExecutionEnvironment and update them with the chained transformations. Besides, it will return the transformation after chaining optimization for the given transformation.
      Throws:
      Exception

    • optimize

      public static List<org.apache.flink.api.dag.Transformation<?>> optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations)
      Perform chaining optimization. It will return the chained transformations for the given transformation list.

    • optimize

      public static org.apache.flink.api.java.tuple.Tuple2<List<org.apache.flink.api.dag.Transformation<?>>,org.apache.flink.api.dag.Transformation<?>> optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations, org.apache.flink.api.dag.Transformation<?> targetTransformation)
      Perform chaining optimization. It will returns the chained transformations and the transformation after chaining optimization for the given transformation.