Package org.apache.flink.streaming.api.graph

Class StreamGraphGenerator

java.lang.Object
org.apache.flink.streaming.api.graph.StreamGraphGenerator
@Internal public class StreamGraphGenerator extends Object
A generator that generates a StreamGraph from a graph of Transformations.

This traverses the tree of Transformations starting from the sinks. At each transformation we recursively transform the inputs, then create a node in the StreamGraph and add edges from the input Nodes to our newly created node. The transformation methods return the IDs of the nodes in the StreamGraph that represent the input transformation. Several IDs can be returned to be able to deal with feedback transformations and unions.

Partitioning, split/select and union don't create actual nodes in the StreamGraph. For these, we create a virtual node in the StreamGraph that holds the specific property, i.e. partitioning, selector and so on. When an edge is created from a virtual node to a downstream node the StreamGraph resolved the id of the original node and creates an edge in the graph with the desired property. For example, if you have this graph: 

     Map-1 -> HashPartition-2 -> Map-3

where the numbers represent transformation IDs. We first recurse all the way down. Map-1 is transformed, i.e. we create a StreamNode with ID 1. Then we transform the HashPartition, for this, we create virtual node of ID 4 that holds the property HashPartition. This transformation returns the ID 4. Then we transform the Map-3. We add the edge 4 -> 3. The StreamGraph resolved the actual node with ID 1 and creates and edge 1 -> 3 with the property HashPartition.

  • Field Details

  • Constructor Details

    • StreamGraphGenerator

      public StreamGraphGenerator(List<org.apache.flink.api.dag.Transformation<?>> transformations, org.apache.flink.api.common.ExecutionConfig executionConfig, CheckpointConfig checkpointConfig)

    • StreamGraphGenerator

      public StreamGraphGenerator(List<org.apache.flink.api.dag.Transformation<?>> transformations, org.apache.flink.api.common.ExecutionConfig executionConfig, CheckpointConfig checkpointConfig, org.apache.flink.configuration.Configuration configuration)

  • Method Details

    • getNewIterationNodeId

      public static int getNewIterationNodeId()

    • setSlotSharingGroupResource

      public StreamGraphGenerator setSlotSharingGroupResource(Map<String,ResourceProfile> slotSharingGroupResources)
      Specify fine-grained resource requirements for slot sharing groups.

      Note that a slot sharing group hints the scheduler that the grouped operators CAN be deployed into a shared slot. There's no guarantee that the scheduler always deploy the grouped operators together. In cases grouped operators are deployed into separate slots, the slot resources will be derived from the specified group requirements.

    • setSavepointRestoreSettings

      public void setSavepointRestoreSettings(SavepointRestoreSettings savepointRestoreSettings)

    • generate

      public StreamGraph generate()