Package org.apache.flink.table.planner.plan.optimize

Class StreamNonDeterministicUpdatePlanVisitor

java.lang.Object
org.apache.flink.table.planner.plan.optimize.StreamNonDeterministicUpdatePlanVisitor
public class StreamNonDeterministicUpdatePlanVisitor extends Object
An inner visitor to validate if there's any NDU problems which may cause wrong result and try to rewrite lookup join node with materialization (to eliminate the non-deterministic result generated by lookup join node only).

The visitor will try to satisfy the required determinism(represent by ImmutableBitSet) from root. The transmission rule of required determinism:

0. all required determinism is under the precondition: input has updates, that is say no update determinism will be passed to an insert only stream

1. the initial required determinism to the root node(e.g., sink node) was none

2. for a relNode, it will process on two aspects: - can satisfy non-empty required determinism - actively requires determinism from input by self requirements(e.g., stateful node works on retract by row mode) 


 Rel3
  | require input
  v
 Rel2 {1. satisfy Rel3's requirement 2. append new requirement to input Rel1}
  | require input
  v
 Rel1

the requiredDeterminism passed to input will exclude columns which were in upsertKey, e.g., 


 Sink {pk=(c3)} requiredDeterminism=(c3)
   | passed requiredDeterminism={}
  GroupAgg{group by c3, day} append requiredDeterminism=(c3, day)
   | passed requiredDeterminism=(c3, day)
 Project{select c1,c2,DATE_FORMAT(CURRENT_TIMESTAMP, 'yyMMdd') day,...} [x] can not satisfy
   |
 Deduplicate{keep last row, dedup on c1,c2}
   |
  Scan

3. for a sink node, it will require key columns' determinism when primary key is defined or require all columns' determinism when no primary key is defined

4. for a cdc source node(which will generate updates), the metadata columns are treated as non-deterministic if changelogNormalize is not enabled.

  • Constructor Details

    • StreamNonDeterministicUpdatePlanVisitor

      public StreamNonDeterministicUpdatePlanVisitor()

  • Method Details

    • visit

      public StreamPhysicalRel visit(StreamPhysicalRel rel)

    • visit

      public StreamPhysicalRel visit(StreamPhysicalRel rel, org.apache.calcite.util.ImmutableBitSet requireDeterminism)