Package org.apache.flink.table.planner.plan.nodes.exec.stream

Class StreamExecDeduplicate

java.lang.Object

org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase<org.apache.flink.table.data.RowData>

org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecDeduplicate

All Implemented Interfaces:

ExecNode<org.apache.flink.table.data.RowData>, ExecNodeTranslator<org.apache.flink.table.data.RowData>, FusionCodegenExecNode, SingleTransformationTranslator<org.apache.flink.table.data.RowData>, StreamExecNode<org.apache.flink.table.data.RowData>

@ExecNodeMetadata(name="stream-exec-deduplicate",
                  version=1,
                  consumedOptions={"table.exec.mini-batch.enabled","table.exec.mini-batch.size","table.exec.deduplicate.insert-update-after-sensitive-enabled","table.exec.deduplicate.mini-batch.compact-changes-enabled"},
                  producedTransformations="deduplicate",
                  minPlanVersion=v1_15,
                  minStateVersion=v1_15)
public class StreamExecDeduplicate
extends ExecNodeBase<org.apache.flink.table.data.RowData>
implements StreamExecNode<org.apache.flink.table.data.RowData>, SingleTransformationTranslator<org.apache.flink.table.data.RowData>

Stream ExecNode which deduplicate on keys and keeps only first row or last row. This node is an optimization of StreamExecRank for some special cases. Compared to StreamExecRank, this node could use mini-batch and access less state.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEDUPLICATE_TRANSFORMATION
static final String	FIELD_NAME_GENERATE_UPDATE_BEFORE
static final String	FIELD_NAME_IS_ROWTIME
static final String	FIELD_NAME_KEEP_LAST_ROW
static final String	FIELD_NAME_UNIQUE_KEYS
static final String	STATE_NAME

Fields inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNode

FIELD_NAME_CONFIGURATION, FIELD_NAME_DESCRIPTION, FIELD_NAME_ID, FIELD_NAME_INPUT_PROPERTIES, FIELD_NAME_OUTPUT_TYPE, FIELD_NAME_STATE, FIELD_NAME_TYPE

Constructor Summary

Constructors

Constructor	Description
StreamExecDeduplicate(int id, ExecNodeContext context, org.apache.flink.configuration.ReadableConfig persistedConfig, int[] uniqueKeys, boolean isRowtime, boolean keepLastRow, boolean generateUpdateBefore, List<StateMetadata> stateMetadataList, List<InputProperty> inputProperties, org.apache.flink.table.types.logical.RowType outputType, String description)
StreamExecDeduplicate(org.apache.flink.configuration.ReadableConfig tableConfig, int[] uniqueKeys, boolean isRowtime, boolean keepLastRow, boolean generateUpdateBefore, InputProperty inputProperty, org.apache.flink.table.types.logical.RowType outputType, String description)

Method Summary

Modifier and Type	Method	Description
protected org.apache.flink.api.dag.Transformation<org.apache.flink.table.data.RowData>	translateToPlanInternal(PlannerBase planner, ExecNodeConfig config)	Internal method, translates this node into a Flink operator.

Methods inherited from class org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase

accept, createFormattedTransformationDescription, createFormattedTransformationName, createTransformationDescription, createTransformationMeta, createTransformationMeta, createTransformationName, createTransformationUid, getContextFromAnnotation, getDescription, getId, getInputEdges, getInputProperties, getOutputType, getPersistedConfig, getSimplifiedName, getTransformation, inputsContainSingleton, replaceInputEdge, setCompiled, setInputEdges, supportFusionCodegen, translateToFusionCodegenSpec, translateToFusionCodegenSpecInternal, translateToPlan

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNode

accept, getDescription, getId, getInputEdges, getInputProperties, getOutputType, replaceInputEdge, setCompiled, setInputEdges

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNodeTranslator

translateToPlan

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.FusionCodegenExecNode

supportFusionCodegen, translateToFusionCodegenSpec

Field Details

DEDUPLICATE_TRANSFORMATION

public static final String DEDUPLICATE_TRANSFORMATION

See Also:

• Constant Field Values

FIELD_NAME_UNIQUE_KEYS

public static final String FIELD_NAME_UNIQUE_KEYS

See Also:

• Constant Field Values

FIELD_NAME_IS_ROWTIME

public static final String FIELD_NAME_IS_ROWTIME

See Also:

• Constant Field Values

FIELD_NAME_KEEP_LAST_ROW

public static final String FIELD_NAME_KEEP_LAST_ROW

See Also:

• Constant Field Values

FIELD_NAME_GENERATE_UPDATE_BEFORE

public static final String FIELD_NAME_GENERATE_UPDATE_BEFORE

See Also:

• Constant Field Values

STATE_NAME

public static final String STATE_NAME

See Also:

• Constant Field Values

Constructor Details

StreamExecDeduplicate

public StreamExecDeduplicate(org.apache.flink.configuration.ReadableConfig tableConfig,
 int[] uniqueKeys,
 boolean isRowtime,
 boolean keepLastRow,
 boolean generateUpdateBefore,
 InputProperty inputProperty,
 org.apache.flink.table.types.logical.RowType outputType,
 String description)

StreamExecDeduplicate

public StreamExecDeduplicate(int id,
 ExecNodeContext context,
 org.apache.flink.configuration.ReadableConfig persistedConfig,
 int[] uniqueKeys,
 boolean isRowtime,
 boolean keepLastRow,
 boolean generateUpdateBefore,
 @Nullable
 List<StateMetadata> stateMetadataList,
 List<InputProperty> inputProperties,
 org.apache.flink.table.types.logical.RowType outputType,
 String description)

Method Details

translateToPlanInternal

protected org.apache.flink.api.dag.Transformation<org.apache.flink.table.data.RowData> translateToPlanInternal(PlannerBase planner,
 ExecNodeConfig config)

Description copied from class: ExecNodeBase

Internal method, translates this node into a Flink operator.

Specified by:

translateToPlanInternal in class ExecNodeBase<org.apache.flink.table.data.RowData>

Parameters:

planner - The planner.

config - per-ExecNode configuration that contains the merged configuration from various layers which all the nodes implementing this method should use, instead of retrieving configuration from the planner. For more details check ExecNodeConfig.