Package org.apache.flink.streaming.runtime.io.recovery

Class RescalingStreamTaskNetworkInput<T>

java.lang.Object

org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

org.apache.flink.streaming.runtime.io.recovery.RescalingStreamTaskNetworkInput<T>

All Implemented Interfaces:

Closeable, AutoCloseable, AvailabilityProvider, PushingAsyncDataInput<T>, RecoverableStreamTaskInput<T>, StreamTaskInput<T>

@Internal
public final class RescalingStreamTaskNetworkInput<T>
extends AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>
implements RecoverableStreamTaskInput<T>

A StreamTaskNetworkInput implementation that demultiplexes virtual channels.

The demultiplexing works in two dimensions for the following cases. *

• Subtasks of the current operator have been collapsed in a round-robin fashion.
• The connected output operator has been rescaled (up and down!) and there is an overlap of channels (mostly relevant to keyed exchanges).

In both cases, records from multiple old channels are received over one new physical channel, which need to demultiplex the record to correctly restore spanning records (similar to how StreamTaskNetworkInput works).

Note that when both cases occur at the same time (downscaling of several operators), there is the cross product of channels. So if two subtasks are collapsed and two channels overlap from the output side, there is a total of 4 virtual channels.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Nested classes/interfaces inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

PushingAsyncDataInput.DataOutput<T>

Field Summary

Fields inherited from class org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput

canEmitBatchOfRecords, checkpointedInputGate, deserializationDelegate, flattenedChannelIndices, inputIndex, inputSerializer, recordDeserializers, statusWatermarkValve, watermarkCombiners

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Fields inherited from interface org.apache.flink.streaming.runtime.io.StreamTaskInput

UNSPECIFIED

Constructor Summary

Constructors

Constructor

Description

RescalingStreamTaskNetworkInput(CheckpointedInputGate checkpointedInputGate, org.apache.flink.api.common.typeutils.TypeSerializer<T> inputSerializer, IOManager ioManager, StatusWatermarkValve statusWatermarkValve, int inputIndex, InflightDataRescalingDescriptor inflightDataRescalingDescriptor, Function<Integer,StreamPartitioner<?>> gatePartitioners, org.apache.flink.api.common.TaskInfo taskInfo, StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)

Method Summary

Modifier and Type	Method	Description
StreamTaskInput<T>	finishRecovery()
protected org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>	getActiveSerializer(InputChannelInfo channelInfo)
CompletableFuture<Void>	prepareSnapshot(ChannelStateWriter channelStateWriter, long checkpointId)	Prepares to spill the in-flight input buffers as checkpoint snapshot.
protected DataInputStatus	processEvent(BufferOrEvent bufferOrEvent, PushingAsyncDataInput.DataOutput<T> output)

Methods inherited from class org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput

close, emitNext, getAvailableFuture, getInputIndex, processBuffer, releaseDeserializer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

getAvailableFuture, isApproximatelyAvailable, isAvailable

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

emitNext

Methods inherited from interface org.apache.flink.streaming.runtime.io.StreamTaskInput

getInputIndex

Constructor Details

RescalingStreamTaskNetworkInput

public RescalingStreamTaskNetworkInput(CheckpointedInputGate checkpointedInputGate,
 org.apache.flink.api.common.typeutils.TypeSerializer<T> inputSerializer,
 IOManager ioManager,
 StatusWatermarkValve statusWatermarkValve,
 int inputIndex,
 InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
 Function<Integer,StreamPartitioner<?>> gatePartitioners,
 org.apache.flink.api.common.TaskInfo taskInfo,
 StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)

Method Details

finishRecovery

public StreamTaskInput<T> finishRecovery()
                                  throws IOException

Specified by:

finishRecovery in interface RecoverableStreamTaskInput<T>

Throws:

IOException

getActiveSerializer

protected org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T> getActiveSerializer(InputChannelInfo channelInfo)

Overrides:

getActiveSerializer in class AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

processEvent

protected DataInputStatus processEvent(BufferOrEvent bufferOrEvent,
 PushingAsyncDataInput.DataOutput<T> output)

Overrides:

processEvent in class AbstractStreamTaskNetworkInput<T,org.apache.flink.streaming.runtime.io.recovery.DemultiplexingRecordDeserializer<T>>

prepareSnapshot

public CompletableFuture<Void> prepareSnapshot(ChannelStateWriter channelStateWriter,
 long checkpointId)
                                        throws CheckpointException

Description copied from interface: StreamTaskInput

Prepares to spill the in-flight input buffers as checkpoint snapshot.

Specified by:

prepareSnapshot in interface StreamTaskInput<T>

Throws:

CheckpointException