Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote

Class RemoteStorageScanner

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteStorageScanner

All Implemented Interfaces:

Runnable

public class RemoteStorageScanner
extends Object
implements Runnable

The RemoteStorageScanner is introduced to notify asynchronously for file reading on remote storage. Asynchronous notifications will prevent RemoteTierConsumerAgent from repeatedly attempting to read remote files and reduce CPU consumption.

It will be invoked by RemoteTierConsumerAgent to watch the required segments and scan the existence status of the segments. If the segment file is found, it will notify the availability of segment file.

Constructor Summary

Constructors

Constructor	Description
RemoteStorageScanner(String baseRemoteStoragePath)

Method Summary

Modifier and Type	Method	Description
void	close()	Close the executor.
void	registerAvailabilityAndPriorityNotifier(BiConsumer<TieredStoragePartitionId,TieredStorageSubpartitionId> availabilityNotifier)
void	run()	Iterate the watched segment ids and check related file status.
void	start()	Start the executor.
void	watchSegment(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId)	Watch the segment for a specific subpartition in the RemoteStorageScanner.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RemoteStorageScanner

public RemoteStorageScanner(String baseRemoteStoragePath)

Method Details

start

public void start()

Start the executor.

watchSegment

public void watchSegment(TieredStoragePartitionId partitionId,
 TieredStorageSubpartitionId subpartitionId,
 int segmentId)

Watch the segment for a specific subpartition in the RemoteStorageScanner.

If a segment with a larger or equal id already exists, the current segment won't be watched.

If a segment with a smaller segment id is still being watched, the current segment will replace it because the smaller segment should have been consumed. This method ensures that only one segment file can be watched for each subpartition.

Parameters:

partitionId - is the id of partition.

subpartitionId - is the id of subpartition.

segmentId - is the id of segment.

close

public void close()

Close the executor.

run

public void run()

Iterate the watched segment ids and check related file status.

Specified by:

run in interface Runnable

registerAvailabilityAndPriorityNotifier

public void registerAvailabilityAndPriorityNotifier(BiConsumer<TieredStoragePartitionId,TieredStorageSubpartitionId> availabilityNotifier)