Package org.apache.flink.connector.file.sink.writer

Class FileWriter<IN>

java.lang.Object

org.apache.flink.connector.file.sink.writer.FileWriter<IN>

Type Parameters:

IN - The type of input elements.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback, org.apache.flink.api.connector.sink2.CommittingSinkWriter<IN,FileSinkCommittable>, org.apache.flink.api.connector.sink2.SinkWriter<IN>, org.apache.flink.api.connector.sink2.StatefulSinkWriter<IN,FileWriterBucketState>

@Internal
public class FileWriter<IN>
extends Object
implements org.apache.flink.api.connector.sink2.StatefulSinkWriter<IN,FileWriterBucketState>, org.apache.flink.api.connector.sink2.CommittingSinkWriter<IN,FileSinkCommittable>, org.apache.flink.api.connector.sink2.SinkWriter<IN>, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback

A SinkWriter implementation for FileSink.

It writes data to and manages the different active buckes in the FileSink.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

org.apache.flink.api.connector.sink2.SinkWriter.Context

Constructor Summary

Constructors

Constructor

Description

FileWriter(org.apache.flink.core.fs.Path basePath, org.apache.flink.metrics.groups.SinkWriterMetricGroup metricGroup, org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,String> bucketAssigner, FileWriterBucketFactory<IN> bucketFactory, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,String> bucketWriter, org.apache.flink.streaming.api.functions.sink.filesystem.RollingPolicy<IN,String> rollingPolicy, org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig, org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService, long bucketCheckInterval)

A constructor creating a new empty bucket manager.

Method Summary

Modifier and Type	Method	Description
void	close()
void	flush(boolean endOfInput)
void	initializeState(Collection<FileWriterBucketState> bucketStates)	Initializes the state after recovery from a failure.
void	onProcessingTime(long time)
Collection<FileSinkCommittable>	prepareCommit()
List<FileWriterBucketState>	snapshotState(long checkpointId)
void	write(IN element, org.apache.flink.api.connector.sink2.SinkWriter.Context context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

writeWatermark

Constructor Details

FileWriter

public FileWriter(org.apache.flink.core.fs.Path basePath,
 org.apache.flink.metrics.groups.SinkWriterMetricGroup metricGroup,
 org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner<IN,String> bucketAssigner,
 FileWriterBucketFactory<IN> bucketFactory,
 org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<IN,String> bucketWriter,
 org.apache.flink.streaming.api.functions.sink.filesystem.RollingPolicy<IN,String> rollingPolicy,
 org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig outputFileConfig,
 org.apache.flink.api.common.operators.ProcessingTimeService processingTimeService,
 long bucketCheckInterval)

A constructor creating a new empty bucket manager.

Parameters:

basePath - The base path for our buckets.

metricGroup - SinkWriterMetricGroup to set sink writer specific metrics.

bucketAssigner - The BucketAssigner provided by the user.

bucketFactory - The FileWriterBucketFactory to be used to create buckets.

bucketWriter - The BucketWriter to be used when writing data.

rollingPolicy - The RollingPolicy as specified by the user.

Method Details

initializeState

public void initializeState(Collection<FileWriterBucketState> bucketStates)
                     throws IOException

Initializes the state after recovery from a failure.

During this process:

1. we set the initial value for part counter to the maximum value used before across all tasks and buckets. This guarantees that we do not overwrite valid data,
2. we commit any pending files for previous checkpoints (previous to the last successful one from which we restore),
3. we resume writing to the previous in-progress file of each bucket, and
4. if we receive multiple states for the same bucket, we merge them.

Parameters:

bucketStates - the state holding recovered state about active buckets.

Throws:

IOException - if anything goes wrong during retrieving the state or restoring/committing of any in-progress/pending part files

write

public void write(IN element,
 org.apache.flink.api.connector.sink2.SinkWriter.Context context)
           throws IOException,
InterruptedException

Specified by:

write in interface org.apache.flink.api.connector.sink2.SinkWriter<IN>

Throws:

IOException

InterruptedException

flush

public void flush(boolean endOfInput)
           throws IOException,
InterruptedException

Specified by:

flush in interface org.apache.flink.api.connector.sink2.SinkWriter<IN>

Throws:

IOException

InterruptedException

prepareCommit

public Collection<FileSinkCommittable> prepareCommit()
                                              throws IOException

Specified by:

prepareCommit in interface org.apache.flink.api.connector.sink2.CommittingSinkWriter<IN,FileSinkCommittable>

Throws:

IOException

snapshotState

public List<FileWriterBucketState> snapshotState(long checkpointId)
                                          throws IOException

Specified by:

snapshotState in interface org.apache.flink.api.connector.sink2.StatefulSinkWriter<IN,FileWriterBucketState>

Throws:

IOException

close

public void close()

Specified by:

close in interface AutoCloseable

onProcessingTime

public void onProcessingTime(long time)
                      throws IOException

Specified by:

onProcessingTime in interface org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback

Throws:

IOException