Package org.apache.flink.streaming.api.functions.sink.filesystem

Class RowWiseBucketWriter<IN,BucketID>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.RowWiseBucketWriter<IN,BucketID>

Type Parameters:

IN - The type of input elements.

BucketID - The type of ids for the buckets, as returned by the BucketAssigner.

All Implemented Interfaces:

BucketWriter<IN,BucketID>

@Internal
public class RowWiseBucketWriter<IN,BucketID>
extends Object

A factory that creates RowWisePartWriters.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter

BucketWriter.PendingFile

Constructor Summary

Constructors

Constructor	Description
RowWiseBucketWriter(org.apache.flink.core.fs.RecoverableWriter recoverableWriter, org.apache.flink.api.common.serialization.Encoder<IN> encoder)

Method Summary

Modifier and Type	Method	Description
boolean	cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)	Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.
WriterProperties	getProperties()
InProgressFileWriter<IN,BucketID>	openNew(BucketID bucketId, org.apache.flink.core.fs.RecoverableFsDataOutputStream stream, org.apache.flink.core.fs.Path path, long creationTime)
CompactingFileWriter	openNewCompactingFile(CompactingFileWriter.Type type, BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime)	Used to create a new CompactingFileWriter of the requesting type.
InProgressFileWriter<IN,BucketID>	openNewInProgressFile(BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime)	Used to create a new InProgressFileWriter.
BucketWriter.PendingFile	recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)	Recovers a pending file for finalizing and committing.
InProgressFileWriter<IN,BucketID>	resumeFrom(BucketID bucketId, org.apache.flink.core.fs.RecoverableFsDataOutputStream stream, org.apache.flink.core.fs.Path path, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable, long creationTime)
InProgressFileWriter<IN,BucketID>	resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable, long creationTime)	Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RowWiseBucketWriter

public RowWiseBucketWriter(org.apache.flink.core.fs.RecoverableWriter recoverableWriter,
 org.apache.flink.api.common.serialization.Encoder<IN> encoder)

Method Details

resumeFrom

public InProgressFileWriter<IN,BucketID> resumeFrom(BucketID bucketId,
 org.apache.flink.core.fs.RecoverableFsDataOutputStream stream,
 org.apache.flink.core.fs.Path path,
 org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable,
 long creationTime)

openNew

public InProgressFileWriter<IN,BucketID> openNew(BucketID bucketId,
 org.apache.flink.core.fs.RecoverableFsDataOutputStream stream,
 org.apache.flink.core.fs.Path path,
 long creationTime)

openNewInProgressFile

public InProgressFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID,
 org.apache.flink.core.fs.Path path,
 long creationTime)
                                                        throws IOException

Description copied from interface: BucketWriter

Used to create a new InProgressFileWriter.

Specified by:

openNewInProgressFile in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

path - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

openNewCompactingFile

public CompactingFileWriter openNewCompactingFile(CompactingFileWriter.Type type,
 BucketID bucketID,
 org.apache.flink.core.fs.Path path,
 long creationTime)
                                           throws IOException

Description copied from interface: BucketWriter

Used to create a new CompactingFileWriter of the requesting type. Requesting a writer of an unsupported type will result in UnsupportedOperationException. By default, only RECORD_WISE type is supported, for which a InProgressFileWriter will be created.

Specified by:

openNewCompactingFile in interface BucketWriter<IN,BucketID>

Parameters:

type - the type of this writer.

bucketID - the id of the bucket this writer is writing to.

path - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

resumeInProgressFileFrom

public InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID,
 InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable,
 long creationTime)
                                                           throws IOException

Description copied from interface: BucketWriter

Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Specified by:

resumeInProgressFileFrom in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

inProgressFileRecoverable - the state of the part file.

creationTime - the creation time of the file.

Returns:

the resumed InProgressFileWriter

Throws:

IOException - Thrown if resuming a writer fails.

recoverPendingFile

public BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
                                            throws IOException

Description copied from interface: BucketWriter

Recovers a pending file for finalizing and committing.

Specified by:

recoverPendingFile in interface BucketWriter<IN,BucketID>

Parameters:

pendingFileRecoverable - The handle with the recovery information.

Returns:

A pending file

Throws:

IOException - Thrown if recovering a pending file fails.

cleanupInProgressFileRecoverable

public boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
                                         throws IOException

Description copied from interface: BucketWriter

Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.

NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.

Specified by:

cleanupInProgressFileRecoverable in interface BucketWriter<IN,BucketID>

Parameters:

inProgressFileRecoverable - the InProgressFileWriter.InProgressFileRecoverable whose state we want to clean-up.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).

Throws:

IOException - if an I/O error occurs

getProperties

public WriterProperties getProperties()

Specified by:

getProperties in interface BucketWriter<IN,BucketID>

Returns:

the property of the BucketWriter