Interface BucketWriter<IN,BucketID>
- All Known Implementing Classes:
BulkBucketWriter,RowWiseBucketWriter
@Internal
public interface BucketWriter<IN,BucketID>
An interface for factories that create the different
writers.-
Nested Class Summary
Nested ClassesModifier and TypeInterfaceDescriptionstatic interfaceThis represents the file that can not write any data to. -
Method Summary
Modifier and TypeMethodDescriptionbooleancleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable) Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.default CompactingFileWriteropenNewCompactingFile(CompactingFileWriter.Type type, BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime) Used to create a newCompactingFileWriterof the requesting type.openNewInProgressFile(BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime) Used to create a newInProgressFileWriter.recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) Recovers a pending file for finalizing and committing.resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime) Used to resume aInProgressFileWriterfrom aInProgressFileWriter.InProgressFileRecoverable.
-
Method Details
-
openNewInProgressFile
InProgressFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime) throws IOException Used to create a newInProgressFileWriter.- Parameters:
bucketID- the id of the bucket this writer is writing to.path- the path this writer will write to.creationTime- the creation time of the file.- Returns:
- the new
InProgressFileWriter - Throws:
IOException- Thrown if creating a writer fails.
-
openNewCompactingFile
default CompactingFileWriter openNewCompactingFile(CompactingFileWriter.Type type, BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime) throws IOException Used to create a newCompactingFileWriterof the requesting type. Requesting a writer of an unsupported type will result in UnsupportedOperationException. By default, only RECORD_WISE type is supported, for which aInProgressFileWriterwill be created.- Parameters:
type- the type of this writer.bucketID- the id of the bucket this writer is writing to.path- the path this writer will write to.creationTime- the creation time of the file.- Returns:
- the new
InProgressFileWriter - Throws:
IOException- Thrown if creating a writer fails.UnsupportedOperationException- Thrown if the bucket writer doesn't support the requesting type.
-
resumeInProgressFileFrom
InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime) throws IOException Used to resume aInProgressFileWriterfrom aInProgressFileWriter.InProgressFileRecoverable.- Parameters:
bucketID- the id of the bucket this writer is writing to.inProgressFileSnapshot- the state of the part file.creationTime- the creation time of the file.- Returns:
- the resumed
InProgressFileWriter - Throws:
IOException- Thrown if resuming a writer fails.
-
getProperties
WriterProperties getProperties()- Returns:
- the property of the
BucketWriter
-
recoverPendingFile
BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) throws IOException Recovers a pending file for finalizing and committing.- Parameters:
pendingFileRecoverable- The handle with the recovery information.- Returns:
- A pending file
- Throws:
IOException- Thrown if recovering a pending file fails.
-
cleanupInProgressFileRecoverable
boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable) throws IOException Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.
- Parameters:
inProgressFileRecoverable- theInProgressFileWriter.InProgressFileRecoverablewhose state we want to clean-up.- Returns:
trueif the resources were successfully freed,falseotherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).- Throws:
IOException- if an I/O error occurs
-