org.apache.hadoop.fs.s3a

Class S3AFastOutputStream

java.lang.Object

java.io.OutputStream

org.apache.hadoop.fs.s3a.S3AFastOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

@InterfaceStability.Unstable
public class S3AFastOutputStream
extends OutputStream

Upload files/parts asap directly from a memory buffer (instead of buffering to a file).

Uploads are managed low-level rather than through the AWS TransferManager. This allows for uploading each part of a multi-part upload as soon as the bytes are in memory, rather than waiting until the file is closed.

Unstable: statistics and error handling might evolve

Constructor Summary

Constructors

Constructor and Description

S3AFastOutputStream(com.amazonaws.services.s3.AmazonS3Client client, S3AFileSystem fs, String bucket, String key, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.FileSystem.Statistics statistics, com.amazonaws.services.s3.model.CannedAccessControlList cannedACL, String serverSideEncryptionAlgorithm, long partSize, long multiPartThreshold, ThreadPoolExecutor threadPoolExecutor) Creates a fast OutputStream that uploads to S3 from memory.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	write(byte[] b, int off, int len) Writes a range of bytes from to the memory buffer.
void	write(int b) Writes a byte to the memory buffer.

Methods inherited from class java.io.OutputStream

flush, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3AFastOutputStream

public S3AFastOutputStream(com.amazonaws.services.s3.AmazonS3Client client,
                   S3AFileSystem fs,
                   String bucket,
                   String key,
                   org.apache.hadoop.util.Progressable progress,
                   org.apache.hadoop.fs.FileSystem.Statistics statistics,
                   com.amazonaws.services.s3.model.CannedAccessControlList cannedACL,
                   String serverSideEncryptionAlgorithm,
                   long partSize,
                   long multiPartThreshold,
                   ThreadPoolExecutor threadPoolExecutor)
                    throws IOException

Creates a fast OutputStream that uploads to S3 from memory. For MultiPartUploads, as soon as sufficient bytes have been written to the stream a part is uploaded immediately (by using the low-level multi-part upload API on the AmazonS3Client).

Parameters:

client - AmazonS3Client used for S3 calls

fs - S3AFilesystem

bucket - S3 bucket name

key - S3 key name

progress - report progress in order to prevent timeouts

statistics - track FileSystem.Statistics on the performed operations

cannedACL - used CannedAccessControlList

serverSideEncryptionAlgorithm - algorithm for server side encryption

partSize - size of a single part in a multi-part upload (except last part)

multiPartThreshold - files at least this size use multi-part upload

Throws:

IOException

Method Detail

write

public void write(int b)
           throws IOException

Writes a byte to the memory buffer. If this causes the buffer to reach its limit, the actual upload is submitted to the threadpool.

Specified by:

write in class OutputStream

Parameters:

b - the int of which the lowest byte is written

Throws:

IOException

write

public void write(byte[] b,
         int off,
         int len)
           throws IOException

Writes a range of bytes from to the memory buffer. If this causes the buffer to reach its limit, the actual upload is submitted to the threadpool and the remainder of the array is written to memory (recursively).

Overrides:

write in class OutputStream

Parameters:

b - byte array containing

off - offset in array where to start

len - number of bytes to be written

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException