Package org.apache.hadoop.fs.aliyun.oss

Class AliyunOSSFileSystemStore

java.lang.Object

org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore

public class AliyunOSSFileSystemStore
extends java.lang.Object

Core implementation of Aliyun OSS Filesystem for Hadoop. Provides the bridging logic between Hadoop's abstract filesystem and Aliyun OSS.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
AliyunOSSFileSystemStore()

Method Summary

Modifier and Type	Method	Description
void	abortMultipartUpload(java.lang.String key, java.lang.String uploadId)	Abort the specific multipart upload.
void	close()	Close OSS client properly.
com.aliyun.oss.model.CompleteMultipartUploadResult	completeMultipartUpload(java.lang.String key, java.lang.String uploadId, java.util.List<com.aliyun.oss.model.PartETag> partETags)	Complete the specific multipart upload.
OSSListResult	continueListObjects(OSSListRequest listRequest, OSSListResult preListResult)	continue to list objects depends on previous list result.
boolean	copyFile(java.lang.String srcKey, long srcLen, java.lang.String dstKey)	Copy an object from source key to destination key.
protected OSSListRequest	createListObjectsRequest(java.lang.String prefix, int maxListingLength, java.lang.String marker, java.lang.String continuationToken, boolean recursive)	create list objects request.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	createLocatedFileStatusIterator(java.lang.String prefix, int maxListingLength, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.PathFilter filter, FileStatusAcceptor acceptor, boolean recursive)
void	deleteDirs(java.lang.String key)	Delete a directory from Aliyun OSS.
void	deleteObject(java.lang.String key)	Delete an object, and update write operation statistics.
void	deleteObjects(java.util.List<java.lang.String> keysToDelete)	Delete a list of keys, and update write operation statistics.
com.aliyun.oss.model.ObjectMetadata	getObjectMetadata(java.lang.String key)	Return metadata of a given object key.
java.lang.String	getUploadId(java.lang.String key)	Initiate multipart upload.
void	initialize(java.net.URI uri, org.apache.hadoop.conf.Configuration conf, java.lang.String user, org.apache.hadoop.fs.FileSystem.Statistics stat)
OSSListResult	listObjects(OSSListRequest listRequest)	list objects.
void	purge(java.lang.String prefix)	Clean up all objects matching the prefix.
java.io.InputStream	retrieve(java.lang.String key, long byteStart, long byteEnd)	Retrieve a part of an object.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	singleStatusRemoteIterator(org.apache.hadoop.fs.FileStatus fileStatus, org.apache.hadoop.fs.BlockLocation[] locations)
void	storeEmptyFile(java.lang.String key)	Upload an empty file as an OSS object, using single upload.
void	uploadObject(java.lang.String key, java.io.File file)	Upload a file as an OSS object, using single upload.
void	uploadObject(java.lang.String key, java.io.InputStream in, long size)	Upload an input stream as an OSS object, using single upload.
com.aliyun.oss.model.PartETag	uploadPart(java.io.File file, java.lang.String key, java.lang.String uploadId, int idx)
com.aliyun.oss.model.PartETag	uploadPart(java.io.InputStream in, long size, java.lang.String key, java.lang.String uploadId, int idx)
com.aliyun.oss.model.PartETag	uploadPart(org.apache.hadoop.fs.aliyun.oss.OSSDataBlocks.BlockUploadData partData, long size, java.lang.String key, java.lang.String uploadId, int idx)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AliyunOSSFileSystemStore

public AliyunOSSFileSystemStore()

Method Detail

initialize

public void initialize(java.net.URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       java.lang.String user,
                       org.apache.hadoop.fs.FileSystem.Statistics stat)
                throws java.io.IOException

Throws:

java.io.IOException

deleteObject

public void deleteObject(java.lang.String key)

Delete an object, and update write operation statistics.

Parameters:

key - key to blob to delete.

deleteObjects

public void deleteObjects(java.util.List<java.lang.String> keysToDelete)
                   throws java.io.IOException

Delete a list of keys, and update write operation statistics.

Parameters:

keysToDelete - collection of keys to delete.

Throws:

java.io.IOException - if failed to delete objects.

deleteDirs

public void deleteDirs(java.lang.String key)
                throws java.io.IOException

Delete a directory from Aliyun OSS.

Parameters:

key - directory key to delete.

Throws:

java.io.IOException - if failed to delete directory.

getObjectMetadata

public com.aliyun.oss.model.ObjectMetadata getObjectMetadata(java.lang.String key)

Return metadata of a given object key.

Parameters:

key - object key.

Returns:

return null if key does not exist.

storeEmptyFile

public void storeEmptyFile(java.lang.String key)
                    throws java.io.IOException

Upload an empty file as an OSS object, using single upload.

Parameters:

key - object key.

Throws:

java.io.IOException - if failed to upload object.

copyFile

public boolean copyFile(java.lang.String srcKey,
                        long srcLen,
                        java.lang.String dstKey)

Copy an object from source key to destination key.

Parameters:

srcKey - source key.

srcLen - source file length.

dstKey - destination key.

Returns:

true if file is successfully copied.

uploadObject

public void uploadObject(java.lang.String key,
                         java.io.File file)
                  throws java.io.IOException

Upload a file as an OSS object, using single upload.

Parameters:

key - object key.

file - local file to upload.

Throws:

java.io.IOException - if failed to upload object.

uploadObject

public void uploadObject(java.lang.String key,
                         java.io.InputStream in,
                         long size)
                  throws java.io.IOException

Upload an input stream as an OSS object, using single upload.

Parameters:

key - object key.

in - input stream to upload.

size - size of the input stream.

Throws:

java.io.IOException - if failed to upload object.

listObjects

public OSSListResult listObjects(OSSListRequest listRequest)

list objects.

Parameters:

listRequest - list request.

Returns:

a list of matches.

continueListObjects

public OSSListResult continueListObjects(OSSListRequest listRequest,
                                         OSSListResult preListResult)

continue to list objects depends on previous list result.

Parameters:

listRequest - list request.

preListResult - previous list result.

Returns:

a list of matches.

createListObjectsRequest

protected OSSListRequest createListObjectsRequest(java.lang.String prefix,
                                                  int maxListingLength,
                                                  java.lang.String marker,
                                                  java.lang.String continuationToken,
                                                  boolean recursive)

create list objects request.

Parameters:

prefix - prefix.

maxListingLength - max no. of entries

marker - last key in any previous search.

continuationToken - list from a specific point.

recursive - whether to list directory recursively.

Returns:

a list of matches.

retrieve

public java.io.InputStream retrieve(java.lang.String key,
                                    long byteStart,
                                    long byteEnd)

Retrieve a part of an object.

Parameters:

key - the object name that is being retrieved from the Aliyun OSS.

byteStart - start position.

byteEnd - end position.

Returns:

This method returns null if the key is not found.

close

public void close()

Close OSS client properly.

purge

public void purge(java.lang.String prefix)
           throws java.io.IOException

Clean up all objects matching the prefix.

Parameters:

prefix - Aliyun OSS object prefix.

Throws:

java.io.IOException - if failed to clean up objects.

singleStatusRemoteIterator

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> singleStatusRemoteIterator(org.apache.hadoop.fs.FileStatus fileStatus,
                                                                                                              org.apache.hadoop.fs.BlockLocation[] locations)

createLocatedFileStatusIterator

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> createLocatedFileStatusIterator(java.lang.String prefix,
                                                                                                                   int maxListingLength,
                                                                                                                   org.apache.hadoop.fs.FileSystem fs,
                                                                                                                   org.apache.hadoop.fs.PathFilter filter,
                                                                                                                   FileStatusAcceptor acceptor,
                                                                                                                   boolean recursive)

uploadPart

public com.aliyun.oss.model.PartETag uploadPart(org.apache.hadoop.fs.aliyun.oss.OSSDataBlocks.BlockUploadData partData,
                                                long size,
                                                java.lang.String key,
                                                java.lang.String uploadId,
                                                int idx)
                                         throws java.io.IOException

Throws:

java.io.IOException

uploadPart

public com.aliyun.oss.model.PartETag uploadPart(java.io.File file,
                                                java.lang.String key,
                                                java.lang.String uploadId,
                                                int idx)
                                         throws java.io.IOException

Throws:

java.io.IOException

uploadPart

public com.aliyun.oss.model.PartETag uploadPart(java.io.InputStream in,
                                                long size,
                                                java.lang.String key,
                                                java.lang.String uploadId,
                                                int idx)
                                         throws java.io.IOException

Throws:

java.io.IOException

getUploadId

public java.lang.String getUploadId(java.lang.String key)

Initiate multipart upload.

Parameters:

key - object key.

Returns:

upload id.

completeMultipartUpload

public com.aliyun.oss.model.CompleteMultipartUploadResult completeMultipartUpload(java.lang.String key,
                                                                                  java.lang.String uploadId,
                                                                                  java.util.List<com.aliyun.oss.model.PartETag> partETags)

Complete the specific multipart upload.

Parameters:

key - object key.

uploadId - upload id of this multipart upload.

partETags - part etags need to be completed.

Returns:

CompleteMultipartUploadResult.

abortMultipartUpload

public void abortMultipartUpload(java.lang.String key,
                                 java.lang.String uploadId)

Abort the specific multipart upload.

Parameters:

key - object key.

uploadId - upload id of this multipart upload.