org.apache.hadoop.fs.s3a

Class S3AFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.s3a.S3AFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class S3AFileSystem
extends org.apache.hadoop.fs.FileSystem

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCKSIZE Default blocksize as used in blocksize and FS status queries
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
S3AFileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) Append to an existing file (optional operation).
void	close()
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) The src file is on the local disk.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive) Delete a file.
void	finishedWrite(String key)
String	getCanonicalServiceName() Override getCononicalServiceName because we don't support token in S3A
long	getDefaultBlockSize() Deprecated. use FileSystem.getDefaultBlockSize(Path) instead
S3AFileStatus	getFileStatus(org.apache.hadoop.fs.Path f) Return a file status object that represents the path.
static ThreadFactory	getNamedThreadFactory(String prefix) Returns a ThreadFactory that names each created thread uniquely, with a common prefix.
String	getScheme() Return the protocol scheme for the FileSystem.
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
org.apache.hadoop.fs.Path	getWorkingDirectory() Get the current working directory for the given file system
void	initialize(URI name, org.apache.hadoop.conf.Configuration conf) Called after a new FileSystem instance is constructed.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission) Make the given file and all non-existent parents into directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames Path src to Path dst.
void	setWorkingDirectory(org.apache.hadoop.fs.Path new_dir) Set the current working directory for the given file system.

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFid, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathId, createSnapshot, createSnapshot, createSymlink, delete, deleteFid, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getXAttr, getXAttrs, getXAttrs, getZkConnectString, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, mkdirsFid, mkdirsFid, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, openFid, openFid, openFid2, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setOwnerFid, setPermission, setReplication, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BLOCKSIZE

public static final int DEFAULT_BLOCKSIZE

Default blocksize as used in blocksize and FS status queries

See Also:

Constant Field Values

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

S3AFileSystem

public S3AFileSystem()

Method Detail

getNamedThreadFactory

public static ThreadFactory getNamedThreadFactory(String prefix)

Returns a ThreadFactory that names each created thread uniquely, with a common prefix.

Parameters:

prefix - The prefix of every created Thread's name

Returns:

a ThreadFactory that names threads

initialize

public void initialize(URI name,
              org.apache.hadoop.conf.Configuration conf)
                throws IOException

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

name - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

Throws:

IOException

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

Returns:

"s3a"

getUri

public URI getUri()

Returns a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                          int bufferSize)
                                            throws IOException

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                             org.apache.hadoop.fs.permission.FsPermission permission,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

permission -

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize -

progress -

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                             int bufferSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Append to an existing file (optional operation).

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
             org.apache.hadoop.fs.Path dst)
               throws IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS. Warning: S3 does not support renames. This method does a copy which can take S3 some time to execute with large files and directories. Since there is no Progressable passed in, this can time out jobs. Note: This implementation differs with other S3 drivers. Specifically: Fails if src is a file and dst is a directory. Fails if src is a directory and dst is a file. Fails if the parent of dst does not exist or is a file. Fails if dst is a directory that is not empty.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

delete

public boolean delete(org.apache.hadoop.fs.Path f,
             boolean recursive)
               throws IOException

Delete a file.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws FileNotFoundException,
                                                    IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path new_dir)

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

new_dir -

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Get the current working directory for the given file system

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

the directory pathname

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
             org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path to create

permission - to apply to f

Throws:

IOException

getFileStatus

public S3AFileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                            throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                     boolean overwrite,
                     org.apache.hadoop.fs.Path src,
                     org.apache.hadoop.fs.Path dst)
                       throws IOException

The src file is on the local disk. Add it to FS at the given dst name. This version doesn't need to create a temporary file to calculate the md5. Sadly this doesn't seem to be used by the shell cp :( delSrc indicates if the source should be removed

Overrides:

copyFromLocalFile in class org.apache.hadoop.fs.FileSystem

Parameters:

delSrc - whether to delete the src

overwrite - whether to overwrite an existing file

src - path

dst - path

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getCanonicalServiceName

public String getCanonicalServiceName()

Override getCononicalServiceName because we don't support token in S3A

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

finishedWrite

public void finishedWrite(String key)
                   throws IOException

Throws:

IOException

getDefaultBlockSize

@Deprecated
public long getDefaultBlockSize()

Deprecated. use FileSystem.getDefaultBlockSize(Path) instead

Return the number of bytes that large input files should be optimally be split into to minimize i/o time.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem