org.apache.hadoop.fs.swift.snative

Class SwiftNativeFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class SwiftNativeFileSystem
extends org.apache.hadoop.fs.FileSystem

Swift file system implementation. Extends Hadoop FileSystem

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	SWIFT filesystem prefix: "swift"

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
SwiftNativeFileSystem() Default constructor for Hadoop
SwiftNativeFileSystem(SwiftNativeFileSystemStore store) This constructor used for testing purposes

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) This optional operation is not supported
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path file, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
protected org.apache.hadoop.fs.swift.snative.SwiftNativeOutputStream	createSwiftOutputStream(org.apache.hadoop.fs.Path path) Create the swift output stream
boolean	delete(org.apache.hadoop.fs.Path f) Delete a file.
boolean	delete(org.apache.hadoop.fs.Path path, boolean recursive) Delete a file or directory
long	getBlockSize(org.apache.hadoop.fs.Path path)
static long	getBytesUploaded(org.apache.hadoop.fs.FSDataOutputStream outputStream) Get the the number of bytes uploaded by an output stream to the swift cluster.
static long	getBytesWritten(org.apache.hadoop.fs.FSDataOutputStream outputStream) Get the the number of bytes written to an output stream This is for testing
long	getDefaultBlockSize() The blocksize of this filesystem is set by the property SwiftProtocolConstants.SWIFT_BLOCKSIZE;the default is the value of SwiftProtocolConstants.DEFAULT_SWIFT_BLOCKSIZE;
long	getDefaultBlockSize(org.apache.hadoop.fs.Path f) The blocksize for this filesystem.
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path path) Return a file status object that represents the path.
List<DurationStats>	getOperationStatistics() Get the current operation statistics
static long	getPartitionSize(org.apache.hadoop.fs.FSDataOutputStream outputStream) Get the size of partitions written by an output stream This is for testing
static int	getPartitionsWritten(org.apache.hadoop.fs.FSDataOutputStream outputStream) Get the number of partitions written by an output stream This is for testing
String	getScheme()
SwiftNativeFileSystemStore	getStore() This is for testing
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory() Path to user working directory
void	initialize(URI fsuri, org.apache.hadoop.conf.Configuration conf) default class initialization
boolean	isDirectory(org.apache.hadoop.fs.Path f)
boolean	isFile(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus[]	listRawFileStatus(org.apache.hadoop.fs.Path path, boolean newest) Low level method to do a deep listing of all entries, not stopping at the next directory entry.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path path) List the statuses of the files/directories in the given path if the path is a directory.
protected org.apache.hadoop.fs.Path	makeAbsolute(org.apache.hadoop.fs.Path path) Makes path absolute
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Create the parent directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize) Opens an FSDataInputStream at the indicated Path.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize, long readBlockSize) Low-level operation to also set the block size for this operation
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames Path src to Path dst.
void	setWorkingDirectory(org.apache.hadoop.fs.Path dir)
String	toString()

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFid, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathId, createSnapshot, createSnapshot, createSymlink, deleteFid, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getXAttr, getXAttrs, getXAttrs, getZkConnectString, globStatus, globStatus, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listXAttrs, makeQualified, mkdirs, mkdirs, mkdirsFid, mkdirsFid, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, openFid, openFid, openFid2, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setOwnerFid, setPermission, setReplication, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SWIFT

public static final String SWIFT

filesystem prefix: "swift"

See Also:

Constant Field Values

Constructor Detail

SwiftNativeFileSystem

public SwiftNativeFileSystem()

Default constructor for Hadoop

SwiftNativeFileSystem

public SwiftNativeFileSystem(SwiftNativeFileSystemStore store)

This constructor used for testing purposes

Method Detail

getStore

public SwiftNativeFileSystemStore getStore()

This is for testing

Returns:

the inner store class

getScheme

public String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(URI fsuri,
              org.apache.hadoop.conf.Configuration conf)
                throws IOException

default class initialization

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

fsuri - path to Swift

conf - Hadoop configuration

Throws:

IOException

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

Returns:

path to Swift

toString

public String toString()

Overrides:

toString in class Object

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Path to user working directory

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

Hadoop path

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path dir)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

dir - user working directory

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path path)
                                              throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path we want information from

Returns:

a FileStatus object

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

The blocksize of this filesystem is set by the property SwiftProtocolConstants.SWIFT_BLOCKSIZE;the default is the value of SwiftProtocolConstants.DEFAULT_SWIFT_BLOCKSIZE;

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

Returns:

the blocksize for this FS.

getDefaultBlockSize

public long getDefaultBlockSize(org.apache.hadoop.fs.Path f)

The blocksize for this filesystem.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path of file

Returns:

the blocksize for the path

See Also:

getDefaultBlockSize()

getBlockSize

public long getBlockSize(org.apache.hadoop.fs.Path path)
                  throws IOException

Overrides:

getBlockSize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

isFile

public boolean isFile(org.apache.hadoop.fs.Path f)
               throws IOException

Overrides:

isFile in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

isDirectory

public boolean isDirectory(org.apache.hadoop.fs.Path f)
                    throws IOException

Overrides:

isDirectory in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus file,
                                                         long start,
                                                         long len)
                                                           throws IOException

Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned.

This call is most helpful with DFS, where it returns hostnames of machines that contain the given file.

The FileSystem will simply return an elt containing 'localhost'.

Overrides:

getFileBlockLocations in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
             org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Create the parent directories. As an optimization, the entire hierarchy of parent directories is Not polled. Instead the tree is walked up from the last to the first, creating directories until one that exists is found. This strategy means if a file is created in an existing directory, one quick poll sufficies. There is a big assumption here: that all parent directories of an existing directory also exists.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create.

permission - to apply to files

Returns:

true if the operation was successful

Throws:

IOException - on a problem

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path path)
                                             throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

path - given path

Returns:

the statuses of the files/directories in the given path

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                             int bufferSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

This optional operation is not supported

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path file,
                                             org.apache.hadoop.fs.permission.FsPermission permission,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

permission - Currently ignored.

Throws:

IOException

createSwiftOutputStream

protected org.apache.hadoop.fs.swift.snative.SwiftNativeOutputStream createSwiftOutputStream(org.apache.hadoop.fs.Path path)
                                                                                      throws IOException

Create the swift output stream

Parameters:

path - path to write to

Returns:

the new file

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                          int bufferSize)
                                            throws IOException

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file name to open

bufferSize - the size of the buffer to be used.

Returns:

the input stream

Throws:

FileNotFoundException - if the file is not found

IOException - any IO problem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                          int bufferSize,
                                          long readBlockSize)
                                            throws IOException

Low-level operation to also set the block size for this operation

Parameters:

path - the file name to open

bufferSize - the size of the buffer to be used.

readBlockSize - how big should the read blockk/buffer size be?

Returns:

the input stream

Throws:

FileNotFoundException - if the file is not found

IOException - any IO problem

rename

public boolean rename(org.apache.hadoop.fs.Path src,
             org.apache.hadoop.fs.Path dst)
               throws IOException

Renames Path src to Path dst. On swift this uses copy-and-delete and is not atomic.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - path

dst - path

Returns:

true if directory renamed, false otherwise

Throws:

IOException - on problems

delete

public boolean delete(org.apache.hadoop.fs.Path path,
             boolean recursive)
               throws IOException

Delete a file or directory

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception if the directory is not empty case of a file the recursive can be set to either true or false.

Returns:

true if the object was deleted

Throws:

IOException - IO problems

delete

public boolean delete(org.apache.hadoop.fs.Path f)
               throws IOException

Delete a file. This method is abstract in Hadoop 1.x; in 2.x+ it is non-abstract and deprecated

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

makeAbsolute

protected org.apache.hadoop.fs.Path makeAbsolute(org.apache.hadoop.fs.Path path)

Makes path absolute

Parameters:

path - path to file

Returns:

absolute path

getOperationStatistics

public List<DurationStats> getOperationStatistics()

Get the current operation statistics

Returns:

a snapshot of the statistics

listRawFileStatus

@InterfaceAudience.Private
public org.apache.hadoop.fs.FileStatus[] listRawFileStatus(org.apache.hadoop.fs.Path path,
                                                                            boolean newest)
                                                    throws IOException

Low level method to do a deep listing of all entries, not stopping at the next directory entry. This is to let tests be confident that recursive deletes &c really are working.

Parameters:

path - path to recurse down

newest - ask for the newest data, potentially slower than not.

Returns:

a potentially empty array of file status

Throws:

IOException - any problem

getPartitionsWritten

@InterfaceAudience.Private
public static int getPartitionsWritten(org.apache.hadoop.fs.FSDataOutputStream outputStream)

Get the number of partitions written by an output stream This is for testing

Parameters:

outputStream - output stream

Returns:

the #of partitions written by that stream

getPartitionSize

@InterfaceAudience.Private
public static long getPartitionSize(org.apache.hadoop.fs.FSDataOutputStream outputStream)

Get the size of partitions written by an output stream This is for testing

Parameters:

outputStream - output stream

Returns:

partition size in bytes

getBytesWritten

@InterfaceAudience.Private
public static long getBytesWritten(org.apache.hadoop.fs.FSDataOutputStream outputStream)

Get the the number of bytes written to an output stream This is for testing

Parameters:

outputStream - output stream

Returns:

partition size in bytes

getBytesUploaded

@InterfaceAudience.Private
public static long getBytesUploaded(org.apache.hadoop.fs.FSDataOutputStream outputStream)

Get the the number of bytes uploaded by an output stream to the swift cluster. This is for testing

Parameters:

outputStream - output stream

Returns:

partition size in bytes