public class SwiftNativeFileSystemStore extends Object
Constructor and Description |
---|
SwiftNativeFileSystemStore() |
Modifier and Type | Method and Description |
---|---|
void |
copy(org.apache.hadoop.fs.Path srcKey,
org.apache.hadoop.fs.Path dstKey) |
void |
createDirectory(org.apache.hadoop.fs.Path path)
Create a directory
|
void |
createManifestForPartUpload(org.apache.hadoop.fs.Path path)
Tell the Swift server to expect a multi-part upload by submitting
a 0-byte file with the X-Object-Manifest header
|
boolean |
delete(org.apache.hadoop.fs.Path absolutePath,
boolean recursive)
Delete the entire tree.
|
boolean |
deleteObject(org.apache.hadoop.fs.Path path)
deletes object from Swift
|
static List<URI> |
extractUris(String json,
org.apache.hadoop.fs.Path path)
extracts URIs from json
|
long |
getBlocksize()
Get the default blocksize of this (bound) filesystem
|
int |
getBufferSizeKB() |
org.apache.hadoop.fs.Path |
getCorrectSwiftPath(org.apache.hadoop.fs.Path path)
Take a Hadoop path and return one which uses the URI prefix and authority
of this FS.
|
HttpBodyContent |
getObject(org.apache.hadoop.fs.Path path)
Get the object as an input stream
|
HttpBodyContent |
getObject(org.apache.hadoop.fs.Path path,
long byteRangeStart,
long length)
Get the input stream starting from a specific point.
|
org.apache.commons.httpclient.Header[] |
getObjectHeaders(org.apache.hadoop.fs.Path path,
boolean newest)
Get the HTTP headers, in case you really need the low-level
metadata
|
List<URI> |
getObjectLocation(org.apache.hadoop.fs.Path path)
Try to find the specific server(s) on which the data lives
|
SwiftFileStatus |
getObjectMetadata(org.apache.hadoop.fs.Path path)
Get the metadata of an object
|
SwiftFileStatus |
getObjectMetadata(org.apache.hadoop.fs.Path path,
boolean newest)
Get the metadata of an object
|
List<DurationStats> |
getOperationStatistics()
Get the current operation statistics
|
long |
getPartsizeKB() |
int |
getThrottleDelay() |
void |
initialize(URI fsURI,
org.apache.hadoop.conf.Configuration configuration)
Initalize the filesystem store -this creates the REST client binding.
|
org.apache.hadoop.fs.FileStatus[] |
listSubPaths(org.apache.hadoop.fs.Path path,
boolean recursive,
boolean newest)
List all elements in this directory
|
boolean |
objectExists(org.apache.hadoop.fs.Path path)
Does the object exist
|
boolean |
objectExists(SwiftObjectPath path)
Does the object exist
|
void |
rename(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
Rename through copy-and-delete.
|
boolean |
rmdir(org.apache.hadoop.fs.Path path)
deletes a directory from Swift.
|
void |
throttle()
Insert a throttled wait if the throttle delay > 0
|
String |
toString() |
void |
uploadFile(org.apache.hadoop.fs.Path path,
InputStream inputStream,
long length)
Upload a file/input stream of a specific length.
|
void |
uploadFilePart(org.apache.hadoop.fs.Path path,
int partNumber,
InputStream inputStream,
long length)
Upload part of a larger file.
|
public void initialize(URI fsURI, org.apache.hadoop.conf.Configuration configuration) throws IOException
fsURI
- URI of the filesystem, which is used to map to the filesystem-specific
options in the configuration fileconfiguration
- configurationIOException
- on any failure.public long getBlocksize()
public long getPartsizeKB()
public int getBufferSizeKB()
public int getThrottleDelay()
public void uploadFile(org.apache.hadoop.fs.Path path, InputStream inputStream, long length) throws IOException
path
- destination path in the swift filesysteminputStream
- input data. This is closed afterwards, alwayslength
- length of the dataIOException
- on a problempublic void uploadFilePart(org.apache.hadoop.fs.Path path, int partNumber, InputStream inputStream, long length) throws IOException
path
- destination pathpartNumber
- item number in the pathinputStream
- input datalength
- length of the dataIOException
- on a problempublic void createManifestForPartUpload(org.apache.hadoop.fs.Path path) throws IOException
path
- path of final finalIOException
public SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path) throws IOException
path
- pathIOException
- on a problemFileNotFoundException
- if there is nothing at the endpublic org.apache.commons.httpclient.Header[] getObjectHeaders(org.apache.hadoop.fs.Path path, boolean newest) throws IOException, FileNotFoundException
path
- path to probenewest
- newest or oldest?IOException
- IO problemFileNotFoundException
- if there is nothing at the endpublic SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path, boolean newest) throws IOException, FileNotFoundException
path
- pathnewest
- flag to say "set the newest header", otherwise take any entryIOException
- on a problemFileNotFoundException
- if there is nothing at the endpublic HttpBodyContent getObject(org.apache.hadoop.fs.Path path) throws IOException
path
- object pathIOException
- IO problemsFileNotFoundException
- path doesn't resolve to an objectpublic HttpBodyContent getObject(org.apache.hadoop.fs.Path path, long byteRangeStart, long length) throws IOException
path
- path to objectbyteRangeStart
- starting pointlength
- no. of bytesIOException
- IO problemspublic org.apache.hadoop.fs.FileStatus[] listSubPaths(org.apache.hadoop.fs.Path path, boolean recursive, boolean newest) throws IOException
path
- path to work withrecursive
- do a recursive getnewest
- ask for the newest, or can some out of date data work?IOException
- on IO problemsFileNotFoundException
- if the path is nonexistentpublic void createDirectory(org.apache.hadoop.fs.Path path) throws IOException
path
- pathIOException
public List<URI> getObjectLocation(org.apache.hadoop.fs.Path path) throws IOException
path
- path to probeIOException
- on problems determining the locationspublic boolean deleteObject(org.apache.hadoop.fs.Path path) throws IOException
path
- path to deleteIOException
- on a failurepublic boolean rmdir(org.apache.hadoop.fs.Path path) throws IOException
path
- path to deleteIOException
- on a failurepublic boolean objectExists(org.apache.hadoop.fs.Path path) throws IOException
path
- object pathIOException
- IO problems other than FileNotFound, which
is downgraded to an object does not exist return codepublic boolean objectExists(SwiftObjectPath path) throws IOException
path
- swift object pathIOException
- IO problems other than FileNotFound, which
is downgraded to an object does not exist return codepublic void rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws FileNotFoundException, SwiftOperationFailedException, IOException
Because of the nature of the operation, it is not atomic.
src
- source file/dirdst
- destinationIOException
- IO failureSwiftOperationFailedException
- if the rename failedFileNotFoundException
- if the source directory is missing, or
the parent directory of the destinationpublic void copy(org.apache.hadoop.fs.Path srcKey, org.apache.hadoop.fs.Path dstKey) throws IOException
IOException
public org.apache.hadoop.fs.Path getCorrectSwiftPath(org.apache.hadoop.fs.Path path) throws SwiftException
path
- path inSwiftException
- URI cannot be created.public static List<URI> extractUris(String json, org.apache.hadoop.fs.Path path) throws SwiftOperationFailedException
json
- json to parsepath
- path (used in exceptions)SwiftOperationFailedException
- on any problem parsing the JSONpublic void throttle() throws InterruptedIOException
InterruptedIOException
- if interrupted during sleeppublic List<DurationStats> getOperationStatistics()
public boolean delete(org.apache.hadoop.fs.Path absolutePath, boolean recursive) throws IOException
FileNotFoundException
is
raised. This lets the caller distinguish a file not found with
other reasons for failure, so handles race conditions in recursive
directory deletes better.
The problem being addressed is: caller A requests a recursive directory of directory /dir ; caller B requests a delete of a file /dir/file, between caller A enumerating the files contents, and requesting a delete of /dir/file. We want to recognise the special case "directed file is no longer there" and not convert that into a failure
absolutePath
- the path to delete.recursive
- if path is a directory and set to
true, the directory is deleted else throws an exception if the
directory is not empty
case of a file the recursive can be set to either true or false.IOException
- IO problemsFileNotFoundException
- if a file/dir being deleted is not there -
this includes entries below the specified path, (if the path is a dir
and recursive is true)Copyright © 2017 Apache Software Foundation. All Rights Reserved.