public class SwiftNativeFileSystemStore extends Object
Constructor and Description |
---|
SwiftNativeFileSystemStore() |
Modifier and Type | Method and Description |
---|---|
void |
copy(org.apache.hadoop.fs.Path srcKey,
org.apache.hadoop.fs.Path dstKey) |
void |
createDirectory(org.apache.hadoop.fs.Path path)
Create a directory
|
void |
createManifestForPartUpload(org.apache.hadoop.fs.Path path)
Tell the Swift server to expect a multi-part upload by submitting
a 0-byte file with the X-Object-Manifest header
|
boolean |
delete(org.apache.hadoop.fs.Path absolutePath,
boolean recursive)
Delete the entire tree.
|
boolean |
deleteObject(org.apache.hadoop.fs.Path path)
deletes object from Swift
|
static List<URI> |
extractUris(String json,
org.apache.hadoop.fs.Path path)
extracts URIs from json
|
long |
getBlocksize()
Get the default blocksize of this (bound) filesystem
|
int |
getBufferSizeKB() |
org.apache.hadoop.fs.Path |
getCorrectSwiftPath(org.apache.hadoop.fs.Path path)
Take a Hadoop path and return one which uses the URI prefix and authority
of this FS.
|
HttpBodyContent |
getObject(org.apache.hadoop.fs.Path path)
Get the object as an input stream
|
HttpBodyContent |
getObject(org.apache.hadoop.fs.Path path,
long byteRangeStart,
long length)
Get the input stream starting from a specific point.
|
org.apache.commons.httpclient.Header[] |
getObjectHeaders(org.apache.hadoop.fs.Path path,
boolean newest)
Get the HTTP headers, in case you really need the low-level
metadata
|
List<URI> |
getObjectLocation(org.apache.hadoop.fs.Path path)
Try to find the specific server(s) on which the data lives
|
SwiftFileStatus |
getObjectMetadata(org.apache.hadoop.fs.Path path)
Get the metadata of an object
|
SwiftFileStatus |
getObjectMetadata(org.apache.hadoop.fs.Path path,
boolean newest)
Get the metadata of an object
|
List<DurationStats> |
getOperationStatistics()
Get the current operation statistics
|
long |
getPartsizeKB() |
int |
getThrottleDelay() |
void |
initialize(URI fsURI,
org.apache.hadoop.conf.Configuration configuration)
Initalize the filesystem store -this creates the REST client binding.
|
org.apache.hadoop.fs.FileStatus[] |
listSubPaths(org.apache.hadoop.fs.Path path,
boolean recursive,
boolean newest)
List all elements in this directory
|
boolean |
objectExists(org.apache.hadoop.fs.Path path)
Does the object exist
|
boolean |
objectExists(SwiftObjectPath path)
Does the object exist
|
void |
rename(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
Rename through copy-and-delete.
|
boolean |
rmdir(org.apache.hadoop.fs.Path path)
deletes a directory from Swift.
|
void |
throttle()
Insert a throttled wait if the throttle delay >0
|
String |
toString() |
void |
uploadFile(org.apache.hadoop.fs.Path path,
InputStream inputStream,
long length)
Upload a file/input stream of a specific length.
|
void |
uploadFilePart(org.apache.hadoop.fs.Path path,
int partNumber,
InputStream inputStream,
long length)
Upload part of a larger file.
|
public void initialize(URI fsURI, org.apache.hadoop.conf.Configuration configuration) throws IOException
fsURI
- URI of the filesystem, which is used to map to the filesystem-specific
options in the configuration fileconfiguration
- configurationIOException
- on any failure.public long getBlocksize()
public long getPartsizeKB()
public int getBufferSizeKB()
public int getThrottleDelay()
public void uploadFile(org.apache.hadoop.fs.Path path, InputStream inputStream, long length) throws IOException
path
- destination path in the swift filesysteminputStream
- input data. This is closed afterwards, alwayslength
- length of the dataIOException
- on a problempublic void uploadFilePart(org.apache.hadoop.fs.Path path, int partNumber, InputStream inputStream, long length) throws IOException
path
- destination pathpartNumber
- item number in the pathinputStream
- input datalength
- length of the dataIOException
- on a problempublic void createManifestForPartUpload(org.apache.hadoop.fs.Path path) throws IOException
path
- path of final finalIOException
public SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path) throws IOException
path
- pathIOException
- on a problemFileNotFoundException
- if there is nothing at the endpublic org.apache.commons.httpclient.Header[] getObjectHeaders(org.apache.hadoop.fs.Path path, boolean newest) throws IOException, FileNotFoundException
path
- path to probenewest
- newest or oldest?IOException
- IO problemFileNotFoundException
- if there is nothing at the endpublic SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path, boolean newest) throws IOException, FileNotFoundException
path
- pathnewest
- flag to say "set the newest header", otherwise take any entryIOException
- on a problemFileNotFoundException
- if there is nothing at the endpublic HttpBodyContent getObject(org.apache.hadoop.fs.Path path) throws IOException
path
- object pathIOException
- IO problemsFileNotFoundException
- path doesn't resolve to an objectpublic HttpBodyContent getObject(org.apache.hadoop.fs.Path path, long byteRangeStart, long length) throws IOException
path
- path to objectbyteRangeStart
- starting pointlength
- no. of bytesIOException
- IO problemspublic org.apache.hadoop.fs.FileStatus[] listSubPaths(org.apache.hadoop.fs.Path path, boolean recursive, boolean newest) throws IOException
path
- path to work withrecursive
- do a recursive getnewest
- ask for the newest, or can some out of date data work?IOException
- on IO problemsFileNotFoundException
- if the path is nonexistentpublic void createDirectory(org.apache.hadoop.fs.Path path) throws IOException
path
- pathIOException
public List<URI> getObjectLocation(org.apache.hadoop.fs.Path path) throws IOException
path
- path to probeIOException
- on problems determining the locationspublic boolean deleteObject(org.apache.hadoop.fs.Path path) throws IOException
path
- path to deleteIOException
- on a failurepublic boolean rmdir(org.apache.hadoop.fs.Path path) throws IOException
path
- path to deleteIOException
- on a failurepublic boolean objectExists(org.apache.hadoop.fs.Path path) throws IOException
path
- object pathIOException
- IO problems other than FileNotFound, which
is downgraded to an object does not exist return codepublic boolean objectExists(SwiftObjectPath path) throws IOException
path
- swift object pathIOException
- IO problems other than FileNotFound, which
is downgraded to an object does not exist return codepublic void rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws FileNotFoundException, SwiftOperationFailedException, IOException
src
- source file/dirdst
- destinationIOException
- IO failureSwiftOperationFailedException
- if the rename failedFileNotFoundException
- if the source directory is missing, or
the parent directory of the destinationpublic void copy(org.apache.hadoop.fs.Path srcKey, org.apache.hadoop.fs.Path dstKey) throws IOException
IOException
public org.apache.hadoop.fs.Path getCorrectSwiftPath(org.apache.hadoop.fs.Path path) throws SwiftException
path
- path inSwiftException
- URI cannot be created.public static List<URI> extractUris(String json, org.apache.hadoop.fs.Path path) throws SwiftOperationFailedException
json
- json to parsepath
- path (used in exceptions)SwiftOperationFailedException
- on any problem parsing the JSONpublic void throttle() throws InterruptedIOException
InterruptedIOException
- if interrupted during sleeppublic List<DurationStats> getOperationStatistics()
public boolean delete(org.apache.hadoop.fs.Path absolutePath, boolean recursive) throws IOException
FileNotFoundException
is
raised. This lets the caller distinguish a file not found with
other reasons for failure, so handles race conditions in recursive
directory deletes better.
The problem being addressed is: caller A requests a recursive directory
of directory /dir ; caller B requests a delete of a file /dir/file,
between caller A enumerating the files contents, and requesting a delete
of /dir/file. We want to recognise the special case
"directed file is no longer there" and not convert that into a failureabsolutePath
- the path to delete.recursive
- if path is a directory and set to
true, the directory is deleted else throws an exception if the
directory is not empty
case of a file the recursive can be set to either true or false.IOException
- IO problemsFileNotFoundException
- if a file/dir being deleted is not there -
this includes entries below the specified path, (if the path is a dir
and recursive is true)Copyright © 2015 Apache Software Foundation. All Rights Reserved.