Package org.apache.hadoop.io.wrappedio.impl

Class DynamicWrappedIO

java.lang.Object

org.apache.hadoop.io.wrappedio.impl.DynamicWrappedIO

public final class DynamicWrappedIO
extends java.lang.Object

The wrapped IO methods in org.apache.hadoop.io.wrappedio.WrappedIO, dynamically loaded.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	PARQUET_READ_POLICIES	Read policy for parquet files: "parquet, columnar, vector, random".

Constructor Summary

Constructors

Constructor	Description
DynamicWrappedIO()
DynamicWrappedIO(java.lang.String classname)

Method Summary

Modifier and Type	Method	Description
boolean	bulkDelete_available()	Are the bulk delete methods available?
java.util.List<java.util.Map.Entry<Path,java.lang.String>>	bulkDelete_delete(FileSystem fs, Path base, java.util.Collection<Path> paths)	Delete a list of files/objects.
int	bulkDelete_pageSize(FileSystem fileSystem, Path path)	Get the maximum number of objects/files to delete in a single request.
boolean	byteBufferPositionedReadable_available()	Are the ByteBufferPositionedReadable methods loaded? This does not check that a specific stream implements the API; use byteBufferPositionedReadable_readFullyAvailable(InputStream).
void	byteBufferPositionedReadable_readFully(java.io.InputStream in, long position, java.nio.ByteBuffer buf)	Delegate to ByteBufferPositionedReadable#read(long, ByteBuffer).
boolean	byteBufferPositionedReadable_readFullyAvailable(java.io.InputStream in)	Probe to see if the input stream is an instance of ByteBufferPositionedReadable.
FSDataInputStream	fileSystem_openFile(FileSystem fs, Path path, java.lang.String policy, FileStatus status, java.lang.Long length, java.util.Map<java.lang.String,java.lang.String> options)	OpenFile assistant, easy reflection-based access to FileSystem#openFile(Path) and blocks awaiting the operation completion.
boolean	fileSystem_openFile_available()	Is the fileSystem_openFile(FileSystem, Path, String, FileStatus, Long, Map) method available.
static DynamicWrappedIO	instance()	Get the singleton instance.
static boolean	isAvailable()	Is the wrapped IO class loaded?
boolean	loaded()	Is the wrapped IO class loaded?
static FSDataInputStream	openFile(FileSystem fs, FileStatus status, java.lang.String readPolicies)	Open a file.
boolean	pathCapabilities_hasPathCapability(java.lang.Object fs, Path path, java.lang.String capability)	Does a path have a given capability? Calls PathCapabilities#hasPathCapability(Path, String), mapping IOExceptions to false.
boolean	streamCapabilities_hasCapability(java.lang.Object object, java.lang.String capability)	Does an object implement StreamCapabilities and, if so, what is the result of the probe for the capability? Calls StreamCapabilities#hasCapability(String),

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_READ_POLICIES

public static final java.lang.String PARQUET_READ_POLICIES

Read policy for parquet files: "parquet, columnar, vector, random".

See Also:

Constant Field Values

Constructor Detail

DynamicWrappedIO

public DynamicWrappedIO()

DynamicWrappedIO

public DynamicWrappedIO(java.lang.String classname)

Method Detail

loaded

public boolean loaded()

Is the wrapped IO class loaded?

Returns:

true if the wrappedIO class was found and loaded.

bulkDelete_available

public boolean bulkDelete_available()

Are the bulk delete methods available?

Returns:

true if the methods were found.

bulkDelete_pageSize

public int bulkDelete_pageSize(FileSystem fileSystem,
                               Path path)
                        throws java.io.IOException

Get the maximum number of objects/files to delete in a single request.

Parameters:

fileSystem - filesystem

path - path to delete under.

Returns:

a number greater than or equal to zero.

Throws:

java.lang.UnsupportedOperationException - bulk delete under that path is not supported.

java.lang.IllegalArgumentException - path not valid.

java.io.IOException - problems resolving paths

java.lang.RuntimeException - invocation failure.

bulkDelete_delete

public java.util.List<java.util.Map.Entry<Path,java.lang.String>> bulkDelete_delete(FileSystem fs,
                                                                                          Path base,
                                                                                          java.util.Collection<Path> paths)
                                                                                   throws java.io.IOException

Delete a list of files/objects.

• Files must be under the path provided in base.
• The size of the list must be equal to or less than the page size.
• Directories are not supported; the outcome of attempting to delete directories is undefined (ignored; undetected, listed as failures...).
• The operation is not atomic.
• The operation is treated as idempotent: network failures may trigger resubmission of the request -any new objects created under a path in the list may then be deleted.
• There is no guarantee that any parent directories exist after this call.

Parameters:

fs - filesystem

base - path to delete under.

paths - list of paths which must be absolute and under the base path.

Returns:

a list of all the paths which couldn't be deleted for a reason other than "not found" and any associated error message.

Throws:

java.lang.UnsupportedOperationException - bulk delete under that path is not supported.

java.lang.IllegalArgumentException - if a path argument is invalid.

java.io.IOException - IO problems including networking, authentication and more.

fileSystem_openFile_available

public boolean fileSystem_openFile_available()

Is the fileSystem_openFile(FileSystem, Path, String, FileStatus, Long, Map) method available.

Returns:

true if the optimized open file method can be invoked.

fileSystem_openFile

public FSDataInputStream fileSystem_openFile(FileSystem fs,
                                             Path path,
                                             java.lang.String policy,
                                             @Nullable
                                             FileStatus status,
                                             @Nullable
                                             java.lang.Long length,
                                             @Nullable
                                             java.util.Map<java.lang.String,java.lang.String> options)
                                      throws java.io.IOException

OpenFile assistant, easy reflection-based access to FileSystem#openFile(Path) and blocks awaiting the operation completion.

Parameters:

fs - filesystem

path - path

policy - read policy

status - optional file status

length - optional file length

options - nullable map of other options

Returns:

stream of the opened file

Throws:

java.io.IOException - if the operation was attempted and failed.

pathCapabilities_hasPathCapability

public boolean pathCapabilities_hasPathCapability(java.lang.Object fs,
                                                  Path path,
                                                  java.lang.String capability)

Does a path have a given capability? Calls PathCapabilities#hasPathCapability(Path, String), mapping IOExceptions to false.

Parameters:

fs - filesystem

path - path to query the capability of.

capability - non-null, non-empty string to query the path for support.

Returns:

true if the capability is supported under that part of the FS false if the method is not loaded or the path lacks the capability.

Throws:

java.lang.IllegalArgumentException - invalid arguments

streamCapabilities_hasCapability

public boolean streamCapabilities_hasCapability(java.lang.Object object,
                                                java.lang.String capability)

Does an object implement StreamCapabilities and, if so, what is the result of the probe for the capability? Calls StreamCapabilities#hasCapability(String),

Parameters:

object - object to probe

capability - capability string

Returns:

true iff the object implements StreamCapabilities and the capability is declared available.

byteBufferPositionedReadable_available

public boolean byteBufferPositionedReadable_available()

Are the ByteBufferPositionedReadable methods loaded? This does not check that a specific stream implements the API; use byteBufferPositionedReadable_readFullyAvailable(InputStream).

Returns:

true if the hadoop libraries have the method.

byteBufferPositionedReadable_readFullyAvailable

public boolean byteBufferPositionedReadable_readFullyAvailable(java.io.InputStream in)
                                                        throws java.io.IOException

Probe to see if the input stream is an instance of ByteBufferPositionedReadable. If the stream is an FSDataInputStream, the wrapped stream is checked.

Parameters:

in - input stream

Returns:

true if the API is available, the stream implements the interface (including the innermost wrapped stream) and that it declares the stream capability.

Throws:

java.io.IOException - if the operation was attempted and failed.

byteBufferPositionedReadable_readFully

public void byteBufferPositionedReadable_readFully(java.io.InputStream in,
                                                   long position,
                                                   java.nio.ByteBuffer buf)
                                            throws java.io.IOException

Delegate to ByteBufferPositionedReadable#read(long, ByteBuffer).

Parameters:

in - input stream

position - position within file

buf - the ByteBuffer to receive the results of the read operation.

Throws:

java.lang.UnsupportedOperationException - if the input doesn't implement the interface or, if when invoked, it is raised. Note: that is the default behaviour of FSDataInputStream#readFully(long, ByteBuffer).

java.io.IOException - if the operation was attempted and failed.

instance

public static DynamicWrappedIO instance()

Get the singleton instance.

Returns:

the instance

isAvailable

public static boolean isAvailable()

Is the wrapped IO class loaded?

Returns:

true if the instance is loaded.

openFile

public static FSDataInputStream openFile(FileSystem fs,
                                         FileStatus status,
                                         java.lang.String readPolicies)
                                  throws java.io.IOException

Open a file.

If the WrappedIO class is found, use it.

If not, falls back to the classic fs.open(Path) call.

Parameters:

fs - filesystem

status - file status

readPolicies - read policy to use

Returns:

the input stream

Throws:

java.io.IOException - any IO failure.