org.apache.hadoop.fs.http.client

Class HttpFSFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.http.client.HttpFSFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.DelegationTokenRenewer.Renewable

Direct Known Subclasses:

HttpsFSFileSystem

@InterfaceAudience.Private
public class HttpFSFileSystem
extends org.apache.hadoop.fs.FileSystem
implements org.apache.hadoop.fs.DelegationTokenRenewer.Renewable

HttpFSServer implementation of the FileSystemAccess FileSystem.

This implementation allows a user to access HDFS over HTTP via a HttpFSServer server.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HttpFSFileSystem.FILE_TYPE
static class	HttpFSFileSystem.Operation

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACCESS_TIME_JSON
static String	ACCESS_TIME_PARAM
static String	BLOCK_SIZE_JSON
static String	BLOCKSIZE_PARAM
static String	CHECKSUM_ALGORITHM_JSON
static String	CHECKSUM_BYTES_JSON
static String	CHECKSUM_LENGTH_JSON
static String	CONTENT_SUMMARY_DIRECTORY_COUNT_JSON
static String	CONTENT_SUMMARY_FILE_COUNT_JSON
static String	CONTENT_SUMMARY_JSON
static String	CONTENT_SUMMARY_LENGTH_JSON
static String	CONTENT_SUMMARY_QUOTA_JSON
static String	CONTENT_SUMMARY_SPACE_CONSUMED_JSON
static String	CONTENT_SUMMARY_SPACE_QUOTA_JSON
static Short	DEFAULT_PERMISSION
static String	DELETE_JSON
static String	DESTINATION_PARAM
static String	DO_AS_PARAM
static String	ERROR_CLASSNAME_JSON
static String	ERROR_EXCEPTION_JSON
static String	ERROR_JSON
static String	ERROR_MESSAGE_JSON
static String	FILE_CHECKSUM_JSON
static String	FILE_STATUS_JSON
static String	FILE_STATUSES_JSON
static String	GROUP_JSON
static String	GROUP_PARAM
static String	HOME_DIR_JSON
static int	HTTP_TEMPORARY_REDIRECT
static String	LENGTH_JSON
static String	MKDIRS_JSON
static String	MODIFICATION_TIME_JSON
static String	MODIFICATION_TIME_PARAM
static String	OP_PARAM
static String	OVERWRITE_PARAM
static String	OWNER_JSON
static String	OWNER_PARAM
static String	PATH_SUFFIX_JSON
static String	PERMISSION_JSON
static String	PERMISSION_PARAM
static String	RECURSIVE_PARAM
static String	RENAME_JSON
static String	REPLICATION_JSON
static String	REPLICATION_PARAM
static String	SCHEME
static String	SERVICE_NAME
static String	SERVICE_VERSION
static String	SET_REPLICATION_JSON
static String	SOURCES_PARAM
static String	TYPE_JSON
static String	UPLOAD_CONTENT_TYPE

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
HttpFSFileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) Append to an existing file (optional operation).
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<?> token)
void	concat(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.Path[] psrcs) Concat existing files together.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(org.apache.hadoop.fs.Path f) Deprecated. Use delete(Path, boolean) instead
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive) Delete a file.
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)
protected int	getDefaultPort() Get the default port for this file system.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(String renewer)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f) Return a file status object that represents the path.
org.apache.hadoop.fs.Path	getHomeDirectory() Return the current user's home directory in this filesystem.
org.apache.hadoop.security.token.Token<?>	getRenewToken()
String	getScheme()
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
org.apache.hadoop.fs.Path	getWorkingDirectory() Get the current working directory for the given file system
void	initialize(URI name, org.apache.hadoop.conf.Configuration conf) Called after a new FileSystem instance is constructed.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission) Make the given file and all non-existent parents into directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
static String	permissionToString(org.apache.hadoop.fs.permission.FsPermission p) Converts a FsPermission to a Unix octal representation.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames Path src to Path dst.
long	renewDelegationToken(org.apache.hadoop.security.token.Token<?> token)
<T extends org.apache.hadoop.security.token.TokenIdentifier> void	setDelegationToken(org.apache.hadoop.security.token.Token<T> token)
void	setOwner(org.apache.hadoop.fs.Path p, String username, String groupname) Set owner of a path (i.e.
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission) Set permission of a path.
boolean	setReplication(org.apache.hadoop.fs.Path src, short replication) Set replication for an existing file.
void	setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime) Set access time of a file
void	setWorkingDirectory(org.apache.hadoop.fs.Path newDir) Set the current working directory for the given file system.

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFid, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathId, createSnapshot, createSnapshot, createSymlink, deleteFid, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileLinkStatus, getFileSystemClass, getFSofPath, getInitialWorkingDirectory, getJobTrackerAddrs, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getZkConnectString, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, makeQualified, mkdirs, mkdirs, mkdirsFid, mkdirsFid, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, openFid, openFid, openFid2, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwnerFid, setVerifyChecksum, setWriteChecksum, startLocalOutput, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SERVICE_NAME

public static final String SERVICE_NAME

See Also:

Constant Field Values

SERVICE_VERSION

public static final String SERVICE_VERSION

See Also:

Constant Field Values

SCHEME

public static final String SCHEME

See Also:

Constant Field Values

OP_PARAM

public static final String OP_PARAM

See Also:

Constant Field Values

DO_AS_PARAM

public static final String DO_AS_PARAM

See Also:

Constant Field Values

OVERWRITE_PARAM

public static final String OVERWRITE_PARAM

See Also:

Constant Field Values

REPLICATION_PARAM

public static final String REPLICATION_PARAM

See Also:

Constant Field Values

BLOCKSIZE_PARAM

public static final String BLOCKSIZE_PARAM

See Also:

Constant Field Values

PERMISSION_PARAM

public static final String PERMISSION_PARAM

See Also:

Constant Field Values

DESTINATION_PARAM

public static final String DESTINATION_PARAM

See Also:

Constant Field Values

RECURSIVE_PARAM

public static final String RECURSIVE_PARAM

See Also:

Constant Field Values

SOURCES_PARAM

public static final String SOURCES_PARAM

See Also:

Constant Field Values

OWNER_PARAM

public static final String OWNER_PARAM

See Also:

Constant Field Values

GROUP_PARAM

public static final String GROUP_PARAM

See Also:

Constant Field Values

MODIFICATION_TIME_PARAM

public static final String MODIFICATION_TIME_PARAM

See Also:

Constant Field Values

ACCESS_TIME_PARAM

public static final String ACCESS_TIME_PARAM

See Also:

Constant Field Values

DEFAULT_PERMISSION

public static final Short DEFAULT_PERMISSION

RENAME_JSON

public static final String RENAME_JSON

See Also:

Constant Field Values

DELETE_JSON

public static final String DELETE_JSON

See Also:

Constant Field Values

MKDIRS_JSON

public static final String MKDIRS_JSON

See Also:

Constant Field Values

HOME_DIR_JSON

public static final String HOME_DIR_JSON

See Also:

Constant Field Values

SET_REPLICATION_JSON

public static final String SET_REPLICATION_JSON

See Also:

Constant Field Values

UPLOAD_CONTENT_TYPE

public static final String UPLOAD_CONTENT_TYPE

See Also:

Constant Field Values

FILE_STATUSES_JSON

public static final String FILE_STATUSES_JSON

See Also:

Constant Field Values

FILE_STATUS_JSON

public static final String FILE_STATUS_JSON

See Also:

Constant Field Values

PATH_SUFFIX_JSON

public static final String PATH_SUFFIX_JSON

See Also:

Constant Field Values

TYPE_JSON

public static final String TYPE_JSON

See Also:

Constant Field Values

LENGTH_JSON

public static final String LENGTH_JSON

See Also:

Constant Field Values

OWNER_JSON

public static final String OWNER_JSON

See Also:

Constant Field Values

GROUP_JSON

public static final String GROUP_JSON

See Also:

Constant Field Values

PERMISSION_JSON

public static final String PERMISSION_JSON

See Also:

Constant Field Values

ACCESS_TIME_JSON

public static final String ACCESS_TIME_JSON

See Also:

Constant Field Values

MODIFICATION_TIME_JSON

public static final String MODIFICATION_TIME_JSON

See Also:

Constant Field Values

BLOCK_SIZE_JSON

public static final String BLOCK_SIZE_JSON

See Also:

Constant Field Values

REPLICATION_JSON

public static final String REPLICATION_JSON

See Also:

Constant Field Values

FILE_CHECKSUM_JSON

public static final String FILE_CHECKSUM_JSON

See Also:

Constant Field Values

CHECKSUM_ALGORITHM_JSON

public static final String CHECKSUM_ALGORITHM_JSON

See Also:

Constant Field Values

CHECKSUM_BYTES_JSON

public static final String CHECKSUM_BYTES_JSON

See Also:

Constant Field Values

CHECKSUM_LENGTH_JSON

public static final String CHECKSUM_LENGTH_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_JSON

public static final String CONTENT_SUMMARY_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_DIRECTORY_COUNT_JSON

public static final String CONTENT_SUMMARY_DIRECTORY_COUNT_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_FILE_COUNT_JSON

public static final String CONTENT_SUMMARY_FILE_COUNT_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_LENGTH_JSON

public static final String CONTENT_SUMMARY_LENGTH_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_QUOTA_JSON

public static final String CONTENT_SUMMARY_QUOTA_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_SPACE_CONSUMED_JSON

public static final String CONTENT_SUMMARY_SPACE_CONSUMED_JSON

See Also:

Constant Field Values

CONTENT_SUMMARY_SPACE_QUOTA_JSON

public static final String CONTENT_SUMMARY_SPACE_QUOTA_JSON

See Also:

Constant Field Values

ERROR_JSON

public static final String ERROR_JSON

See Also:

Constant Field Values

ERROR_EXCEPTION_JSON

public static final String ERROR_EXCEPTION_JSON

See Also:

Constant Field Values

ERROR_CLASSNAME_JSON

public static final String ERROR_CLASSNAME_JSON

See Also:

Constant Field Values

ERROR_MESSAGE_JSON

public static final String ERROR_MESSAGE_JSON

See Also:

Constant Field Values

HTTP_TEMPORARY_REDIRECT

public static final int HTTP_TEMPORARY_REDIRECT

See Also:

Constant Field Values

Constructor Detail

HttpFSFileSystem

public HttpFSFileSystem()

Method Detail

initialize

public void initialize(URI name,
              org.apache.hadoop.conf.Configuration conf)
                throws IOException

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

name - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

Throws:

IOException

getScheme

public String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

getUri

public URI getUri()

Returns a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

Returns:

the URI whose scheme and authority identify this FileSystem.

getDefaultPort

protected int getDefaultPort()

Get the default port for this file system.

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

Returns:

the default port or 0 if there isn't one

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                          int bufferSize)
                                            throws IOException

Opens an FSDataInputStream at the indicated Path.

IMPORTANT: the returned does not support the PositionReadable and Seekable methods.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException

permissionToString

public static String permissionToString(org.apache.hadoop.fs.permission.FsPermission p)

Converts a FsPermission to a Unix octal representation.

Parameters:

p - the permission.

Returns:

the Unix string symbolic reprentation.

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                             org.apache.hadoop.fs.permission.FsPermission permission,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Opens an FSDataOutputStream at the indicated Path with write-progress reporting.

IMPORTANT: The Progressable parameter is not used.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open.

permission - file permission.

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size.

progress - progressable.

Throws:

IOException

See Also:

setPermission(Path, FsPermission)

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                             int bufferSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Append to an existing file (optional operation).

IMPORTANT: The Progressable parameter is not used.

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException

concat

public void concat(org.apache.hadoop.fs.Path f,
          org.apache.hadoop.fs.Path[] psrcs)
            throws IOException

Concat existing files together.

Overrides:

concat in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path to the target destination.

psrcs - the paths to the sources to use for the concatenation.

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
             org.apache.hadoop.fs.Path dst)
               throws IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

@Deprecated
public boolean delete(org.apache.hadoop.fs.Path f)
               throws IOException

Deprecated. Use delete(Path, boolean) instead

Delete a file.

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
             boolean recursive)
               throws IOException

Delete a file.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

newDir - new directory.

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Get the current working directory for the given file system

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

the directory pathname

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
             org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

Return the current user's home directory in this filesystem. The default implementation returns "/user/$USER/".

Overrides:

getHomeDirectory in class org.apache.hadoop.fs.FileSystem

setOwner

public void setOwner(org.apache.hadoop.fs.Path p,
            String username,
            String groupname)
              throws IOException

Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Parameters:

p - The path

username - If it is null, the original username remains unchanged.

groupname - If it is null, the original groupname remains unchanged.

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path p,
                 org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Set permission of a path.

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Parameters:

p - path.

permission - permission.

Throws:

IOException

setTimes

public void setTimes(org.apache.hadoop.fs.Path p,
            long mtime,
            long atime)
              throws IOException

Set access time of a file

Overrides:

setTimes in class org.apache.hadoop.fs.FileSystem

Parameters:

p - The path

mtime - Set the modification time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set modification time.

atime - Set the access time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set access time.

Throws:

IOException

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path src,
                     short replication)
                       throws IOException

Set replication for an existing file.

Overrides:

setReplication in class org.apache.hadoop.fs.FileSystem

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException

getContentSummary

public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                      throws IOException

Overrides:

getContentSummary in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileChecksum

public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f)
                                                  throws IOException

Overrides:

getFileChecksum in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getDelegationToken

public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer)
                                                             throws IOException

Overrides:

getDelegationToken in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

renewDelegationToken

public long renewDelegationToken(org.apache.hadoop.security.token.Token<?> token)
                          throws IOException

Throws:

IOException

cancelDelegationToken

public void cancelDelegationToken(org.apache.hadoop.security.token.Token<?> token)
                           throws IOException

Throws:

IOException

getRenewToken

public org.apache.hadoop.security.token.Token<?> getRenewToken()

Specified by:

getRenewToken in interface org.apache.hadoop.fs.DelegationTokenRenewer.Renewable

setDelegationToken

public <T extends org.apache.hadoop.security.token.TokenIdentifier> void setDelegationToken(org.apache.hadoop.security.token.Token<T> token)

Specified by:

setDelegationToken in interface org.apache.hadoop.fs.DelegationTokenRenewer.Renewable