Package org.apache.hadoop.hdfs.server.namenode

Class INodeFile

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.INode

org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields

org.apache.hadoop.hdfs.server.namenode.INodeFile

All Implemented Interfaces:

java.lang.Comparable<byte[]>, BlockCollection, INodeAttributes, INodeFileAttributes, Diff.Element<byte[]>, org.apache.hadoop.util.LightWeightGSet.LinkedElement

@Private
public class INodeFile
extends INodeWithAdditionalFields
implements INodeFileAttributes, BlockCollection

I-node for closed file.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hdfs.server.namenode.INode

INode.BlocksMapUpdateInfo, INode.Feature, INode.QuotaDelta, INode.ReclaimContext

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.namenode.INodeAttributes

INodeAttributes.SnapshotCopy

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes

INodeFileAttributes.SnapshotCopy

Field Summary

Fields

Modifier and Type	Field	Description
static short	DEFAULT_REPL_FOR_STRIPED_BLOCKS	Erasure Coded striped blocks have replication factor of 1.

Fields inherited from class org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields

features

Fields inherited from class org.apache.hadoop.hdfs.server.namenode.INode

LOG

Constructor Summary

Constructors

Constructor	Description
INodeFile(INodeFile that)

Method Summary

Modifier and Type	Method	Description
void	accept(NamespaceVisitor visitor, int snapshot)	Accept a visitor to visit this INode.
FileWithSnapshotFeature	addSnapshotFeature(FileDiffList diffs)
INodeFile	asFile()	Cast this inode to an INodeFile.
void	cleanSubtree(INode.ReclaimContext reclaimContext, int snapshot, int priorSnapshotId)	Clean the subtree under this inode and collect the blocks from the descents for further block deletion/update.
void	clearBlocks()	Clear all blocks of the file.
void	clearFile(INode.ReclaimContext reclaimContext)
long	collectBlocksBeyondMax(long max, INode.BlocksMapUpdateInfo collectedBlocks, java.util.Set<BlockInfo> toRetain)	Remove full blocks at the end file up to newLength
void	collectBlocksBeyondSnapshot(BlockInfo[] snapshotBlocks, INode.BlocksMapUpdateInfo collectedBlocks)	This function is only called when block list is stored in snapshot diffs.
ContentSummaryComputationContext	computeContentSummary(int snapshotId, ContentSummaryComputationContext summary)	Count subtree content summary with a ContentCounts.
long	computeFileSize()	The same as computeFileSize(null).
long	computeFileSize(boolean includesLastUcBlock, boolean usePreferredBlockSize4LastUcBlock)	Compute file size of the current file.
long	computeFileSize(int snapshotId)	Compute file size of the current file if the given snapshot is null; otherwise, get the file size from the given snapshot.
long	computeFileSizeNotIncludingLastUcBlock()	Compute file size of the current file size but not including the last block if it is under construction.
QuotaCounts	computeQuotaUsage(BlockStoragePolicySuite bsps, byte blockStoragePolicyId, boolean useCache, int lastSnapshotId)	Count subtree Quota.NAMESPACE and Quota.STORAGESPACE usages.
QuotaCounts	computeQuotaUsageWithStriped(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp, QuotaCounts counts)	Compute quota of striped file.
void	convertLastBlockToUC(BlockInfo lastBlock, DatanodeStorageInfo[] locations)	Convert the last block of the collection to an under-construction block and set the locations.
void	destroyAndCollectBlocks(INode.ReclaimContext reclaimContext)	Destroy self and clear everything! If the INode is a file, this method collects its blocks for further block deletion.
void	dumpINodeFile(java.io.PrintWriter out, java.lang.StringBuilder prefix, int snapshotId)
void	dumpTreeRecursively(java.io.PrintWriter out, java.lang.StringBuilder prefix, int snapshotId)	Dump tree recursively.
BlockInfo[]	getBlocks()	Get the blocks (striped or contiguous).
BlockInfo[]	getBlocks(int snapshot)
org.apache.hadoop.hdfs.protocol.BlockType	getBlockType()
FileDiffList	getDiffs()
byte	getErasureCodingPolicyID()
short	getFileReplication()	The same as getFileReplication(null).
short	getFileReplication(int snapshot)
FileUnderConstructionFeature	getFileUnderConstructionFeature()	If the inode contains a FileUnderConstructionFeature, return it; otherwise, return null.
FileWithSnapshotFeature	getFileWithSnapshotFeature()	If feature list contains a FileWithSnapshotFeature, return it; otherwise, return null.
long	getHeaderLong()
BlockInfo	getLastBlock()	Get the last block of the collection.
byte	getLocalStoragePolicyID()
java.lang.String	getName()	Get the name of the collection.
short	getPreferredBlockReplication()	Get block replication for the collection.
long	getPreferredBlockSize()	Get preferred block size for the collection
INodeFileAttributes	getSnapshotINode(int snapshotId)
byte	getStoragePolicyID()
boolean	isFile()	Check whether it's a file.
boolean	isStriped()
boolean	isUnderConstruction()	Is this file under construction?
boolean	isWithSnapshot()	Is this file has the snapshot feature?
INodeFile	loadSnapshotFeature(FileDiffList diffs)	Used by FSImage.
boolean	metadataEquals(INodeFileAttributes other)
int	numBlocks()
void	recordModification(int latestSnapshotId)
void	recordModification(int latestSnapshotId, boolean withBlocks)
void	setBlock(int index, BlockInfo blk)	Set the block (contiguous or striped) at the given index.
INodeFile	setFileReplication(short replication, int latestSnapshotId)	Set the replication factor of this file.
void	setStoragePolicyID(byte storagePolicyId, int latestSnapshotId)
QuotaCounts	storagespaceConsumed(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp)	Compute size consumed by all blocks of the current file, including blocks in its snapshots.
QuotaCounts	storagespaceConsumedContiguous(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp)
QuotaCounts	storagespaceConsumedStriped()
java.lang.String	toDetailString()
static INodeFile	valueOf(INode inode, java.lang.String path)	The same as valueOf(inode, path, false).
static INodeFile	valueOf(INode inode, java.lang.String path, boolean acceptNull)	Cast INode to INodeFile.

Methods inherited from class org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields

addAclFeature, addFeature, addXAttrFeature, getAclFeature, getFeature, getFeatures, getFsPermissionShort, getId, getLocalNameBytes, getNext, getPermissionLong, getPermissionStatus, removeAclFeature, removeFeature, removeXAttrFeature, setAccessTime, setLocalName, setModificationTime, setNext, updateModificationTime

Methods inherited from class org.apache.hadoop.hdfs.server.namenode.INode

addSpaceConsumed, asDirectory, asReference, asSymlink, clear, compareTo, computeAndConvertContentSummary, computeContentSummary, computeQuotaUsage, computeQuotaUsage, dumpINode, dumpParentINodes, dumpTreeRecursively, dumpTreeRecursively, equals, getAccessTime, getAclFeature, getFsPermission, getFullPathAndObjectString, getFullPathName, getGroupName, getKey, getLocalName, getModificationTime, getObjectString, getParent, getParentReference, getParentString, getPathComponents, getPathComponents, getPathNames, getQuotaCounts, getStoragePolicyIDForQuota, getUserName, getXAttrFeature, hashCode, isAncestorDirectory, isDeleted, isDirectory, isInCurrentState, isInLatestSnapshot, isLastReference, isQuotaSet, isReference, isSetStoragePolicy, isSymlink, setAccessTime, setModificationTime, setParent, setParentReference, shouldRecordInSrcSnapshot, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hdfs.server.blockmanagement.BlockCollection

computeContentSummary, getId

Methods inherited from interface org.apache.hadoop.hdfs.server.namenode.INodeAttributes

getAccessTime, getAclFeature, getFsPermission, getFsPermissionShort, getGroupName, getLocalNameBytes, getModificationTime, getPermissionLong, getUserName, getXAttrFeature, isDirectory

Field Detail

DEFAULT_REPL_FOR_STRIPED_BLOCKS

public static final short DEFAULT_REPL_FOR_STRIPED_BLOCKS

Erasure Coded striped blocks have replication factor of 1.

See Also:

Constant Field Values

Constructor Detail

INodeFile

public INodeFile(INodeFile that)

Method Detail

valueOf

public static INodeFile valueOf(INode inode,
                                java.lang.String path)
                         throws java.io.FileNotFoundException

The same as valueOf(inode, path, false).

Throws:

java.io.FileNotFoundException

valueOf

public static INodeFile valueOf(INode inode,
                                java.lang.String path,
                                boolean acceptNull)
                         throws java.io.FileNotFoundException

Cast INode to INodeFile.

Throws:

java.io.FileNotFoundException

isFile

public final boolean isFile()

Description copied from class: INode

Check whether it's a file.

Overrides:

isFile in class INode

Returns:

true unconditionally.

asFile

public final INodeFile asFile()

Description copied from class: INode

Cast this inode to an INodeFile.

Overrides:

asFile in class INode

Returns:

this object.

metadataEquals

public boolean metadataEquals(INodeFileAttributes other)

Specified by:

metadataEquals in interface INodeFileAttributes

getFileUnderConstructionFeature

public final FileUnderConstructionFeature getFileUnderConstructionFeature()

If the inode contains a FileUnderConstructionFeature, return it; otherwise, return null.

isUnderConstruction

public boolean isUnderConstruction()

Is this file under construction?

Specified by:

isUnderConstruction in interface BlockCollection

Returns:

whether the block collection is under construction.

setBlock

public void setBlock(int index,
                     BlockInfo blk)

Description copied from interface: BlockCollection

Set the block (contiguous or striped) at the given index.

Specified by:

setBlock in interface BlockCollection

convertLastBlockToUC

public void convertLastBlockToUC(BlockInfo lastBlock,
                                 DatanodeStorageInfo[] locations)
                          throws java.io.IOException

Description copied from interface: BlockCollection

Convert the last block of the collection to an under-construction block and set the locations.

Specified by:

convertLastBlockToUC in interface BlockCollection

Throws:

java.io.IOException

addSnapshotFeature

public FileWithSnapshotFeature addSnapshotFeature(FileDiffList diffs)

loadSnapshotFeature

public INodeFile loadSnapshotFeature(FileDiffList diffs)

Used by FSImage.

getFileWithSnapshotFeature

public final FileWithSnapshotFeature getFileWithSnapshotFeature()

If feature list contains a FileWithSnapshotFeature, return it; otherwise, return null.

isWithSnapshot

public final boolean isWithSnapshot()

Is this file has the snapshot feature?

toDetailString

public java.lang.String toDetailString()

Overrides:

toDetailString in class INode

getSnapshotINode

public INodeFileAttributes getSnapshotINode(int snapshotId)

Overrides:

getSnapshotINode in class INode

Returns:

if the given snapshot id is Snapshot.CURRENT_STATE_ID, return this; otherwise return the corresponding snapshot inode.

recordModification

public void recordModification(int latestSnapshotId)

recordModification

public void recordModification(int latestSnapshotId,
                               boolean withBlocks)

getDiffs

public FileDiffList getDiffs()

getFileReplication

public final short getFileReplication(int snapshot)

Returns:

the replication factor of the file.

getFileReplication

public final short getFileReplication()

The same as getFileReplication(null). For erasure coded files, this returns the EC policy ID.

Specified by:

getFileReplication in interface INodeFileAttributes

Returns:

the file replication.

getPreferredBlockReplication

public short getPreferredBlockReplication()

Description copied from interface: BlockCollection

Get block replication for the collection.

Specified by:

getPreferredBlockReplication in interface BlockCollection

Returns:

block replication value. Return 0 if the file is erasure coded.

setFileReplication

public final INodeFile setFileReplication(short replication,
                                          int latestSnapshotId)
                                   throws org.apache.hadoop.hdfs.protocol.QuotaExceededException

Set the replication factor of this file.

Throws:

org.apache.hadoop.hdfs.protocol.QuotaExceededException

getPreferredBlockSize

public long getPreferredBlockSize()

Description copied from interface: BlockCollection

Get preferred block size for the collection

Specified by:

getPreferredBlockSize in interface BlockCollection

Specified by:

getPreferredBlockSize in interface INodeFileAttributes

Returns:

preferred block size (in bytes) of the file.

getLocalStoragePolicyID

public byte getLocalStoragePolicyID()

Specified by:

getLocalStoragePolicyID in interface INodeFileAttributes

Specified by:

getLocalStoragePolicyID in class INode

Returns:

the storage policy directly specified on the INode. Return HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED if no policy has been specified.

getStoragePolicyID

public byte getStoragePolicyID()

Specified by:

getStoragePolicyID in interface BlockCollection

Specified by:

getStoragePolicyID in class INode

Returns:

the latest block storage policy id of the INode. Specifically, if a storage policy is directly specified on the INode then return the ID of that policy. Otherwise follow the latest parental path and return the ID of the first specified storage policy.

setStoragePolicyID

public final void setStoragePolicyID(byte storagePolicyId,
                                     int latestSnapshotId)
                              throws org.apache.hadoop.hdfs.protocol.QuotaExceededException

Throws:

org.apache.hadoop.hdfs.protocol.QuotaExceededException

getErasureCodingPolicyID

@VisibleForTesting
public byte getErasureCodingPolicyID()

Specified by:

getErasureCodingPolicyID in interface INodeFileAttributes

Returns:

The ID of the erasure coding policy on the file.

isStriped

@VisibleForTesting
public boolean isStriped()

Specified by:

isStriped in interface BlockCollection

Specified by:

isStriped in interface INodeFileAttributes

Returns:

true if the file is in the striping layout.

getBlockType

@VisibleForTesting
public org.apache.hadoop.hdfs.protocol.BlockType getBlockType()

Specified by:

getBlockType in interface INodeFileAttributes

Returns:

The type of the INodeFile based on block id.

getHeaderLong

public long getHeaderLong()

Specified by:

getHeaderLong in interface INodeFileAttributes

Returns:

the header as a long.

getBlocks

public BlockInfo[] getBlocks()

Description copied from interface: BlockCollection

Get the blocks (striped or contiguous).

Specified by:

getBlocks in interface BlockCollection

Returns:

the blocks of the file.

getBlocks

public BlockInfo[] getBlocks(int snapshot)

Returns:

blocks of the file corresponding to the snapshot.

clearBlocks

public void clearBlocks()

Clear all blocks of the file.

cleanSubtree

public void cleanSubtree(INode.ReclaimContext reclaimContext,
                         int snapshot,
                         int priorSnapshotId)

Description copied from class: INode

Clean the subtree under this inode and collect the blocks from the descents for further block deletion/update. The current inode can either resides in the current tree or be stored as a snapshot copy.

 In general, we have the following rules. 
 1. When deleting a file/directory in the current tree, we have different 
 actions according to the type of the node to delete. 
 
 1.1 The current inode (this) is an INodeFile. 
 1.1.1 If prior is null, there is no snapshot taken on ancestors 
 before. Thus we simply destroy (i.e., to delete completely, no need to save 
 snapshot copy) the current INode and collect its blocks for further 
 cleansing.
 1.1.2 Else do nothing since the current INode will be stored as a snapshot
 copy.
 
 1.2 The current inode is an INodeDirectory.
 1.2.1 If prior is null, there is no snapshot taken on ancestors 
 before. Similarly, we destroy the whole subtree and collect blocks.
 1.2.2 Else do nothing with the current INode. Recursively clean its 
 children.
 
 1.3 The current inode is a file with snapshot.
 Call recordModification(..) to capture the current states.
 Mark the INode as deleted.
 
 1.4 The current inode is an INodeDirectory with snapshot feature.
 Call recordModification(..) to capture the current states. 
 Destroy files/directories created after the latest snapshot 
 (i.e., the inodes stored in the created list of the latest snapshot).
 Recursively clean remaining children. 

 2. When deleting a snapshot.
 2.1 To clean INodeFile: do nothing.
 2.2 To clean INodeDirectory: recursively clean its children.
 2.3 To clean INodeFile with snapshot: delete the corresponding snapshot in
 its diff list.
 2.4 To clean INodeDirectory with snapshot: delete the corresponding 
 snapshot in its diff list. Recursively clean its children.
 

Specified by:

cleanSubtree in class INode

Parameters:

reclaimContext - Record blocks and inodes that need to be reclaimed.

snapshot - The id of the snapshot to delete. Snapshot.CURRENT_STATE_ID means to delete the current file/directory.

priorSnapshotId - The id of the latest snapshot before the to-be-deleted snapshot. When deleting a current inode, this parameter captures the latest snapshot.

destroyAndCollectBlocks

public void destroyAndCollectBlocks(INode.ReclaimContext reclaimContext)

Description copied from class: INode

Destroy self and clear everything! If the INode is a file, this method collects its blocks for further block deletion. If the INode is a directory, the method goes down the subtree and collects blocks from the descents, and clears its parent/children references as well. The method also clears the diff list if the INode contains snapshot diff list.

Specified by:

destroyAndCollectBlocks in class INode

Parameters:

reclaimContext - Record blocks and inodes that need to be reclaimed.

clearFile

public void clearFile(INode.ReclaimContext reclaimContext)

getName

public java.lang.String getName()

Description copied from interface: BlockCollection

Get the name of the collection.

Specified by:

getName in interface BlockCollection

computeQuotaUsage

public final QuotaCounts computeQuotaUsage(BlockStoragePolicySuite bsps,
                                           byte blockStoragePolicyId,
                                           boolean useCache,
                                           int lastSnapshotId)

Description copied from class: INode

Count subtree Quota.NAMESPACE and Quota.STORAGESPACE usages. With the existence of INodeReference, the same inode and its subtree may be referred by multiple INodeReference.WithName nodes and a INodeReference.DstReference node. To avoid circles while quota usage computation, we have the following rules:

 1. For a INodeReference.DstReference node, since the node must be in the current
 tree (or has been deleted as the end point of a series of rename 
 operations), we compute the quota usage of the referred node (and its 
 subtree) in the regular manner, i.e., including every inode in the current
 tree and in snapshot copies, as well as the size of diff list.
 
 2. For a INodeReference.WithName node, since the node must be in a snapshot, we 
 only count the quota usage for those nodes that still existed at the 
 creation time of the snapshot associated with the INodeReference.WithName node.
 We do not count in the size of the diff list.
 

Specified by:

computeQuotaUsage in class INode

Parameters:

bsps - Block storage policy suite to calculate intended storage type usage

blockStoragePolicyId - block storage policy id of the current INode

useCache - Whether to use cached quota usage. Note that INodeReference.WithName node never uses cache for its subtree.

lastSnapshotId - Snapshot.CURRENT_STATE_ID indicates the computation is in the current tree. Otherwise the id indicates the computation range for a INodeReference.WithName node.

Returns:

The subtree quota counts.

computeQuotaUsageWithStriped

public final QuotaCounts computeQuotaUsageWithStriped(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp,
                                                      QuotaCounts counts)

Compute quota of striped file. Note that currently EC files do not support append/hflush/hsync, thus the file length recorded in snapshots should be the same with the current file length.

computeContentSummary

public final ContentSummaryComputationContext computeContentSummary(int snapshotId,
                                                                    ContentSummaryComputationContext summary)

Description copied from class: INode

Count subtree content summary with a ContentCounts.

Specified by:

computeContentSummary in class INode

Parameters:

snapshotId - Specify the time range for the calculation. If this parameter equals to Snapshot.CURRENT_STATE_ID, the result covers both the current states and all the snapshots. Otherwise the result only covers all the files/directories contained in the specific snapshot.

summary - the context object holding counts for the subtree.

Returns:

The same objects as summary.

computeFileSize

public final long computeFileSize()

The same as computeFileSize(null).

computeFileSize

public final long computeFileSize(int snapshotId)

Compute file size of the current file if the given snapshot is null; otherwise, get the file size from the given snapshot.

computeFileSizeNotIncludingLastUcBlock

public final long computeFileSizeNotIncludingLastUcBlock()

Compute file size of the current file size but not including the last block if it is under construction.

computeFileSize

public final long computeFileSize(boolean includesLastUcBlock,
                                  boolean usePreferredBlockSize4LastUcBlock)

Compute file size of the current file.

Parameters:

includesLastUcBlock - If the last block is under construction, should it be included?

usePreferredBlockSize4LastUcBlock - If the last block is under construction, should we use actual block size or preferred block size? Note that usePreferredBlockSize4LastUcBlock is ignored if includesLastUcBlock == false.

Returns:

file size

storagespaceConsumed

public final QuotaCounts storagespaceConsumed(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp)

Compute size consumed by all blocks of the current file, including blocks in its snapshots. Use preferred block size for the last block if it is under construction.

storagespaceConsumedStriped

public final QuotaCounts storagespaceConsumedStriped()

storagespaceConsumedContiguous

public final QuotaCounts storagespaceConsumedContiguous(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy bsp)

getLastBlock

public BlockInfo getLastBlock()

Description copied from interface: BlockCollection

Get the last block of the collection.

Specified by:

getLastBlock in interface BlockCollection

numBlocks

public int numBlocks()

Specified by:

numBlocks in interface BlockCollection

Returns:

the number of blocks or block groups

dumpTreeRecursively

@VisibleForTesting
public void dumpTreeRecursively(java.io.PrintWriter out,
                                java.lang.StringBuilder prefix,
                                int snapshotId)

Description copied from class: INode

Dump tree recursively.

Overrides:

dumpTreeRecursively in class INode

prefix - The prefix string that each line should print.

dumpINodeFile

public void dumpINodeFile(java.io.PrintWriter out,
                          java.lang.StringBuilder prefix,
                          int snapshotId)

accept

public void accept(NamespaceVisitor visitor,
                   int snapshot)

Description copied from class: INode

Accept a visitor to visit this INode.

Overrides:

accept in class INode

collectBlocksBeyondMax

public long collectBlocksBeyondMax(long max,
                                   INode.BlocksMapUpdateInfo collectedBlocks,
                                   java.util.Set<BlockInfo> toRetain)

Remove full blocks at the end file up to newLength

Returns:

sum of sizes of the remained blocks

collectBlocksBeyondSnapshot

public void collectBlocksBeyondSnapshot(BlockInfo[] snapshotBlocks,
                                        INode.BlocksMapUpdateInfo collectedBlocks)

This function is only called when block list is stored in snapshot diffs. Note that this can only happen when truncation happens with snapshots. Since we do not support truncation with striped blocks, we only need to handle contiguous blocks here.