Package org.apache.hadoop.hdfs.server.namenode

Class FSEditLog

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.FSEditLog

@Private
@Evolving
public class FSEditLog
extends java.lang.Object

FSEditLog maintains a log of the namespace modifications.

Field Summary

Fields

Modifier and Type	Field	Description
protected FSEditLogOp.OpInstanceCache	cache
static org.slf4j.Logger	LOG

Method Summary

Modifier and Type	Method	Description
protected void	beginTransaction(FSEditLogOp op)
boolean	canRollBackSharedLog(StorageInfo prevStorage, int targetLayoutVersion)
void	discardSegments(long markerTxid)
void	doFinalizeOfSharedLog()
void	doPreUpgradeOfSharedLog()
void	doRollback()
void	doUpgradeOfSharedLog()
void	endCurrentLogSegment(boolean writeEndTxn)	Finalize the current log segment.
long	getCurSegmentTxId()
RemoteEditLogManifest	getEditLogManifest(long fromTxId)	Return a manifest of what finalized edit logs are available
JournalSet	getJournalSet()	Used only by tests.
long	getLastWrittenTxId()	Return the transaction ID of the last transaction written to the log.
long	getSharedLogCTime()
long	getSyncTxId()	Return the txid of the last synced transaction.
long	getTotalSyncCount()	Return total number of syncs happened on this edit log.
void	initJournalsForWrite()
void	initSharedJournalsForRead()
boolean	isOpenForRead()
void	journal(long firstTxId, int numTxns, byte[] data)	Write the batch of edits to edit log.
void	logAddBlock(java.lang.String path, INodeFile file)
void	logAppendFile(java.lang.String path, INodeFile file, boolean newBlock, boolean toLogRpcIds)
void	logCloseFile(java.lang.String path, INodeFile newNode)	Add close lease record to edit log.
void	logMkDir(java.lang.String path, INode newNode)	Add create directory record to edit log
void	logOpenFile(java.lang.String path, INodeFile newNode, boolean overwrite, boolean toLogRpcIds)	Add open lease record to edit log.
void	logSync()	Sync all modifications done by this thread.
protected void	logSync(long mytxid)
void	logUpdateBlocks(java.lang.String path, INodeFile file, boolean toLogRpcIds)
void	purgeLogsOlderThan(long minTxIdToKeep)	Archive any log files that are older than the given txid.
void	restart()
java.util.Collection<EditLogInputStream>	selectInputStreams(long fromTxId, long toAtLeastTxId)
java.util.Collection<EditLogInputStream>	selectInputStreams(long fromTxId, long toAtLeastTxId, MetaRecoveryContext recovery, boolean inProgressOK)
java.util.Collection<EditLogInputStream>	selectInputStreams(long fromTxId, long toAtLeastTxId, MetaRecoveryContext recovery, boolean inProgressOk, boolean onlyDurableTxns)	Select a list of input streams.
void	selectInputStreams(java.util.Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns)
void	startLogSegment(long txid, boolean abortCurrentLogSegment, int layoutVersion)	Remote namenode just has started a log segment, start log segment locally.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

cache

protected final FSEditLogOp.OpInstanceCache cache

Method Detail

initJournalsForWrite

public void initJournalsForWrite()

initSharedJournalsForRead

public void initSharedJournalsForRead()

isOpenForRead

public boolean isOpenForRead()

Returns:

true if the log is open in read mode.

beginTransaction

protected void beginTransaction(FSEditLogOp op)

getLastWrittenTxId

public long getLastWrittenTxId()

Return the transaction ID of the last transaction written to the log.

getCurSegmentTxId

@VisibleForTesting
public long getCurSegmentTxId()

Returns:

the first transaction ID in the current log segment

logSync

public void logSync()

Sync all modifications done by this thread. The internal concurrency design of this class is as follows: - Log items are written synchronized into an in-memory buffer, and each assigned a transaction ID. - When a thread (client) would like to sync all of its edits, logSync() uses a ThreadLocal transaction ID to determine what edit number must be synced to. - The isSyncRunning volatile boolean tracks whether a sync is currently under progress. The data is double-buffered within each edit log implementation so that in-memory writing can occur in parallel with the on-disk writing. Each sync occurs in three steps: 1. synchronized, it swaps the double buffer and sets the isSyncRunning flag. 2. unsynchronized, it flushes the data to storage 3. synchronized, it resets the flag and notifies anyone waiting on the sync. The lack of synchronization on step 2 allows other threads to continue to write into the memory buffer while the sync is in progress. Because this step is unsynchronized, actions that need to avoid concurrency with sync() should be synchronized and also call waitForSyncToFinish() before assuming they are running alone.

logSync

protected void logSync(long mytxid)

logAppendFile

public void logAppendFile(java.lang.String path,
                          INodeFile file,
                          boolean newBlock,
                          boolean toLogRpcIds)

logOpenFile

public void logOpenFile(java.lang.String path,
                        INodeFile newNode,
                        boolean overwrite,
                        boolean toLogRpcIds)

Add open lease record to edit log. Records the block locations of the last block.

logCloseFile

public void logCloseFile(java.lang.String path,
                         INodeFile newNode)

Add close lease record to edit log.

logAddBlock

public void logAddBlock(java.lang.String path,
                        INodeFile file)

logUpdateBlocks

public void logUpdateBlocks(java.lang.String path,
                            INodeFile file,
                            boolean toLogRpcIds)

logMkDir

public void logMkDir(java.lang.String path,
                     INode newNode)

Add create directory record to edit log

getJournalSet

@VisibleForTesting
public JournalSet getJournalSet()

Used only by tests.

getEditLogManifest

public RemoteEditLogManifest getEditLogManifest(long fromTxId)
                                         throws java.io.IOException

Return a manifest of what finalized edit logs are available

Throws:

java.io.IOException

startLogSegment

public void startLogSegment(long txid,
                            boolean abortCurrentLogSegment,
                            int layoutVersion)
                     throws java.io.IOException

Remote namenode just has started a log segment, start log segment locally.

Throws:

java.io.IOException

endCurrentLogSegment

public void endCurrentLogSegment(boolean writeEndTxn)

Finalize the current log segment. Transitions from IN_SEGMENT state to BETWEEN_LOG_SEGMENTS state.

purgeLogsOlderThan

public void purgeLogsOlderThan(long minTxIdToKeep)

Archive any log files that are older than the given txid. If the edit log is not open for write, then this call returns with no effect.

getSyncTxId

public long getSyncTxId()

Return the txid of the last synced transaction.

journal

public void journal(long firstTxId,
                    int numTxns,
                    byte[] data)

Write the batch of edits to edit log.

getSharedLogCTime

public long getSharedLogCTime()
                       throws java.io.IOException

Throws:

java.io.IOException

doPreUpgradeOfSharedLog

public void doPreUpgradeOfSharedLog()
                             throws java.io.IOException

Throws:

java.io.IOException

doUpgradeOfSharedLog

public void doUpgradeOfSharedLog()
                          throws java.io.IOException

Throws:

java.io.IOException

doFinalizeOfSharedLog

public void doFinalizeOfSharedLog()
                           throws java.io.IOException

Throws:

java.io.IOException

canRollBackSharedLog

public boolean canRollBackSharedLog(StorageInfo prevStorage,
                                    int targetLayoutVersion)
                             throws java.io.IOException

Throws:

java.io.IOException

doRollback

public void doRollback()
                throws java.io.IOException

Throws:

java.io.IOException

discardSegments

public void discardSegments(long markerTxid)
                     throws java.io.IOException

Throws:

java.io.IOException

selectInputStreams

public void selectInputStreams(java.util.Collection<EditLogInputStream> streams,
                               long fromTxId,
                               boolean inProgressOk,
                               boolean onlyDurableTxns)
                        throws java.io.IOException

Throws:

java.io.IOException

selectInputStreams

public java.util.Collection<EditLogInputStream> selectInputStreams(long fromTxId,
                                                                   long toAtLeastTxId)
                                                            throws java.io.IOException

Throws:

java.io.IOException

selectInputStreams

public java.util.Collection<EditLogInputStream> selectInputStreams(long fromTxId,
                                                                   long toAtLeastTxId,
                                                                   MetaRecoveryContext recovery,
                                                                   boolean inProgressOK)
                                                            throws java.io.IOException

Throws:

java.io.IOException

selectInputStreams

public java.util.Collection<EditLogInputStream> selectInputStreams(long fromTxId,
                                                                   long toAtLeastTxId,
                                                                   MetaRecoveryContext recovery,
                                                                   boolean inProgressOk,
                                                                   boolean onlyDurableTxns)
                                                            throws java.io.IOException

Select a list of input streams.

Parameters:

fromTxId - first transaction in the selected streams

toAtLeastTxId - the selected streams must contain this transaction

recovery - recovery context

inProgressOk - set to true if in-progress streams are OK

onlyDurableTxns - set to true if streams are bounded by the durable TxId

Throws:

java.io.IOException

restart

@VisibleForTesting
public void restart()

getTotalSyncCount

public long getTotalSyncCount()

Return total number of syncs happened on this edit log.

Returns:

long - count