Package org.apache.hadoop.hdfs.server.namenode

Class FileJournalManager

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.FileJournalManager

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, Storage.FormatConfirmable, JournalManager

@Private
public class FileJournalManager
extends java.lang.Object
implements JournalManager

Journal manager for the common case of edits files being written to a storage directory. Note: this class is not thread-safe and should be externally synchronized.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FileJournalManager.EditLogFile	Record of an edit log that has been located and had its filename parsed.

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.namenode.JournalManager

JournalManager.CorruptionException

Constructor Summary

Constructors

Constructor	Description
FileJournalManager(org.apache.hadoop.conf.Configuration conf, Storage.StorageDirectory sd, StorageErrorReporter errorReporter)

Method Summary

Modifier and Type	Method	Description
boolean	canRollBack(StorageInfo storage, StorageInfo prevStorage, int targetLayoutVersion)	Return true if this JM can roll back to the previous storage state, false otherwise.
void	close()	Close the journal manager, freeing any resources it may hold.
void	discardSegments(long startTxid)	Discard the segments whose first txid is >= the given txid.
void	doFinalize()	Finalize the upgrade.
void	doPreUpgrade()	Perform any steps that must succeed across all JournalManagers involved in an upgrade before proceeding onto the actual upgrade stage.
void	doRollback()	Perform the rollback to the previous FS state.
void	doUpgrade(Storage storage)	This method assumes that the fields of the Storage object have already been updated to the appropriate new values for the upgrade.
void	finalizeLogSegment(long firstTxId, long lastTxId)	Mark the log segment that spans from firstTxId to lastTxId as finalized and complete.
void	format(NamespaceInfo ns, boolean force)	Format the underlying storage, removing any previously stored data.
long	getJournalCTime()
long	getLastReadableTxId()
FileJournalManager.EditLogFile	getLogFile(long startTxId)
FileJournalManager.EditLogFile	getLogFile(long startTxId, boolean inProgressOk)
static FileJournalManager.EditLogFile	getLogFile(java.io.File dir, long startTxId)
static FileJournalManager.EditLogFile	getLogFile(java.io.File dir, long startTxId, boolean inProgressOk)
java.util.List<FileJournalManager.EditLogFile>	getLogFiles(long fromTxId)
java.util.List<RemoteEditLog>	getRemoteEditLogs(long firstTxId, boolean inProgressOk)	Find all editlog segments starting at or above the given txid.
Storage.StorageDirectory	getStorageDirectory()
boolean	hasSomeData()
static java.util.List<FileJournalManager.EditLogFile>	matchEditLogs(java.io.File logDir)	returns matching edit logs via the log directory.
void	purgeLogsOlderThan(long minTxIdToKeep)	Purges the unnecessary edits and edits_inprogress files.
void	recoverUnfinalizedSegments()	Recover segments which have not been finalized.
void	selectInputStreams(java.util.Collection<EditLogInputStream> streams, long fromTxnId, boolean inProgressOk)
void	selectInputStreams(java.util.Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns)
void	setLastReadableTxId(long id)
void	setOutputBufferCapacity(int size)	Set the amount of memory that this stream should use to buffer edits
EditLogOutputStream	startLogSegment(long txid, int layoutVersion)	Begin writing to a new segment of the log stream, which starts at the given transaction ID.
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FileJournalManager

public FileJournalManager(org.apache.hadoop.conf.Configuration conf,
                          Storage.StorageDirectory sd,
                          StorageErrorReporter errorReporter)

Method Detail

close

public void close()
           throws java.io.IOException

Description copied from interface: JournalManager

Close the journal manager, freeing any resources it may hold.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface JournalManager

Throws:

java.io.IOException

format

public void format(NamespaceInfo ns,
                   boolean force)
            throws java.io.IOException

Description copied from interface: JournalManager

Format the underlying storage, removing any previously stored data.

Specified by:

format in interface JournalManager

Throws:

java.io.IOException

hasSomeData

public boolean hasSomeData()

Specified by:

hasSomeData in interface Storage.FormatConfirmable

Returns:

true if the storage seems to have some valid data in it, and the user should be required to confirm the format. Otherwise, false.

startLogSegment

public EditLogOutputStream startLogSegment(long txid,
                                           int layoutVersion)
                                    throws java.io.IOException

Description copied from interface: JournalManager

Begin writing to a new segment of the log stream, which starts at the given transaction ID.

Specified by:

startLogSegment in interface JournalManager

Throws:

java.io.IOException

finalizeLogSegment

public void finalizeLogSegment(long firstTxId,
                               long lastTxId)
                        throws java.io.IOException

Description copied from interface: JournalManager

Mark the log segment that spans from firstTxId to lastTxId as finalized and complete.

Specified by:

finalizeLogSegment in interface JournalManager

Throws:

java.io.IOException

getStorageDirectory

@VisibleForTesting
public Storage.StorageDirectory getStorageDirectory()

setOutputBufferCapacity

public void setOutputBufferCapacity(int size)

Description copied from interface: JournalManager

Set the amount of memory that this stream should use to buffer edits

Specified by:

setOutputBufferCapacity in interface JournalManager

getLastReadableTxId

public long getLastReadableTxId()

setLastReadableTxId

public void setLastReadableTxId(long id)

purgeLogsOlderThan

public void purgeLogsOlderThan(long minTxIdToKeep)
                        throws java.io.IOException

Purges the unnecessary edits and edits_inprogress files. Edits files that are ending before the minTxIdToKeep are purged. Edits in progress files that are starting before minTxIdToKeep are purged. Edits in progress files that are marked as empty, trash, corrupted or stale by file extension and starting before minTxIdToKeep are purged. Edits in progress files that are after minTxIdToKeep, but before the current edits in progress files are marked as stale for clarity. In case file removal or rename is failing a warning is logged, but that does not fail the operation.

Parameters:

minTxIdToKeep - the lowest transaction ID that should be retained

Throws:

java.io.IOException - if listing the storage directory fails.

getRemoteEditLogs

public java.util.List<RemoteEditLog> getRemoteEditLogs(long firstTxId,
                                                       boolean inProgressOk)
                                                throws java.io.IOException

Find all editlog segments starting at or above the given txid.

Parameters:

firstTxId - the txnid which to start looking

inProgressOk - whether or not to include the in-progress edit log segment

Returns:

a list of remote edit logs

Throws:

java.io.IOException - if edit logs cannot be listed.

matchEditLogs

public static java.util.List<FileJournalManager.EditLogFile> matchEditLogs(java.io.File logDir)
                                                                    throws java.io.IOException

returns matching edit logs via the log directory. Simple helper function that lists the files in the logDir and calls matchEditLogs(File[])

Parameters:

logDir - directory to match edit logs in

Returns:

matched edit logs

Throws:

java.io.IOException - IOException thrown for invalid logDir

selectInputStreams

public void selectInputStreams(java.util.Collection<EditLogInputStream> streams,
                               long fromTxnId,
                               boolean inProgressOk)
                        throws java.io.IOException

Throws:

java.io.IOException

selectInputStreams

public void selectInputStreams(java.util.Collection<EditLogInputStream> streams,
                               long fromTxId,
                               boolean inProgressOk,
                               boolean onlyDurableTxns)
                        throws java.io.IOException

Throws:

java.io.IOException

recoverUnfinalizedSegments

public void recoverUnfinalizedSegments()
                                throws java.io.IOException

Description copied from interface: JournalManager

Recover segments which have not been finalized.

Specified by:

recoverUnfinalizedSegments in interface JournalManager

Throws:

java.io.IOException

getLogFiles

public java.util.List<FileJournalManager.EditLogFile> getLogFiles(long fromTxId)
                                                           throws java.io.IOException

Throws:

java.io.IOException

getLogFile

public FileJournalManager.EditLogFile getLogFile(long startTxId)
                                          throws java.io.IOException

Throws:

java.io.IOException

getLogFile

public FileJournalManager.EditLogFile getLogFile(long startTxId,
                                                 boolean inProgressOk)
                                          throws java.io.IOException

Throws:

java.io.IOException

getLogFile

public static FileJournalManager.EditLogFile getLogFile(java.io.File dir,
                                                        long startTxId)
                                                 throws java.io.IOException

Throws:

java.io.IOException

getLogFile

public static FileJournalManager.EditLogFile getLogFile(java.io.File dir,
                                                        long startTxId,
                                                        boolean inProgressOk)
                                                 throws java.io.IOException

Throws:

java.io.IOException

toString

public java.lang.String toString()

Specified by:

toString in interface Storage.FormatConfirmable

Overrides:

toString in class java.lang.Object

Returns:

a string representation of the formattable item, suitable for display to the user inside a prompt

doPreUpgrade

public void doPreUpgrade()
                  throws java.io.IOException

Description copied from interface: JournalManager

Perform any steps that must succeed across all JournalManagers involved in an upgrade before proceeding onto the actual upgrade stage. If a call to any JM's doPreUpgrade method fails, then doUpgrade will not be called for any JM.

Specified by:

doPreUpgrade in interface JournalManager

Throws:

java.io.IOException

doUpgrade

public void doUpgrade(Storage storage)
               throws java.io.IOException

This method assumes that the fields of the Storage object have already been updated to the appropriate new values for the upgrade.

Specified by:

doUpgrade in interface JournalManager

Parameters:

storage - info about the new upgraded versions.

Throws:

java.io.IOException

doFinalize

public void doFinalize()
                throws java.io.IOException

Description copied from interface: JournalManager

Finalize the upgrade. JMs should purge any state that they had been keeping around during the upgrade process. After this is completed, rollback is no longer allowed.

Specified by:

doFinalize in interface JournalManager

Throws:

java.io.IOException

canRollBack

public boolean canRollBack(StorageInfo storage,
                           StorageInfo prevStorage,
                           int targetLayoutVersion)
                    throws java.io.IOException

Description copied from interface: JournalManager

Return true if this JM can roll back to the previous storage state, false otherwise. The NN will refuse to run the rollback operation unless at least one JM or fsimage storage directory can roll back.

Specified by:

canRollBack in interface JournalManager

Parameters:

storage - the storage info for the current state

prevStorage - the storage info for the previous (unupgraded) state

targetLayoutVersion - the layout version we intend to roll back to

Returns:

true if this JM can roll back, false otherwise.

Throws:

java.io.IOException

doRollback

public void doRollback()
                throws java.io.IOException

Description copied from interface: JournalManager

Perform the rollback to the previous FS state. JMs which do not need to roll back their state should just return without error.

Specified by:

doRollback in interface JournalManager

Throws:

java.io.IOException

discardSegments

public void discardSegments(long startTxid)
                     throws java.io.IOException

Description copied from interface: JournalManager

Discard the segments whose first txid is >= the given txid.

Specified by:

discardSegments in interface JournalManager

Parameters:

startTxid - The given txid should be right at the segment boundary, i.e., it should be the first txid of some segment, if segment corresponding to the txid exists.

Throws:

java.io.IOException

getJournalCTime

public long getJournalCTime()
                     throws java.io.IOException

Specified by:

getJournalCTime in interface JournalManager

Returns:

the CTime of the journal manager.

Throws:

java.io.IOException