Package org.apache.hadoop.hdfs.server.namenode.sps

Class StoragePolicySatisfier

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier

All Implemented Interfaces:

java.lang.Runnable, SPSService

@Private
public class StoragePolicySatisfier
extends java.lang.Object
implements SPSService, java.lang.Runnable

Setting storagePolicy on a file after the file write will only update the new storage policy type in Namespace, but physical block storage movement will not happen until user runs "Mover Tool" explicitly for such files. The StoragePolicySatisfier Daemon thread implemented for addressing the case where users may want to physically move the blocks by a dedicated daemon (can run inside Namenode or stand alone) instead of running mover tool explicitly. Just calling client API to satisfyStoragePolicy on a file/dir will automatically trigger to move its physical storage locations as expected in asynchronous manner. Here SPS will pick the file blocks which are expecting to change its storages, then it will build the mapping of source block location and expected storage type and location to move. After that this class will also prepare requests to send to Datanode for processing the physical block movements.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	StoragePolicySatisfier.AttemptedItemInfo	This class contains information of an attempted blocks and its last attempted or reported time stamp.
static class	StoragePolicySatisfier.DatanodeMap	Maintains storage type map with the available datanodes in the cluster.
static class	StoragePolicySatisfier.DatanodeWithStorage	Keeps datanode with its respective set of supported storage types.

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
StoragePolicySatisfier(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
void	addAllFilesToProcess(long startPath, java.util.List<ItemInfo> itemInfoList, boolean scanCompleted)	Adds all the Item information(file etc) to processing queue.
void	addFileToProcess(ItemInfo trackInfo, boolean scanCompleted)	Adds the Item information(file etc) to processing queue.
void	clearQueues()	Clear the queues from to be storage movement needed lists and items tracked in storage movement monitor.
BlockStorageMovementAttemptedItems	getAttemptedItemsMonitor()
org.apache.hadoop.conf.Configuration	getConf()
BlockStorageMovementNeeded	getStorageMovementQueue()
void	init(Context context)	Initializes the helper services.
boolean	isRunning()	Check whether StoragePolicySatisfier is running.
void	join()	Join main SPS thread.
void	markScanCompletedForPath(long inodeId)	Marks the scanning of directory if finished.
void	notifyStorageMovementAttemptFinishedBlk(org.apache.hadoop.hdfs.protocol.DatanodeInfo dnInfo, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.hdfs.protocol.Block block)	Receives storage movement attempt finished block report.
int	processingQueueSize()
void	run()
void	start(org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode serviceMode)	Start storage policy satisfier demon thread.
void	stop(boolean forceStop)	Stops the SPS service.
void	stopGracefully()	Stops the SPS service gracefully.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

StoragePolicySatisfier

public StoragePolicySatisfier(org.apache.hadoop.conf.Configuration conf)

Method Detail

init

public void init(Context context)

Description copied from interface: SPSService

Initializes the helper services.

Specified by:

init in interface SPSService

Parameters:

context - - context is an helper service to provide communication channel between NN and SPS

start

public void start(org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode serviceMode)

Start storage policy satisfier demon thread. Also start block storage movements monitor for retry the attempts if needed.

Specified by:

start in interface SPSService

Parameters:

serviceMode - sps service mode

stop

public void stop(boolean forceStop)

Description copied from interface: SPSService

Stops the SPS service.

Specified by:

stop in interface SPSService

Parameters:

forceStop - true represents to clear all the sps path's hint, false otherwise.

stopGracefully

public void stopGracefully()

Description copied from interface: SPSService

Stops the SPS service gracefully. Timed wait to stop storage policy satisfier daemon threads.

Specified by:

stopGracefully in interface SPSService

isRunning

public boolean isRunning()

Description copied from interface: SPSService

Check whether StoragePolicySatisfier is running.

Specified by:

isRunning in interface SPSService

Returns:

true if running

run

public void run()

Specified by:

run in interface java.lang.Runnable

notifyStorageMovementAttemptFinishedBlk

public void notifyStorageMovementAttemptFinishedBlk(org.apache.hadoop.hdfs.protocol.DatanodeInfo dnInfo,
                                                    org.apache.hadoop.fs.StorageType storageType,
                                                    org.apache.hadoop.hdfs.protocol.Block block)

Receives storage movement attempt finished block report.

Specified by:

notifyStorageMovementAttemptFinishedBlk in interface SPSService

Parameters:

dnInfo - reported datanode

storageType - - storage type

block - movement attempt finished block.

getAttemptedItemsMonitor

@VisibleForTesting
public BlockStorageMovementAttemptedItems getAttemptedItemsMonitor()

clearQueues

public void clearQueues()

Clear the queues from to be storage movement needed lists and items tracked in storage movement monitor.

addFileToProcess

public void addFileToProcess(ItemInfo trackInfo,
                             boolean scanCompleted)

Description copied from interface: SPSService

Adds the Item information(file etc) to processing queue.

Specified by:

addFileToProcess in interface SPSService

Parameters:

trackInfo - file info object for which need to satisfy the policy

addAllFilesToProcess

public void addAllFilesToProcess(long startPath,
                                 java.util.List<ItemInfo> itemInfoList,
                                 boolean scanCompleted)

Description copied from interface: SPSService

Adds all the Item information(file etc) to processing queue.

Specified by:

addAllFilesToProcess in interface SPSService

Parameters:

startPath - - directoryId/fileId, on which SPS was called.

itemInfoList - - list of item infos

scanCompleted - - whether the scanning of directory fully done with itemInfoList

processingQueueSize

public int processingQueueSize()

Specified by:

processingQueueSize in interface SPSService

Returns:

current processing queue size.

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface SPSService

Returns:

the configuration.

getStorageMovementQueue

@VisibleForTesting
public BlockStorageMovementNeeded getStorageMovementQueue()

markScanCompletedForPath

public void markScanCompletedForPath(long inodeId)

Description copied from interface: SPSService

Marks the scanning of directory if finished.

Specified by:

markScanCompletedForPath in interface SPSService

Parameters:

inodeId - - satisfier path id

join

public void join()
          throws java.lang.InterruptedException

Join main SPS thread.

Throws:

java.lang.InterruptedException