Package org.apache.hadoop.hdfs.shortcircuit

Class ShortCircuitCache

java.lang.Object

org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

@Private
public class ShortCircuitCache
extends java.lang.Object
implements java.io.Closeable

The ShortCircuitCache tracks things which the client needs to access HDFS block files via short-circuit. These things include: memory-mapped regions, file descriptors, and shared memory areas for communicating with the DataNode.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	ShortCircuitCache.CacheVisitor
static interface	ShortCircuitCache.ShortCircuitReplicaCreator

Field Summary

Fields

Modifier and Type	Field	Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
ShortCircuitCache(int maxTotalSize, long maxNonMmappedEvictableLifespanMs, int maxEvictableMmapedSize, long maxEvictableMmapedLifespanMs, long mmapRetryTimeoutMs, long staleThresholdMs, int shmInterruptCheckMs)

Method Summary

Modifier and Type	Method	Description
void	accept(ShortCircuitCache.CacheVisitor visitor)
ShortCircuitShm.Slot	allocShmSlot(DatanodeInfo datanode, DomainPeer peer, org.apache.commons.lang3.mutable.MutableBoolean usedPeer, ExtendedBlockId blockId, java.lang.String clientName)	Allocate a new shared memory slot.
void	close()	Close the cache and free all associated resources.
protected ShortCircuitReplicaInfo	fetch(ExtendedBlockId key, org.apache.hadoop.util.Waitable<ShortCircuitReplicaInfo> waitable)	Fetch an existing ReplicaInfo object.
ShortCircuitReplicaInfo	fetchOrCreate(ExtendedBlockId key, ShortCircuitCache.ShortCircuitReplicaCreator creator)	Fetch or create a replica.
void	freeSlot(ShortCircuitShm.Slot slot)	Free a slot immediately.
static ShortCircuitCache	fromConf(DfsClientConf.ShortCircuitConf conf)
DfsClientShmManager	getDfsClientShmManager()
int	getReplicaInfoMapSize()	Can be used in testing to verify whether a read went through SCR, after the read is done and before the stream is closed.
long	getStaleThresholdMs()
void	scheduleSlotReleaser(ShortCircuitShm.Slot slot)	Schedule a shared memory slot to be released.
void	setMaxTotalSize(int maxTotalSize)
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

ShortCircuitCache

public ShortCircuitCache(int maxTotalSize,
                         long maxNonMmappedEvictableLifespanMs,
                         int maxEvictableMmapedSize,
                         long maxEvictableMmapedLifespanMs,
                         long mmapRetryTimeoutMs,
                         long staleThresholdMs,
                         int shmInterruptCheckMs)

Method Detail

fromConf

public static ShortCircuitCache fromConf(DfsClientConf.ShortCircuitConf conf)

getStaleThresholdMs

public long getStaleThresholdMs()

setMaxTotalSize

@VisibleForTesting
public void setMaxTotalSize(int maxTotalSize)

fetchOrCreate

public ShortCircuitReplicaInfo fetchOrCreate(ExtendedBlockId key,
                                             ShortCircuitCache.ShortCircuitReplicaCreator creator)

Fetch or create a replica. You must hold the cache lock while calling this function.

Parameters:

key - Key to use for lookup.

creator - Replica creator callback. Will be called without the cache lock being held.

Returns:

Null if no replica could be found or created. The replica, otherwise.

fetch

@VisibleForTesting
protected ShortCircuitReplicaInfo fetch(ExtendedBlockId key,
                                        org.apache.hadoop.util.Waitable<ShortCircuitReplicaInfo> waitable)
                                 throws org.apache.hadoop.ipc.RetriableException

Fetch an existing ReplicaInfo object.

Parameters:

key - The key that we're using.

waitable - The waitable object to wait on.

Returns:

The existing ReplicaInfo object, or null if there is none.

Throws:

org.apache.hadoop.ipc.RetriableException - If the caller needs to retry.

close

public void close()

Close the cache and free all associated resources.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

accept

@VisibleForTesting
public void accept(ShortCircuitCache.CacheVisitor visitor)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

allocShmSlot

public ShortCircuitShm.Slot allocShmSlot(DatanodeInfo datanode,
                                         DomainPeer peer,
                                         org.apache.commons.lang3.mutable.MutableBoolean usedPeer,
                                         ExtendedBlockId blockId,
                                         java.lang.String clientName)
                                  throws java.io.IOException

Allocate a new shared memory slot.

Parameters:

datanode - The datanode to allocate a shm slot with.

peer - A peer connected to the datanode.

usedPeer - Will be set to true if we use up the provided peer.

blockId - The block id and block pool id of the block we're allocating this slot for.

clientName - The name of the DFSClient allocating the shared memory.

Returns:

Null if short-circuit shared memory is disabled; a short-circuit memory slot otherwise.

Throws:

java.io.IOException - An exception if there was an error talking to the datanode.

freeSlot

public void freeSlot(ShortCircuitShm.Slot slot)

Free a slot immediately. ONLY use this if the DataNode is not yet aware of the slot.

Parameters:

slot - The slot to free.

scheduleSlotReleaser

public void scheduleSlotReleaser(ShortCircuitShm.Slot slot)

Schedule a shared memory slot to be released.

Parameters:

slot - The slot to release.

getDfsClientShmManager

@VisibleForTesting
public DfsClientShmManager getDfsClientShmManager()

getReplicaInfoMapSize

@VisibleForTesting
public int getReplicaInfoMapSize()

Can be used in testing to verify whether a read went through SCR, after the read is done and before the stream is closed.