Package org.apache.flink.table.runtime.util.collections.binary

Class AbstractBytesHashMap<K>

java.lang.Object

org.apache.flink.table.runtime.util.collections.binary.BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

org.apache.flink.table.runtime.util.collections.binary.AbstractBytesHashMap<K>

Direct Known Subclasses:

BytesHashMap, WindowBytesHashMap

public abstract class AbstractBytesHashMap<K>
extends BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

Bytes based hash map. It can be used for performing aggregations where the aggregated values are fixed-width, because the data is stored in continuous memory, AggBuffer of variable length cannot be applied to this HashMap. The KeyValue form in hash map is designed to reduce the cost of key fetching in lookup. The memory is divided into two areas:

Bucket area: pointer + hashcode.

• Bytes 0 to 4: a pointer to the record in the record area
• Bytes 4 to 8: key's full 32-bit hashcode

Record area: the actual data in linked list records, a record has four parts:

• Bytes 0 to 4: len(k)
• Bytes 4 to 4 + len(k): key data
• Bytes 4 + len(k) to 8 + len(k): len(v)
• Bytes 8 + len(k) to 8 + len(k) + len(v): value data

BytesHashMap are influenced by Apache Spark BytesToBytesMap.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.table.runtime.util.collections.binary.BytesMap

BytesMap.LookupInfo<K,V>

Field Summary

Fields

Modifier and Type	Field	Description
protected final PagedTypeSerializer<K>	keySerializer	Used to serialize map key into RecordArea's MemorySegments.

Fields inherited from class org.apache.flink.table.runtime.util.collections.binary.BytesMap

BUCKET_SIZE, BUCKET_SIZE_BITS, bucketSegments, ELEMENT_POINT_LENGTH, END_OF_LIST, growthThreshold, INIT_BUCKET_MEMORY_IN_BYTES, lastBucketPosition, LOAD_FACTOR, log2NumBuckets, memoryPool, numBucketsMask, numBucketsMask2, numBucketsPerSegment, numBucketsPerSegmentBits, numBucketsPerSegmentMask, numElements, numSpillFiles, RECORD_EXTRA_LENGTH, recordArea, reservedNumBuffers, reusedKey, reusedValue, segmentSize, spillInBytes, STEP_INCREMENT

Constructor Summary

Constructors

Constructor	Description
AbstractBytesHashMap(Object owner, org.apache.flink.runtime.memory.MemoryManager memoryManager, long memorySize, PagedTypeSerializer<K> keySerializer, int valueArity)
AbstractBytesHashMap(Object owner, org.apache.flink.runtime.memory.MemoryManager memoryManager, long memorySize, PagedTypeSerializer<K> keySerializer, org.apache.flink.table.types.logical.LogicalType[] valueTypes)

Method Summary

Modifier and Type	Method	Description
org.apache.flink.table.data.binary.BinaryRowData	append(BytesMap.LookupInfo<K,org.apache.flink.table.data.binary.BinaryRowData> lookupInfo, org.apache.flink.table.data.binary.BinaryRowData value)	Append an value into the hash map's record area.
void	free()	release the map's record and bucket area's memory segments.
void	free(boolean reservedRecordMemory)
List<org.apache.flink.core.memory.MemorySegment>	getBucketAreaMemorySegments()
KeyValueIterator<K,org.apache.flink.table.data.binary.BinaryRowData>	getEntryIterator(boolean requiresCopy)	Returns an iterator for iterating over the entries of this map.
long	getNumElements()
long	getNumKeys()	Returns the number of keys in this map.
long	getNumSpillFiles()
ArrayList<org.apache.flink.core.memory.MemorySegment>	getRecordAreaMemorySegments()
long	getSpillInBytes()
long	getUsedMemoryInBytes()
void	reset()	reset the map's record and bucket area's memory segments for reusing.

Methods inherited from class org.apache.flink.table.runtime.util.collections.binary.BytesMap

calcSecondHashCode, growAndRehash, initBucketSegments, lookup, resetBucketSegments, returnSegments

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

keySerializer

protected final PagedTypeSerializer<K> keySerializer

Used to serialize map key into RecordArea's MemorySegments.

Constructor Details

AbstractBytesHashMap

public AbstractBytesHashMap(Object owner,
 org.apache.flink.runtime.memory.MemoryManager memoryManager,
 long memorySize,
 PagedTypeSerializer<K> keySerializer,
 org.apache.flink.table.types.logical.LogicalType[] valueTypes)

AbstractBytesHashMap

public AbstractBytesHashMap(Object owner,
 org.apache.flink.runtime.memory.MemoryManager memoryManager,
 long memorySize,
 PagedTypeSerializer<K> keySerializer,
 int valueArity)

Method Details

getNumKeys

public long getNumKeys()

Description copied from class: BytesMap

Returns the number of keys in this map.

Specified by:

getNumKeys in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

append

public org.apache.flink.table.data.binary.BinaryRowData append(BytesMap.LookupInfo<K,org.apache.flink.table.data.binary.BinaryRowData> lookupInfo,
 org.apache.flink.table.data.binary.BinaryRowData value)
                                                        throws IOException

Append an value into the hash map's record area.

Returns:

An BinaryRowData mapping to the memory segments in the map's record area belonging to the newly appended value.

Throws:

EOFException - if the map can't allocate much more memory.

IOException

getNumSpillFiles

public long getNumSpillFiles()

Overrides:

getNumSpillFiles in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

getUsedMemoryInBytes

public long getUsedMemoryInBytes()

getSpillInBytes

public long getSpillInBytes()

Overrides:

getSpillInBytes in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

getNumElements

public long getNumElements()

Overrides:

getNumElements in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

getEntryIterator

public KeyValueIterator<K,org.apache.flink.table.data.binary.BinaryRowData> getEntryIterator(boolean requiresCopy)

Returns an iterator for iterating over the entries of this map.

getRecordAreaMemorySegments

public ArrayList<org.apache.flink.core.memory.MemorySegment> getRecordAreaMemorySegments()

Returns:

the underlying memory segments of the hash map's record area

getBucketAreaMemorySegments

public List<org.apache.flink.core.memory.MemorySegment> getBucketAreaMemorySegments()

free

public void free()

release the map's record and bucket area's memory segments.

free

public void free(boolean reservedRecordMemory)

Overrides:

free in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>

Parameters:

reservedRecordMemory - reserved fixed memory or not.

reset

public void reset()

reset the map's record and bucket area's memory segments for reusing.

Overrides:

reset in class BytesMap<K,org.apache.flink.table.data.binary.BinaryRowData>