parquet.column.values.dictionary

Class DictionaryValuesWriter

java.lang.Object

parquet.column.values.ValuesWriter

parquet.column.values.dictionary.DictionaryValuesWriter

All Implemented Interfaces:

RequiresFallback

Direct Known Subclasses:

DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter, DictionaryValuesWriter.PlainDoubleDictionaryValuesWriter, DictionaryValuesWriter.PlainFloatDictionaryValuesWriter, DictionaryValuesWriter.PlainIntegerDictionaryValuesWriter, DictionaryValuesWriter.PlainLongDictionaryValuesWriter

public abstract class DictionaryValuesWriter
extends ValuesWriter
implements RequiresFallback

Will attempt to encode values using a dictionary and fall back to plain encoding if the dictionary gets too big

Author:

Julien Le Dem

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainDoubleDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainFixedLenArrayDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainFloatDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainIntegerDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainLongDictionaryValuesWriter

Field Summary

Fields

Modifier and Type	Field and Description
protected int	dictionaryByteSize
protected boolean	dictionaryTooBig
protected IntList	encodedValues
protected Encoding	encodingForDictionaryPage
protected int	lastUsedDictionaryByteSize
protected int	lastUsedDictionarySize
protected int	maxDictionaryByteSize

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	DictionaryValuesWriter(int maxDictionaryByteSize, Encoding encodingForDataPage, Encoding encodingForDictionaryPage)

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract void	clearDictionaryContent() clear/free the underlying dictionary content
protected DictionaryPage	dictPage(ValuesWriter dictionaryEncoder)
void	fallBackAllValuesTo(ValuesWriter writer) When falling back to a different encoding we must re-encode all the values seen so far
protected abstract void	fallBackDictionaryEncodedData(ValuesWriter writer)
long	getAllocatedSize() ( > #getBufferedMemorySize )
long	getBufferedSize() used to decide if we want to work to the next page
BytesInput	getBytes()
protected abstract int	getDictionarySize()
Encoding	getEncoding() called after getBytes() and before reset()
boolean	isCompressionSatisfying(long rawSize, long encodedSize) Before writing the first page we will verify if the encoding is worth it.
String	memUsageString(String prefix)
void	reset() called after getBytes() to reset the current buffer and start writing the next page
void	resetDictionary() reset the dictionary when a new block starts
boolean	shouldFallBack() In the case of a dictionary based encoding we will fallback if the dictionary becomes too big

Methods inherited from class parquet.column.values.ValuesWriter

createDictionaryPage, writeBoolean, writeByte, writeBytes, writeDouble, writeFloat, writeInteger, writeLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

encodingForDictionaryPage

protected final Encoding encodingForDictionaryPage

maxDictionaryByteSize

protected final int maxDictionaryByteSize

dictionaryTooBig

protected boolean dictionaryTooBig

dictionaryByteSize

protected int dictionaryByteSize

lastUsedDictionaryByteSize

protected int lastUsedDictionaryByteSize

lastUsedDictionarySize

protected int lastUsedDictionarySize

encodedValues

protected IntList encodedValues

Constructor Detail

DictionaryValuesWriter

protected DictionaryValuesWriter(int maxDictionaryByteSize,
                      Encoding encodingForDataPage,
                      Encoding encodingForDictionaryPage)

Parameters:

maxDictionaryByteSize -

Method Detail

dictPage

protected DictionaryPage dictPage(ValuesWriter dictionaryEncoder)

shouldFallBack

public boolean shouldFallBack()

Description copied from interface: RequiresFallback

In the case of a dictionary based encoding we will fallback if the dictionary becomes too big

Specified by:

shouldFallBack in interface RequiresFallback

Returns:

true to notify the parent that we should fallback to another encoding

isCompressionSatisfying

public boolean isCompressionSatisfying(long rawSize,
                              long encodedSize)

Description copied from interface: RequiresFallback

Before writing the first page we will verify if the encoding is worth it. and fall back if a simpler encoding would be better in that case

Specified by:

isCompressionSatisfying in interface RequiresFallback

Parameters:

rawSize - the size if encoded with plain

encodedSize - the size as encoded by the current encoding

Returns:

true if we keep this encoding

fallBackAllValuesTo

public void fallBackAllValuesTo(ValuesWriter writer)

Description copied from interface: RequiresFallback

When falling back to a different encoding we must re-encode all the values seen so far

Specified by:

fallBackAllValuesTo in interface RequiresFallback

Parameters:

writer - the new encoder to write the current values to

fallBackDictionaryEncodedData

protected abstract void fallBackDictionaryEncodedData(ValuesWriter writer)

getBufferedSize

public long getBufferedSize()

Description copied from class: ValuesWriter

used to decide if we want to work to the next page

Specified by:

getBufferedSize in class ValuesWriter

Returns:

the size of the currently buffered data (in bytes)

getAllocatedSize

public long getAllocatedSize()

Description copied from class: ValuesWriter

( > #getBufferedMemorySize )

Specified by:

getAllocatedSize in class ValuesWriter

Returns:

the allocated size of the buffer

getBytes

public BytesInput getBytes()

Specified by:

getBytes in class ValuesWriter

Returns:

the bytes buffered so far to write to the current page

getEncoding

public Encoding getEncoding()

Description copied from class: ValuesWriter

called after getBytes() and before reset()

Specified by:

getEncoding in class ValuesWriter

Returns:

the encoding that was used to encode the bytes

reset

public void reset()

Description copied from class: ValuesWriter

called after getBytes() to reset the current buffer and start writing the next page

Specified by:

reset in class ValuesWriter

resetDictionary

public void resetDictionary()

Description copied from class: ValuesWriter

reset the dictionary when a new block starts

Overrides:

resetDictionary in class ValuesWriter

clearDictionaryContent

protected abstract void clearDictionaryContent()

clear/free the underlying dictionary content

getDictionarySize

protected abstract int getDictionarySize()

Returns:

size in items

memUsageString

public String memUsageString(String prefix)

Specified by:

memUsageString in class ValuesWriter