parquet.column.page

Interface PageWriter

public interface PageWriter

a writer for all the pages of a given column chunk

Author:

Julien Le Dem

Method Summary

Methods

Modifier and Type	Method and Description
long	allocatedSize()
long	getMemSize()
String	memUsageString(String prefix)
void	writeDictionaryPage(DictionaryPage dictionaryPage) writes a dictionary page
void	writePage(BytesInput bytesInput, int valueCount, Statistics<?> statistics, Encoding rlEncoding, Encoding dlEncoding, Encoding valuesEncoding) writes a single page
void	writePageV2(int rowCount, int nullCount, int valueCount, BytesInput repetitionLevels, BytesInput definitionLevels, Encoding dataEncoding, BytesInput data, Statistics<?> statistics) writes a single page in the new format

Method Detail

writePage

void writePage(BytesInput bytesInput,
             int valueCount,
             Statistics<?> statistics,
             Encoding rlEncoding,
             Encoding dlEncoding,
             Encoding valuesEncoding)
               throws IOException

writes a single page

Parameters:

bytesInput - the bytes for the page

valueCount - the number of values in that page

statistics - the statistics for that page

rlEncoding - repetition level encoding

dlEncoding - definition level encoding

valuesEncoding - values encoding

Throws:

IOException

writePageV2

void writePageV2(int rowCount,
               int nullCount,
               int valueCount,
               BytesInput repetitionLevels,
               BytesInput definitionLevels,
               Encoding dataEncoding,
               BytesInput data,
               Statistics<?> statistics)
                 throws IOException

writes a single page in the new format

Parameters:

rowCount - the number of rows in this page

nullCount - the number of null values (out of valueCount)

valueCount - the number of values in that page (there could be multiple values per row for repeated fields)

repetitionLevels - the repetition levels encoded in RLE without any size header

definitionLevels - the definition levels encoded in RLE without any size header

dataEncoding - the encoding for the data

data - the data encoded with dataEncoding

statistics - optional stats for this page

metadata - optional free form key values

Throws:

IOException

getMemSize

long getMemSize()

Returns:

the current size used in the memory buffer for that column chunk

allocatedSize

long allocatedSize()

Returns:

the allocated size for the buffer ( > getMemSize() )

writeDictionaryPage

void writeDictionaryPage(DictionaryPage dictionaryPage)
                         throws IOException

writes a dictionary page

Parameters:

dictionaryPage - the dictionary page containing the dictionary data

Throws:

IOException

memUsageString

String memUsageString(String prefix)

Parameters:

prefix - a prefix header to add at every line

Returns:

a string presenting a summary of how memory is used