parquet.avro

Class AvroParquetWriter<T extends org.apache.avro.generic.IndexedRecord>

java.lang.Object

parquet.hadoop.ParquetWriter<T>

parquet.avro.AvroParquetWriter<T>

All Implemented Interfaces:

Closeable, AutoCloseable

public class AvroParquetWriter<T extends org.apache.avro.generic.IndexedRecord>
extends ParquetWriter<T>

Write Avro records to a Parquet file.

Field Summary

Fields inherited from class parquet.hadoop.ParquetWriter

DEFAULT_BLOCK_SIZE, DEFAULT_COMPRESSION_CODEC_NAME, DEFAULT_IS_DICTIONARY_ENABLED, DEFAULT_IS_VALIDATING_ENABLED, DEFAULT_PAGE_SIZE, DEFAULT_WRITER_VERSION

Constructor Summary

Constructors

Constructor and Description
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema) Create a new AvroParquetWriter.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, CompressionCodecName compressionCodecName, int blockSize, int pageSize) Create a new AvroParquetWriter.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary) Create a new AvroParquetWriter.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary, org.apache.hadoop.conf.Configuration conf) Create a new AvroParquetWriter.

Method Summary

Methods inherited from class parquet.hadoop.ParquetWriter

close, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroParquetWriter

public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                 org.apache.avro.Schema avroSchema,
                 CompressionCodecName compressionCodecName,
                 int blockSize,
                 int pageSize)
                  throws IOException

Create a new AvroParquetWriter.

Parameters:

file -

avroSchema -

compressionCodecName -

blockSize -

pageSize -

Throws:

IOException

AvroParquetWriter

public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                 org.apache.avro.Schema avroSchema,
                 CompressionCodecName compressionCodecName,
                 int blockSize,
                 int pageSize,
                 boolean enableDictionary)
                  throws IOException

Create a new AvroParquetWriter.

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

compressionCodecName - Compression code to use, or CompressionCodecName.UNCOMPRESSED

blockSize - the block size threshold.

pageSize - See parquet write up. Blocks are subdivided into pages for alignment and other purposes.

enableDictionary - Whether to use a dictionary to compress columns.

Throws:

IOException

AvroParquetWriter

public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                 org.apache.avro.Schema avroSchema)
                  throws IOException

Create a new AvroParquetWriter. The default block size is 50 MB.The default page size is 1 MB. Default compression is no compression. (Inherited from ParquetWriter)

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

Throws:

IOException

AvroParquetWriter

public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                 org.apache.avro.Schema avroSchema,
                 CompressionCodecName compressionCodecName,
                 int blockSize,
                 int pageSize,
                 boolean enableDictionary,
                 org.apache.hadoop.conf.Configuration conf)
                  throws IOException

Create a new AvroParquetWriter.

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

compressionCodecName - Compression code to use, or CompressionCodecName.UNCOMPRESSED

blockSize - the block size threshold.

pageSize - See parquet write up. Blocks are subdivided into pages for alignment and other purposes.

enableDictionary - Whether to use a dictionary to compress columns.

conf - The Configuration to use.

Throws:

IOException