class MultiFileCloudOrcPartitionReader extends MultiFileCloudPartitionReaderBase with MultiFileReaderFunctions with OrcPartitionReaderBase
A PartitionReader that can read multiple ORC files in parallel. This is most efficient running in a cloud environment where the I/O of reading is slow.
Efficiently reading a ORC split on the GPU requires re-constructing the ORC file in memory that contains just the Stripes that are needed. This avoids sending unnecessary data to the GPU and saves GPU memory.
- Alphabetic
- By Inheritance
- MultiFileCloudOrcPartitionReader
- OrcPartitionReaderBase
- OrcCommonFunctions
- OrcCodecWritingHelper
- MultiFileReaderFunctions
- MultiFileCloudPartitionReaderBase
- FilePartitionReaderBase
- Arm
- ScanWithMetrics
- Logging
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
MultiFileCloudOrcPartitionReader(conf: Configuration, files: Array[PartitionedFile], dataSchema: StructType, readDataSchema: StructType, partitionSchema: StructType, maxReadBatchSizeRows: Integer, maxReadBatchSizeBytes: Long, numThreads: Int, maxNumFileProcessed: Int, debugDumpPrefix: Option[String], filters: Array[Filter], filterHandler: GpuOrcFileFilterHandler, execMetrics: Map[String, GpuMetric], ignoreMissingFiles: Boolean, ignoreCorruptFiles: Boolean)
- conf
the Hadoop configuration
- files
the partitioned files to read
- dataSchema
schema of the data
- readDataSchema
the Spark schema describing what will be read
- partitionSchema
Schema of partitions.
- maxReadBatchSizeRows
soft limit on the maximum number of rows the reader reads per batch
- maxReadBatchSizeBytes
soft limit on the maximum number of bytes the reader reads per batch
- numThreads
the size of the threadpool
- maxNumFileProcessed
threshold to control the maximum file number to be submitted to threadpool
- debugDumpPrefix
a path prefix to use for dumping the fabricated ORC data or null
- filters
filters passed into the filterHandler
- filterHandler
used to filter the ORC stripes
- execMetrics
the metrics
- ignoreMissingFiles
Whether to ignore missing files
- ignoreCorruptFiles
Whether to ignore corrupt files
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
addPartitionValues(batch: Option[ColumnarBatch], inPartitionValues: InternalRow, partitionSchema: StructType): Option[ColumnarBatch]
- Attributes
- protected
- Definition Classes
- MultiFileReaderFunctions
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
var
batch: Option[ColumnarBatch]
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
def
buildReaderSchema(updatedSchema: TypeDescription, requestedMapping: Option[Array[Int]]): TypeDescription
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
buildReaderSchema(ctx: OrcPartitionReaderContext): TypeDescription
Get the ORC schema corresponding to the file being constructed for the GPU
Get the ORC schema corresponding to the file being constructed for the GPU
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
def
close(): Unit
- Definition Classes
- MultiFileCloudPartitionReaderBase → FilePartitionReaderBase → Closeable → AutoCloseable
-
def
closeOnExcept[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, closing the resource only if an exception occurs
Executes the provided code block, closing the resource only if an exception occurs
- Definition Classes
- Arm
-
def
copyStripeData(ctx: OrcPartitionReaderContext, out: WritableByteChannel, inputDataRanges: DiskRangeList): Unit
Copy the stripe to the channel
Copy the stripe to the channel
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
var
currentFileHostBuffers: Option[HostMemoryBuffersWithMetaDataBase]
- Attributes
- protected
- Definition Classes
- MultiFileCloudPartitionReaderBase
-
def
currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- PartitionReader
-
val
debugDumpPrefix: Option[String]
- Definition Classes
- MultiFileCloudOrcPartitionReader → OrcCommonFunctions
-
final
def
decodeToBatch(hostBuf: HostMemoryBuffer, bufSize: Long, memFileSchema: TypeDescription, requestedMapping: Option[Array[Int]], isCaseSensitive: Boolean, splits: Array[PartitionedFile]): Option[ColumnarBatch]
Send a host buffer to GPU for ORC decoding, and return it as a ColumnarBatch.
Send a host buffer to GPU for ORC decoding, and return it as a ColumnarBatch. The input hostBuf will be closed after returning, please do not use it anymore. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- OrcPartitionReaderBase
-
def
decodeToTable(hostBuf: HostMemoryBuffer, bufSize: Long, memFileSchema: TypeDescription, requestedMapping: Option[Array[Int]], isCaseSensitive: Boolean, splits: Array[PartitionedFile]): Table
Read the host data to GPU for ORC decoding, and return it as a cuDF Table.
Read the host data to GPU for ORC decoding, and return it as a cuDF Table. The input host buffer should contain valid data, otherwise the behavior is undefined. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
dumpDataToFile(hmb: HostMemoryBuffer, dataLength: Long, splits: Array[PartitionedFile], debugDumpPrefix: Option[String] = None, format: Option[String] = None): Unit
Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format
Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format
- hmb
host data to be dumped
- dataLength
data size
- splits
PartitionedFile to be handled
- debugDumpPrefix
file name prefix, if it is None, will not dump
- format
file name suffix, if it is None, will not dump
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
fileSystemBytesRead(): Long
- Attributes
- protected
- Definition Classes
- MultiFileReaderFunctions
- Annotations
- @nowarn()
-
def
freeOnExcept[T <: RapidsBuffer, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
- Definition Classes
- Arm
-
def
get(): ColumnarBatch
- Definition Classes
- FilePartitionReaderBase → PartitionReader
-
def
getBatchRunner(tc: TaskContext, file: PartitionedFile, conf: Configuration, filters: Array[Filter]): Callable[HostMemoryBuffersWithMetaDataBase]
The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool
The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool
- tc
task context to use
- file
file to be read
- conf
the Configuration parameters
- filters
push down filters
- returns
Callable[HostMemoryBuffersWithMetaDataBase]
- Definition Classes
- MultiFileCloudOrcPartitionReader → MultiFileCloudPartitionReaderBase
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
getFileFormatShortName: String
File format short name used for logging and other things to uniquely identity which file format is being used.
File format short name used for logging and other things to uniquely identity which file format is being used.
- returns
the file format short name
- Definition Classes
- MultiFileCloudOrcPartitionReader → MultiFileCloudPartitionReaderBase
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
var
isDone: Boolean
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
var
maxDeviceMemory: Long
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
val
metrics: Map[String, GpuMetric]
- Definition Classes
- ScanWithMetrics
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
next(): Boolean
- Definition Classes
- MultiFileCloudPartitionReaderBase → PartitionReader
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
populateCurrentBlockChunk(blockIterator: BufferedIterator[OrcOutputStripe], maxReadBatchSizeRows: Int, maxReadBatchSizeBytes: Long): Seq[OrcOutputStripe]
- Definition Classes
- OrcPartitionReaderBase
-
def
readBatch(fileBufsAndMeta: HostMemoryBuffersWithMetaDataBase): Option[ColumnarBatch]
Decode HostMemoryBuffers in GPU
Decode HostMemoryBuffers in GPU
- fileBufsAndMeta
the file HostMemoryBuffer read from a PartitionedFile
- returns
Option[ColumnarBatch] which has been decoded by GPU
- Definition Classes
- MultiFileCloudOrcPartitionReader → MultiFileCloudPartitionReaderBase
-
val
readDataSchema: StructType
- Definition Classes
- MultiFileCloudOrcPartitionReader → OrcCommonFunctions
-
def
readPartFile(ctx: OrcPartitionReaderContext, stripes: Seq[OrcOutputStripe]): (HostMemoryBuffer, Long)
Read the stripes into HostMemoryBuffer.
Read the stripes into HostMemoryBuffer.
- ctx
the context to provide some necessary information
- stripes
a sequence of Stripe to be read into HostMemeoryBuffer
- returns
HostMemeoryBuffer and its data size
- Attributes
- protected
- Definition Classes
- OrcPartitionReaderBase
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
withCodecOutputStream[T](ctx: OrcPartitionReaderContext, out: HostMemoryOutputStream)(block: (WritableByteChannel, CodedOutputStream, OutStream) ⇒ T): T
Executes the provided code block in the codec environment
Executes the provided code block in the codec environment
- Definition Classes
- OrcCodecWritingHelper
-
def
withResource[T <: AutoCloseable, V](h: CloseableHolder[T])(block: (CloseableHolder[T]) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block and then closes the array buffer of resources
Executes the provided code block and then closes the array buffer of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block and then closes the array of resources
Executes the provided code block and then closes the array of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block and then closes the sequence of resources
Executes the provided code block and then closes the sequence of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block and then closes the Option[resource]
Executes the provided code block and then closes the Option[resource]
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResourceIfAllowed[T, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the value if it is AutoCloseable
Executes the provided code block and then closes the value if it is AutoCloseable
- Definition Classes
- Arm
-
def
writeOrcFileFooter(ctx: OrcPartitionReaderContext, fileFooterBuilder: Builder, rawOut: HostMemoryOutputStream, footerStartOffset: Long, numRows: Long, protoWriter: CodedOutputStream, codecStream: OutStream): Unit
write the ORC file Footer and PostScript
write the ORC file Footer and PostScript
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions