class GpuOrcPartitionReader extends FilePartitionReaderBase with OrcPartitionReaderBase
A PartitionReader that reads an ORC file split on the GPU.
Efficiently reading an ORC split on the GPU requires rebuilding the ORC file in memory such that only relevant data is present in the memory file. This avoids sending unnecessary data to the GPU and saves GPU memory.
- Alphabetic
- By Inheritance
- GpuOrcPartitionReader
- OrcPartitionReaderBase
- OrcCommonFunctions
- OrcCodecWritingHelper
- FilePartitionReaderBase
- Arm
- ScanWithMetrics
- Logging
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
GpuOrcPartitionReader(conf: Configuration, partFile: PartitionedFile, ctx: OrcPartitionReaderContext, readDataSchema: StructType, debugDumpPrefix: Option[String], maxReadBatchSizeRows: Integer, maxReadBatchSizeBytes: Long, execMetrics: Map[String, GpuMetric], isCaseSensitive: Boolean)
- conf
Hadoop configuration
- partFile
file split to read
- ctx
the context to provide some necessary information
- readDataSchema
Spark schema of what will be read from the file
- debugDumpPrefix
path prefix for dumping the memory file or null
- maxReadBatchSizeRows
maximum number of rows to read in a batch
- maxReadBatchSizeBytes
maximum number of bytes to read in a batch
- execMetrics
metrics to update during read
- isCaseSensitive
whether the name check should be case sensitive or not
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
var
batch: Option[ColumnarBatch]
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
def
buildReaderSchema(updatedSchema: TypeDescription, requestedMapping: Option[Array[Int]]): TypeDescription
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
buildReaderSchema(ctx: OrcPartitionReaderContext): TypeDescription
Get the ORC schema corresponding to the file being constructed for the GPU
Get the ORC schema corresponding to the file being constructed for the GPU
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
def
close(): Unit
- Definition Classes
- GpuOrcPartitionReader → FilePartitionReaderBase → Closeable → AutoCloseable
-
def
closeOnExcept[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, closing the resource only if an exception occurs
Executes the provided code block, closing the resource only if an exception occurs
- Definition Classes
- Arm
-
def
copyStripeData(ctx: OrcPartitionReaderContext, out: WritableByteChannel, inputDataRanges: DiskRangeList): Unit
Copy the stripe to the channel
Copy the stripe to the channel
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- PartitionReader
-
val
debugDumpPrefix: Option[String]
- Definition Classes
- GpuOrcPartitionReader → OrcCommonFunctions
-
final
def
decodeToBatch(hostBuf: HostMemoryBuffer, bufSize: Long, memFileSchema: TypeDescription, requestedMapping: Option[Array[Int]], isCaseSensitive: Boolean, splits: Array[PartitionedFile]): Option[ColumnarBatch]
Send a host buffer to GPU for ORC decoding, and return it as a ColumnarBatch.
Send a host buffer to GPU for ORC decoding, and return it as a ColumnarBatch. The input hostBuf will be closed after returning, please do not use it anymore. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- OrcPartitionReaderBase
-
def
decodeToTable(hostBuf: HostMemoryBuffer, bufSize: Long, memFileSchema: TypeDescription, requestedMapping: Option[Array[Int]], isCaseSensitive: Boolean, splits: Array[PartitionedFile]): Table
Read the host data to GPU for ORC decoding, and return it as a cuDF Table.
Read the host data to GPU for ORC decoding, and return it as a cuDF Table. The input host buffer should contain valid data, otherwise the behavior is undefined. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions
-
def
dumpDataToFile(hmb: HostMemoryBuffer, dataLength: Long, splits: Array[PartitionedFile], debugDumpPrefix: Option[String] = None, format: Option[String] = None): Unit
Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format
Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format
- hmb
host data to be dumped
- dataLength
data size
- splits
PartitionedFile to be handled
- debugDumpPrefix
file name prefix, if it is None, will not dump
- format
file name suffix, if it is None, will not dump
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
freeOnExcept[T <: RapidsBuffer, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
- Definition Classes
- Arm
-
def
get(): ColumnarBatch
- Definition Classes
- FilePartitionReaderBase → PartitionReader
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
var
isDone: Boolean
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
var
maxDeviceMemory: Long
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
val
metrics: Map[String, GpuMetric]
- Definition Classes
- ScanWithMetrics
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
next(): Boolean
- Definition Classes
- GpuOrcPartitionReader → PartitionReader
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
populateCurrentBlockChunk(blockIterator: BufferedIterator[OrcOutputStripe], maxReadBatchSizeRows: Int, maxReadBatchSizeBytes: Long): Seq[OrcOutputStripe]
- Definition Classes
- OrcPartitionReaderBase
-
val
readDataSchema: StructType
- Definition Classes
- GpuOrcPartitionReader → OrcCommonFunctions
-
def
readPartFile(ctx: OrcPartitionReaderContext, stripes: Seq[OrcOutputStripe]): (HostMemoryBuffer, Long)
Read the stripes into HostMemoryBuffer.
Read the stripes into HostMemoryBuffer.
- ctx
the context to provide some necessary information
- stripes
a sequence of Stripe to be read into HostMemeoryBuffer
- returns
HostMemeoryBuffer and its data size
- Attributes
- protected
- Definition Classes
- OrcPartitionReaderBase
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
withCodecOutputStream[T](ctx: OrcPartitionReaderContext, out: HostMemoryOutputStream)(block: (WritableByteChannel, CodedOutputStream, OutStream) ⇒ T): T
Executes the provided code block in the codec environment
Executes the provided code block in the codec environment
- Definition Classes
- OrcCodecWritingHelper
-
def
withResource[T <: AutoCloseable, V](h: CloseableHolder[T])(block: (CloseableHolder[T]) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block and then closes the array buffer of resources
Executes the provided code block and then closes the array buffer of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block and then closes the array of resources
Executes the provided code block and then closes the array of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block and then closes the sequence of resources
Executes the provided code block and then closes the sequence of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block and then closes the Option[resource]
Executes the provided code block and then closes the Option[resource]
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResourceIfAllowed[T, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the value if it is AutoCloseable
Executes the provided code block and then closes the value if it is AutoCloseable
- Definition Classes
- Arm
-
def
writeOrcFileFooter(ctx: OrcPartitionReaderContext, fileFooterBuilder: Builder, rawOut: HostMemoryOutputStream, footerStartOffset: Long, numRows: Long, protoWriter: CodedOutputStream, codecStream: OutStream): Unit
write the ORC file Footer and PostScript
write the ORC file Footer and PostScript
- Attributes
- protected
- Definition Classes
- OrcCommonFunctions