Packages

c

com.nvidia.spark.rapids

MultiFileCoalescingPartitionReaderBase

abstract class MultiFileCoalescingPartitionReaderBase extends FilePartitionReaderBase with MultiFileReaderFunctions

The abstracted multi-file coalescing reading class, which tries to coalesce small ColumnarBatch into a bigger ColumnarBatch according to maxReadBatchSizeRows, maxReadBatchSizeBytes and the checkIfNeedToSplitDataBlock.

Please be note, this class is applied to below similar file format

| HEADER | -> optional

| block | -> repeated

| FOOTER | -> optional

The data driven:

next() -> populateCurrentBlockChunk (try the best to coalesce ColumnarBatch) -> allocate a bigger HostMemoryBuffer for HEADER + the populated block chunks + FOOTER -> write header to HostMemoryBuffer -> launch tasks to copy the blocks to the HostMemoryBuffer -> wait all tasks finished -> write footer to HostMemoryBuffer -> decode the HostMemoryBuffer in the GPU

Linear Supertypes
MultiFileReaderFunctions, FilePartitionReaderBase, Arm, ScanWithMetrics, Logging, PartitionReader[ColumnarBatch], Closeable, AutoCloseable, AnyRef, Any
Known Subclasses
MultiFileOrcPartitionReader, MultiFileParquetPartitionReader, GpuMultiFileAvroPartitionReader
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MultiFileCoalescingPartitionReaderBase
  2. MultiFileReaderFunctions
  3. FilePartitionReaderBase
  4. Arm
  5. ScanWithMetrics
  6. Logging
  7. PartitionReader
  8. Closeable
  9. AutoCloseable
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MultiFileCoalescingPartitionReaderBase(conf: Configuration, clippedBlocks: Seq[SingleDataBlockInfo], readDataSchema: StructType, partitionSchema: StructType, maxReadBatchSizeRows: Integer, maxReadBatchSizeBytes: Long, numThreads: Int, execMetrics: Map[String, GpuMetric])

    conf

    Configuration

    clippedBlocks

    the block metadata from the original file that has been clipped to only contain the column chunks to be read

    readDataSchema

    the Spark schema describing what will be read

    partitionSchema

    schema of partitions

    maxReadBatchSizeRows

    soft limit on the maximum number of rows the reader reads per batch

    maxReadBatchSizeBytes

    soft limit on the maximum number of bytes the reader reads per batch

    numThreads

    the size of the threadpool

    execMetrics

    metrics

Abstract Value Members

  1. abstract def calculateEstimatedBlocksOutputSize(batchContext: BatchContext): Long

    Calculate the output size according to the block chunks and the schema, and the estimated output size will be used as the initialized size of allocating HostMemoryBuffer

    Calculate the output size according to the block chunks and the schema, and the estimated output size will be used as the initialized size of allocating HostMemoryBuffer

    Please be note, the estimated size should be at least equal to size of HEAD + Blocks + FOOTER

    batchContext

    the batch building context

    returns

    Long, the estimated output size

  2. abstract def calculateFinalBlocksOutputSize(footerOffset: Long, blocks: Seq[DataBlockBase], batchContext: BatchContext): Long

    Calculate the final block output size which will be used to decide if re-allocate HostMemoryBuffer

    Calculate the final block output size which will be used to decide if re-allocate HostMemoryBuffer

    There is no need to re-calculate the block size, just calculate the footer size and plus footerOffset.

    If the size calculated by this function is bigger than the one calculated by calculateEstimatedBlocksOutputSize, then it will cause HostMemoryBuffer re-allocating, and cause the performance issue.

    footerOffset

    footer offset

    blocks

    blocks to be evaluated

    batchContext

    the batch building context

    returns

    the output size

  3. abstract def checkIfNeedToSplitDataBlock(currentBlockInfo: SingleDataBlockInfo, nextBlockInfo: SingleDataBlockInfo): Boolean

    To check if the next block will be split into another ColumnarBatch

    To check if the next block will be split into another ColumnarBatch

    currentBlockInfo

    current SingleDataBlockInfo

    nextBlockInfo

    next SingleDataBlockInfo

    returns

    true: split the next block into another ColumnarBatch and vice versa

  4. abstract def getBatchRunner(tc: TaskContext, file: Path, outhmb: HostMemoryBuffer, blocks: ArrayBuffer[DataBlockBase], offset: Long, batchContext: BatchContext): Callable[(Seq[DataBlockBase], Long)]

    The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool

    The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool

    tc

    task context to use

    file

    file to be read

    outhmb

    the sliced HostMemoryBuffer to hold the blocks, and the implementation is in charge of closing it in sub-class

    blocks

    blocks meta info to specify which blocks to be read

    offset

    used as the offset adjustment

    batchContext

    the batch building context

    returns

    Callable[(Seq[DataBlockBase], Long)], which will be submitted to a ThreadPoolExecutor, and the Callable will return a tuple result and result._1 is block meta info with the offset adjusted result._2 is the bytes read

  5. abstract def getFileFormatShortName: String

    File format short name used for logging and other things to uniquely identity which file format is being used.

    File format short name used for logging and other things to uniquely identity which file format is being used.

    returns

    the file format short name

  6. abstract def readBufferToTable(dataBuffer: HostMemoryBuffer, dataSize: Long, clippedSchema: SchemaBase, extraInfo: ExtraInfo): Table

    Sent host memory to GPU to decode

    Sent host memory to GPU to decode

    dataBuffer

    the data which can be decoded in GPU

    dataSize

    data size

    clippedSchema

    the clipped schema

    extraInfo

    the extra information for specific file format

    returns

    Table

  7. abstract def writeFileFooter(buffer: HostMemoryBuffer, bufferSize: Long, footerOffset: Long, blocks: Seq[DataBlockBase], batchContext: BatchContext): (HostMemoryBuffer, Long)

    Writer a footer for a specific file format.

    Writer a footer for a specific file format. If there is no footer for the file format, just return (hmb, offset)

    Please be note, some file formats may re-allocate the HostMemoryBuffer because of the estimated initialized buffer size may be a little smaller than the actual size. So in this case, the hmb should be closed in the implementation.

    buffer

    The buffer holding (header + data blocks)

    bufferSize

    The total buffer size which equals to size of (header + blocks + footer)

    footerOffset

    Where begin to write the footer

    blocks

    The data block meta info

    batchContext

    The batch building context

    returns

    the buffer and the buffer size

  8. abstract def writeFileHeader(buffer: HostMemoryBuffer, batchContext: BatchContext): Long

    Write a header for a specific file format.

    Write a header for a specific file format. If there is no header for the file format, just ignore it and return 0

    buffer

    where the header will be written

    batchContext

    the batch building context

    returns

    how many bytes written

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addPartitionValues(batch: Option[ColumnarBatch], inPartitionValues: InternalRow, partitionSchema: StructType): Option[ColumnarBatch]
    Attributes
    protected
    Definition Classes
    MultiFileReaderFunctions
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. var batch: Option[ColumnarBatch]
    Attributes
    protected
    Definition Classes
    FilePartitionReaderBase
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  8. def close(): Unit
    Definition Classes
    FilePartitionReaderBase → Closeable → AutoCloseable
  9. def closeOnExcept[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V

    Executes the provided code block, closing the resources only if an exception occurs

    Executes the provided code block, closing the resources only if an exception occurs

    Definition Classes
    Arm
  10. def closeOnExcept[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V

    Executes the provided code block, closing the resources only if an exception occurs

    Executes the provided code block, closing the resources only if an exception occurs

    Definition Classes
    Arm
  11. def closeOnExcept[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V

    Executes the provided code block, closing the resources only if an exception occurs

    Executes the provided code block, closing the resources only if an exception occurs

    Definition Classes
    Arm
  12. def closeOnExcept[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V

    Executes the provided code block, closing the resources only if an exception occurs

    Executes the provided code block, closing the resources only if an exception occurs

    Definition Classes
    Arm
  13. def closeOnExcept[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V

    Executes the provided code block, closing the resource only if an exception occurs

    Executes the provided code block, closing the resource only if an exception occurs

    Definition Classes
    Arm
  14. def createBatchContext(chunkedBlocks: LinkedHashMap[Path, ArrayBuffer[DataBlockBase]], clippedSchema: SchemaBase): BatchContext

    Return a batch context which will be shared during the process of building a memory file, aka with the following APIs.

    Return a batch context which will be shared during the process of building a memory file, aka with the following APIs.

    • calculateEstimatedBlocksOutputSize
    • writeFileHeader
    • getBatchRunner
    • calculateFinalBlocksOutputSize
    • writeFileFooter It is useful when something is needed by some or all of the above APIs. Children can override this to return a customized batch context.
    chunkedBlocks

    mapping of file path to data blocks

    clippedSchema

    schema info

    Attributes
    protected
  15. def currentMetricsValues(): Array[CustomTaskMetric]
    Definition Classes
    PartitionReader
  16. def dumpDataToFile(hmb: HostMemoryBuffer, dataLength: Long, splits: Array[PartitionedFile], debugDumpPrefix: Option[String] = None, format: Option[String] = None): Unit

    Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format

    Dump the data from HostMemoryBuffer to a file named by debugDumpPrefix + random + format

    hmb

    host data to be dumped

    dataLength

    data size

    splits

    PartitionedFile to be handled

    debugDumpPrefix

    file name prefix, if it is None, will not dump

    format

    file name suffix, if it is None, will not dump

    Attributes
    protected
    Definition Classes
    FilePartitionReaderBase
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  19. def fileSystemBytesRead(): Long
    Attributes
    protected
    Definition Classes
    MultiFileReaderFunctions
    Annotations
    @nowarn()
  20. def freeOnExcept[T <: RapidsBuffer, V](r: T)(block: (T) ⇒ V): V

    Executes the provided code block, freeing the RapidsBuffer only if an exception occurs

    Executes the provided code block, freeing the RapidsBuffer only if an exception occurs

    Definition Classes
    Arm
  21. def get(): ColumnarBatch
    Definition Classes
    FilePartitionReaderBase → PartitionReader
  22. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  24. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  25. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. var isDone: Boolean
    Attributes
    protected
    Definition Classes
    FilePartitionReaderBase
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  37. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. var maxDeviceMemory: Long
    Attributes
    protected
    Definition Classes
    FilePartitionReaderBase
  42. val metrics: Map[String, GpuMetric]
    Definition Classes
    ScanWithMetrics
  43. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  44. def next(): Boolean
    Definition Classes
    MultiFileCoalescingPartitionReaderBase → PartitionReader
  45. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  46. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  47. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  48. def toString(): String
    Definition Classes
    AnyRef → Any
  49. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  51. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  52. def withResource[T <: AutoCloseable, V](h: CloseableHolder[T])(block: (CloseableHolder[T]) ⇒ V): V

    Executes the provided code block and then closes the resource

    Executes the provided code block and then closes the resource

    Definition Classes
    Arm
  53. def withResource[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V

    Executes the provided code block and then closes the array buffer of resources

    Executes the provided code block and then closes the array buffer of resources

    Definition Classes
    Arm
  54. def withResource[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V

    Executes the provided code block and then closes the array of resources

    Executes the provided code block and then closes the array of resources

    Definition Classes
    Arm
  55. def withResource[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V

    Executes the provided code block and then closes the sequence of resources

    Executes the provided code block and then closes the sequence of resources

    Definition Classes
    Arm
  56. def withResource[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V

    Executes the provided code block and then closes the Option[resource]

    Executes the provided code block and then closes the Option[resource]

    Definition Classes
    Arm
  57. def withResource[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V

    Executes the provided code block and then closes the resource

    Executes the provided code block and then closes the resource

    Definition Classes
    Arm
  58. def withResourceIfAllowed[T, V](r: T)(block: (T) ⇒ V): V

    Executes the provided code block and then closes the value if it is AutoCloseable

    Executes the provided code block and then closes the value if it is AutoCloseable

    Definition Classes
    Arm

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from MultiFileReaderFunctions

Inherited from FilePartitionReaderBase

Inherited from Arm

Inherited from ScanWithMetrics

Inherited from Logging

Inherited from PartitionReader[ColumnarBatch]

Inherited from Closeable

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped