abstract class GpuTextBasedPartitionReader extends PartitionReader[ColumnarBatch] with ScanWithMetrics with Arm
The text based PartitionReader
- Alphabetic
- By Inheritance
- GpuTextBasedPartitionReader
- Arm
- ScanWithMetrics
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
GpuTextBasedPartitionReader(conf: Configuration, partFile: PartitionedFile, dataSchema: StructType, readDataSchema: StructType, lineSeparatorInRead: Option[Array[Byte]], maxRowsPerChunk: Integer, maxBytesPerChunk: Long, execMetrics: Map[String, GpuMetric])
- conf
the Hadoop configuration
- partFile
file split to read
- dataSchema
schema of the data
- readDataSchema
the Spark schema describing what will be read
- lineSeparatorInRead
An optional byte line sep.
- maxRowsPerChunk
maximum number of rows to read in a batch
- maxBytesPerChunk
maximum number of bytes to read in a batch
- execMetrics
metrics to update during read
Abstract Value Members
- abstract def castStringToBool(input: ColumnVector): ColumnVector
- abstract def dateFormat: String
-
abstract
def
getFileFormatShortName: String
File format short name used for logging and other things to uniquely identity which file format is being used.
File format short name used for logging and other things to uniquely identity which file format is being used.
- returns
the file format short name
-
abstract
def
readToTable(dataBuffer: HostMemoryBuffer, dataSize: Long, cudfSchema: Schema, readDataSchema: StructType, isFirstChunk: Boolean): Table
Read the host buffer to GPU table
Read the host buffer to GPU table
- dataBuffer
host buffer to be read
- dataSize
the size of host buffer
- cudfSchema
the cudf schema of the data
- readDataSchema
the Spark schema describing what will be read
- isFirstChunk
if it is the first chunk
- returns
table
- abstract def timestampFormat: String
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
- def castStringToDate(input: ColumnVector, dt: DType, failOnInvalid: Boolean): ColumnVector
- def castStringToDecimal(input: ColumnVector, dt: DecimalType): ColumnVector
- def castStringToFloat(input: ColumnVector, dt: DType): ColumnVector
- def castStringToInt(input: ColumnVector, intType: DType): ColumnVector
- def castStringToTimestamp(lhs: ColumnVector, sparkFormat: String, dtype: DType): ColumnVector
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
def
close(): Unit
- Definition Classes
- GpuTextBasedPartitionReader → Closeable → AutoCloseable
-
def
closeOnExcept[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, closing the resource only if an exception occurs
Executes the provided code block, closing the resource only if an exception occurs
- Definition Classes
- Arm
-
def
currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- PartitionReader
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
freeOnExcept[T <: RapidsBuffer, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
- Definition Classes
- Arm
-
def
get(): ColumnarBatch
- Definition Classes
- GpuTextBasedPartitionReader → PartitionReader
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
handleResult(readDataSchema: StructType, table: Table): Option[Table]
Handle the table decoded by GPU
Handle the table decoded by GPU
Please note that, this function owns table which is supposed to be closed in this function But for the optimization, we just return the original table.
- readDataSchema
the Spark schema describing what will be read
- table
the table decoded by GPU
- returns
the new optional Table
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
val
metrics: Map[String, GpuMetric]
- Definition Classes
- ScanWithMetrics
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
next(): Boolean
- Definition Classes
- GpuTextBasedPartitionReader → PartitionReader
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
withResource[T <: AutoCloseable, V](h: CloseableHolder[T])(block: (CloseableHolder[T]) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block and then closes the array buffer of resources
Executes the provided code block and then closes the array buffer of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block and then closes the array of resources
Executes the provided code block and then closes the array of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block and then closes the sequence of resources
Executes the provided code block and then closes the sequence of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block and then closes the Option[resource]
Executes the provided code block and then closes the Option[resource]
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResourceIfAllowed[T, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the value if it is AutoCloseable
Executes the provided code block and then closes the value if it is AutoCloseable
- Definition Classes
- Arm