class ParquetCachedBatchSerializer extends GpuCachedBatchSerializer with Arm
This class assumes, the data is Columnar and the plugin is on. Note, this class should not be referenced directly in source code. It should be loaded by reflection using ShimLoader.newInstanceOf, see ./docs/dev/shims.md
- Attributes
- protected
- Alphabetic
- By Inheritance
- ParquetCachedBatchSerializer
- Arm
- GpuCachedBatchSerializer
- CachedBatchSerializer
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new ParquetCachedBatchSerializer()
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
buildFilter(predicates: Seq[Expression], cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) ⇒ Iterator[CachedBatch]
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
def
closeOnExcept[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block, closing the resources only if an exception occurs
Executes the provided code block, closing the resources only if an exception occurs
- Definition Classes
- Arm
-
def
closeOnExcept[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, closing the resource only if an exception occurs
Executes the provided code block, closing the resource only if an exception occurs
- Definition Classes
- Arm
-
def
convertCachedBatchToColumnarBatch(input: RDD[CachedBatch], cacheAttributes: Seq[Attribute], selectedAttributes: Seq[Attribute], conf: SQLConf): RDD[ColumnarBatch]
Convert the cached data into a ColumnarBatch taking the result data back to the host
Convert the cached data into a ColumnarBatch taking the result data back to the host
- input
the cached batches that should be converted.
- cacheAttributes
the attributes of the data in the batch.
- selectedAttributes
the fields that should be loaded from the data and the order they should appear in the output batch.
- conf
the configuration for the job.
- returns
an RDD of the input cached batches transformed into the ColumnarBatch format.
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
def
convertCachedBatchToInternalRow(input: RDD[CachedBatch], cacheAttributes: Seq[Attribute], selectedAttributes: Seq[Attribute], conf: SQLConf): RDD[InternalRow]
Convert the cached batch into
InternalRows.Convert the cached batch into
InternalRows.- input
the cached batches that should be converted.
- cacheAttributes
the attributes of the data in the batch.
- selectedAttributes
the field that should be loaded from the data and the order they should appear in the output rows.
- conf
the configuration for the job.
- returns
RDD of the rows that were stored in the cached batches.
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
def
convertColumnarBatchToCachedBatch(input: RDD[ColumnarBatch], schema: Seq[Attribute], storageLevel: StorageLevel, conf: SQLConf): RDD[CachedBatch]
Convert an
RDD[ColumnarBatch]into anRDD[CachedBatch]in preparation for caching the data.Convert an
RDD[ColumnarBatch]into anRDD[CachedBatch]in preparation for caching the data. This method uses Parquet Writer on the GPU to write the cached batch- input
the input
RDDto be converted.- schema
the schema of the data being stored.
- storageLevel
where the data will be stored.
- conf
the config for the query.
- returns
The data converted into a format more suitable for caching.
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
def
convertInternalRowToCachedBatch(input: RDD[InternalRow], schema: Seq[Attribute], storageLevel: StorageLevel, conf: SQLConf): RDD[CachedBatch]
Convert an
RDD[InternalRow]into anRDD[CachedBatch]in preparation for caching the data.Convert an
RDD[InternalRow]into anRDD[CachedBatch]in preparation for caching the data. We use the RowToColumnarIterator and convert each batch at a time- input
the input
RDDto be converted.- schema
the schema of the data being stored.
- storageLevel
where the data will be stored.
- conf
the config for the query.
- returns
The data converted into a format more suitable for caching.
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
freeOnExcept[T <: RapidsBuffer, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
Executes the provided code block, freeing the RapidsBuffer only if an exception occurs
- Definition Classes
- Arm
- def getBytesAllowedPerBatch(conf: SQLConf): Long
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- def getParquetWriterOptions(useCompression: Boolean, schema: StructType): ParquetWriterOptions
-
def
gpuConvertCachedBatchToColumnarBatch(input: RDD[CachedBatch], cacheAttributes: Seq[Attribute], selectedAttributes: Seq[Attribute], conf: SQLConf): RDD[ColumnarBatch]
This method decodes the CachedBatch leaving it on the GPU to avoid the extra copying back to the host
This method decodes the CachedBatch leaving it on the GPU to avoid the extra copying back to the host
- input
the cached batches that should be converted.
- cacheAttributes
the attributes of the data in the batch.
- selectedAttributes
the fields that should be loaded from the data and the order they should appear in the output batch.
- conf
the configuration for the job.
- returns
an RDD of the input cached batches transformed into the ColumnarBatch format.
- Definition Classes
- ParquetCachedBatchSerializer → GpuCachedBatchSerializer
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isSchemaSupportedByCudf(schema: Seq[Attribute]): Boolean
- def isSupportedByCudf(dataType: DataType): Boolean
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
supportsColumnarInput(schema: Seq[Attribute]): Boolean
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
def
supportsColumnarOutput(schema: StructType): Boolean
- Definition Classes
- ParquetCachedBatchSerializer → CachedBatchSerializer
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
vectorTypes(attributes: Seq[Attribute], conf: SQLConf): Option[Seq[String]]
- Definition Classes
- CachedBatchSerializer
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
withResource[T <: AutoCloseable, V](h: CloseableHolder[T])(block: (CloseableHolder[T]) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: ArrayBuffer[T])(block: (ArrayBuffer[T]) ⇒ V): V
Executes the provided code block and then closes the array buffer of resources
Executes the provided code block and then closes the array buffer of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Array[T])(block: (Array[T]) ⇒ V): V
Executes the provided code block and then closes the array of resources
Executes the provided code block and then closes the array of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Seq[T])(block: (Seq[T]) ⇒ V): V
Executes the provided code block and then closes the sequence of resources
Executes the provided code block and then closes the sequence of resources
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: Option[T])(block: (Option[T]) ⇒ V): V
Executes the provided code block and then closes the Option[resource]
Executes the provided code block and then closes the Option[resource]
- Definition Classes
- Arm
-
def
withResource[T <: AutoCloseable, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the resource
Executes the provided code block and then closes the resource
- Definition Classes
- Arm
-
def
withResourceIfAllowed[T, V](r: T)(block: (T) ⇒ V): V
Executes the provided code block and then closes the value if it is AutoCloseable
Executes the provided code block and then closes the value if it is AutoCloseable
- Definition Classes
- Arm