Packages

package shims

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class GpuBatchScanExec(output: Seq[AttributeReference], scan: Scan, runtimeFilters: Seq[Expression] = Seq.empty) extends SparkPlan with DataSourceV2ScanExecBase with GpuBatchScanExecMetrics with Product with Serializable
  2. class GpuCustomShuffleReaderMeta extends SparkPlanMeta[AQEShuffleReadExec]
  3. class GpuDataSourceRDD extends DataSourceRDD

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric.

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric. DataSourceRDD assumes all reads occur on the task thread, and some GPU input sources use multithreaded readers that cannot generate proper metrics with DataSourceRDD.

    Note

    It is the responsibility of users of this RDD to generate the bytes read input metric explicitly!

  4. trait GpuDeterministicFirstLastCollectShim extends Expression
  5. case class GpuHashPartitioning(expressions: Seq[Expression], numPartitions: Int) extends GpuHashPartitioningBase with Product with Serializable
  6. case class GpuRangePartitioning(gpuOrdering: Seq[SortOrder], numPartitions: Int) extends Expression with GpuExpression with ShimExpression with GpuPartitioning with Product with Serializable

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges.

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges. The ranges are determined by sampling the content of the RDD passed in.

    Note

    The actual number of partitions created might not be the same as the numPartitions parameter, in the case where the number of sampled records is less than the value of partitions. The GpuRangePartitioner is where all of the processing actually happens.

  7. class GpuSpecifiedWindowFrameMeta extends GpuSpecifiedWindowFrameMetaBase
  8. class GpuWindowExpressionMeta extends GpuWindowExpressionMetaBase
  9. case class GpuWindowInPandasExec(windowExpression: Seq[Expression], gpuPartitionSpec: Seq[Expression], cpuOrderSpec: Seq[SortOrder], child: SparkPlan)(cpuPartitionSpec: Seq[Expression]) extends SparkPlan with GpuWindowInPandasExecBase with Product with Serializable
  10. abstract class OffsetWindowFunctionMeta[INPUT <: OffsetWindowFunction] extends ExprMeta[INPUT]

    Spark 3.2-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta.

    Spark 3.2-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta. This is required primarily for two reasons:

    1. com.nvidia.spark.rapids.OffsetWindowFunctionMeta (compiled against Spark 3.0.x) fails class load in Spark 3.1.x. (expr.input is not recognized as an Expression.) 2. The semantics of offsets in LAG() are reversed/negated in Spark 3.1.1. E.g. The expression LAG(col, 5) causes Lag.offset to be set to -5, as opposed to 5, in prior versions of Spark. 3. IGNORE NULL and RESPECT NULL was added in 3.2.0 and we need to fall back until this can be supported. This class adjusts the LAG offset to use similar semantics to Spark 3.0.x.
  11. trait OrcShims320untilAllBase extends AnyRef
  12. class RapidsCsvScanMeta extends ScanMeta[CSVScan]
  13. class RapidsOrcScanMeta extends ScanMeta[OrcScan]
  14. class RapidsParquetScanMeta extends ScanMeta[ParquetScan]
  15. trait RebaseShims extends AnyRef
  16. trait ShimBinaryExecNode extends SparkPlan with BinaryExecNode
  17. trait ShimBinaryExpression extends BinaryExpression
  18. trait ShimBroadcastExchangeLike extends Exchange with BroadcastExchangeLike

    This shim handles the completion future differences between Apache Spark and Databricks.

  19. trait ShimExpression extends Expression
  20. abstract class ShimFilePartitionReaderFactory extends FilePartitionReaderFactory
  21. trait ShimSparkPlan extends SparkPlan
  22. trait ShimSupportsRuntimeFiltering extends SupportsRuntimeFiltering

    Shim interface for Apache Spark's SupportsRuntimeFiltering interface which was added in Spark 3.2.0.

  23. trait ShimTernaryExpression extends TernaryExpression
  24. trait ShimUnaryCommand extends LogicalPlan with UnaryCommand
  25. trait ShimUnaryExecNode extends SparkPlan with UnaryExecNode
  26. trait ShimUnaryExpression extends UnaryExpression
  27. trait Spark31Xuntil33XShims extends SparkShims
  28. trait Spark320PlusNonDBShims extends SparkShims

    Shim methods that can be compiled with every supported 3.2.0+ except Databricks versions

  29. trait Spark320PlusShims extends SparkShims with RebaseShims with Logging

    Shim base class that can be compiled with every supported 3.2.0+

  30. trait Spark320until340Shims extends SparkShims

Value Members

  1. object AQEUtils

    Utility methods for manipulating Catalyst classes involved in Adaptive Query Execution

  2. object AggregationTagging
  3. object AnsiUtil
  4. object GlobalLimitShims
  5. object GpuDataSourceRDD extends Serializable
  6. object GpuHashPartitioning extends Serializable
  7. object GpuIntervalUtils

    Should not support in this Shim

  8. object GpuParquetCrypto
  9. object GpuTypeShims
  10. object GpuWindowUtil
  11. object HashUtils extends Arm
  12. object OrcReadingShims
  13. object OrcShims extends OrcShims320untilAllBase
  14. object ParquetFieldIdShims
  15. object ParquetSchemaClipShims
  16. object ParquetStringPredShims
  17. object ParquetTimestampNTZShims
  18. object RapidsFileSourceMetaUtils
  19. object RegExpShim
  20. object SparkShimImpl extends Spark320PlusShims with Spark320PlusNonDBShims with Spark31Xuntil33XShims with Spark320until340Shims
  21. object TypeSigUtil extends TypeSigUtilBase

    Add DayTimeIntervalType and YearMonthIntervalType support

  22. object YearParseUtil

Ungrouped