t

com.nvidia.spark.rapids.shims

Spark320PlusNonDBShims

trait Spark320PlusNonDBShims extends SparkShims

Shim methods that can be compiled with every supported 3.2.0+ except Databricks versions

Linear Supertypes
SparkShims, AnyRef, Any
Known Subclasses
SparkShimImpl
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Spark320PlusNonDBShims
  2. SparkShims
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def ansiCastRule: ExprRule[_ <: Expression]

    Return the replacement rule for AnsiCast.

    Return the replacement rule for AnsiCast. 'AnsiCast' is removed from Spark 3.4.0, so need to handle it separately.

    Definition Classes
    SparkShims
  2. abstract def aqeShuffleReaderExec: ExecRule[_ <: SparkPlan]
    Definition Classes
    SparkShims
  3. abstract def attachTreeIfSupported[TreeType <: TreeNode[_], A](tree: TreeType, msg: String = "")(f: ⇒ A): A

    dropped by SPARK-34234

    dropped by SPARK-34234

    Definition Classes
    SparkShims
  4. abstract def avroRebaseReadKey: String
    Definition Classes
    SparkShims
  5. abstract def avroRebaseWriteKey: String
    Definition Classes
    SparkShims
  6. abstract def columnarAdaptivePlan(a: AdaptiveSparkPlanExec, goal: CoalesceSizeGoal): SparkPlan
    Definition Classes
    SparkShims
  7. abstract def findOperators(plan: SparkPlan, predicate: (SparkPlan) ⇒ Boolean): Seq[SparkPlan]

    Walk the plan recursively and return a list of operators that match the predicate

    Walk the plan recursively and return a list of operators that match the predicate

    Definition Classes
    SparkShims
  8. abstract def getAdaptiveInputPlan(adaptivePlan: AdaptiveSparkPlanExec): SparkPlan
    Definition Classes
    SparkShims
  9. abstract def getDateFormatter(): DateFormatter
    Definition Classes
    SparkShims
  10. abstract def getExecs: Map[Class[_ <: SparkPlan], ExecRule[_ <: SparkPlan]]
    Definition Classes
    SparkShims
  11. abstract def getExprs: Map[Class[_ <: Expression], ExprRule[_ <: Expression]]
    Definition Classes
    SparkShims
  12. abstract def getFileScanRDD(sparkSession: SparkSession, readFunction: (PartitionedFile) ⇒ Iterator[InternalRow], filePartitions: Seq[FilePartition], readDataSchema: StructType, metadataColumns: Seq[AttributeReference] = Seq.empty): RDD[InternalRow]
    Definition Classes
    SparkShims
  13. abstract def getParquetFilters(schema: MessageType, pushDownDate: Boolean, pushDownTimestamp: Boolean, pushDownDecimal: Boolean, pushDownStartWith: Boolean, pushDownInFilterThreshold: Int, caseSensitive: Boolean, lookupFileMeta: (String) ⇒ String, dateTimeRebaseModeFromConf: String): ParquetFilters
    Definition Classes
    SparkShims
  14. abstract def getScans: Map[Class[_ <: Scan], ScanRule[_ <: Scan]]
    Definition Classes
    SparkShims
  15. abstract def getSparkShimVersion: ShimVersion
    Definition Classes
    SparkShims
  16. abstract def hasAliasQuoteFix: Boolean
    Definition Classes
    SparkShims
  17. abstract def hasCastFloatTimestampUpcast: Boolean
    Definition Classes
    SparkShims
  18. abstract def int96ParquetRebaseRead(conf: SQLConf): String
    Definition Classes
    SparkShims
  19. abstract def int96ParquetRebaseReadKey: String
    Definition Classes
    SparkShims
  20. abstract def int96ParquetRebaseWrite(conf: SQLConf): String
    Definition Classes
    SparkShims
  21. abstract def int96ParquetRebaseWriteKey: String
    Definition Classes
    SparkShims
  22. abstract def isAqePlan(p: SparkPlan): Boolean
    Definition Classes
    SparkShims
  23. abstract def isCustomReaderExec(x: SparkPlan): Boolean
    Definition Classes
    SparkShims
  24. abstract def isEmptyRelation(relation: Any): Boolean
    Definition Classes
    SparkShims
  25. abstract def isExchangeOp(plan: SparkPlanMeta[_]): Boolean
    Definition Classes
    SparkShims
  26. abstract def isWindowFunctionExec(plan: SparkPlan): Boolean
    Definition Classes
    SparkShims
  27. abstract def leafNodeDefaultParallelism(ss: SparkSession): Int
    Definition Classes
    SparkShims
  28. abstract def neverReplaceShowCurrentNamespaceCommand: ExecRule[_ <: SparkPlan]
    Definition Classes
    SparkShims
  29. abstract def parquetRebaseRead(conf: SQLConf): String
    Definition Classes
    SparkShims
  30. abstract def parquetRebaseReadKey: String
    Definition Classes
    SparkShims
  31. abstract def parquetRebaseWrite(conf: SQLConf): String
    Definition Classes
    SparkShims
  32. abstract def parquetRebaseWriteKey: String
    Definition Classes
    SparkShims
  33. abstract def sessionFromPlan(plan: SparkPlan): SparkSession
    Definition Classes
    SparkShims
  34. abstract def shouldFailDivOverflow: Boolean
    Definition Classes
    SparkShims
  35. abstract def skipAssertIsOnTheGpu(plan: SparkPlan): Boolean

    Our tests, by default, will check that all operators are running on the GPU, but there are some operators that we do not translate to GPU plans, so we need a way to bypass the check for those.

    Our tests, by default, will check that all operators are running on the GPU, but there are some operators that we do not translate to GPU plans, so we need a way to bypass the check for those.

    Definition Classes
    SparkShims
  36. abstract def supportsColumnarAdaptivePlans: Boolean

    Determine if the Spark version allows the supportsColumnar flag to be overridden in AdaptiveSparkPlanExec.

    Determine if the Spark version allows the supportsColumnar flag to be overridden in AdaptiveSparkPlanExec. This feature was introduced in Spark 3.2 as part of SPARK-35881.

    Definition Classes
    SparkShims
  37. abstract def tryTransformIfEmptyRelation(mode: BroadcastMode): Option[Any]

    This call can produce an EmptyHashedRelation or an empty array, allowing the AQE rule EliminateJoinToEmptyRelation in Spark 3.1.x to optimize certain joins.

    This call can produce an EmptyHashedRelation or an empty array, allowing the AQE rule EliminateJoinToEmptyRelation in Spark 3.1.x to optimize certain joins.

    In Spark 3.2.0, the optimization is still performed (under AQEPropagateEmptyRelation), but the AQE optimizer is looking at the metrics for the query stage to determine if numRows == 0, and if so it can eliminate certain joins.

    The call is implemented only for Spark 3.1.x+. It is disabled in Databricks because it requires a task context to perform the BroadcastMode.transform call, but we'd like to call this from the driver.

    Definition Classes
    SparkShims
  38. abstract def v1RepairTableCommand(tableName: TableIdentifier): RunnableCommand
    Definition Classes
    SparkShims

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. final def broadcastModeTransform(mode: BroadcastMode, rows: Array[InternalRow]): Any
    Definition Classes
    Spark320PlusNonDBShims → SparkShims
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. final def filesFromFileIndex(fileIndex: PartitioningAwareFileIndex): Seq[FileStatus]
    Definition Classes
    Spark320PlusNonDBShims → SparkShims
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  11. def getWindowExpressions(winPy: WindowInPandasExec): Seq[NamedExpression]
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. def isCastingStringToNegDecimalScaleSupported: Boolean
    Definition Classes
    SparkShims
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def newBroadcastQueryStageExec(old: BroadcastQueryStageExec, newPlan: SparkPlan): BroadcastQueryStageExec
    Definition Classes
    Spark320PlusNonDBShims → SparkShims
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  19. def reusedExchangeExecPfn: PartialFunction[SparkPlan, ReusedExchangeExec]

    Case class ShuffleQueryStageExec holds an additional field shuffleOrigin affecting the unapply method signature

    Case class ShuffleQueryStageExec holds an additional field shuffleOrigin affecting the unapply method signature

    Definition Classes
    Spark320PlusNonDBShims → SparkShims
  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from SparkShims

Inherited from AnyRef

Inherited from Any

Ungrouped