Packages

c

org.apache.spark.sql.execution

CollapseCodegenStages

case class CollapseCodegenStages(codegenStageCounter: AtomicInteger = new AtomicInteger(0)) extends Rule[SparkPlan] with Product with Serializable

Find the chained plans that support codegen, collapse them together as WholeStageCodegen.

The codegenStageCounter generates ID for codegen stages within a query plan. It does not affect equality, nor does it participate in destructuring pattern matching of WholeStageCodegenExec.

This ID is used to help differentiate between codegen stages. It is included as a part of the explain output for physical plans, e.g.

Physical Plan

*(5) SortMergeJoin [x#3L], [y#9L], Inner :- *(2) Sort [x#3L ASC NULLS FIRST], false, 0 : +- Exchange hashpartitioning(x#3L, 200) : +- *(1) Project [(id#0L % 2) AS x#3L] : +- *(1) Filter isnotnull((id#0L % 2)) : +- *(1) Range (0, 5, step=1, splits=8) +- *(4) Sort [y#9L ASC NULLS FIRST], false, 0 +- Exchange hashpartitioning(y#9L, 200) +- *(3) Project [(id#6L % 2) AS y#9L] +- *(3) Filter isnotnull((id#6L % 2)) +- *(3) Range (0, 5, step=1, splits=8)

where the ID makes it obvious that not all adjacent codegen'd plan operators are of the same codegen stage.

The codegen stage ID is also optionally included in the name of the generated classes as a suffix, so that it's easier to associate a generated class back to the physical operator. This is controlled by SQLConf: spark.sql.codegen.useIdInClassName

The ID is also included in various log messages.

Within a query, a codegen stage in a plan starts counting from 1, in "insertion order". WholeStageCodegenExec operators are inserted into a plan in depth-first post-order. See CollapseCodegenStages.insertWholeStageCodegen for the definition of insertion order.

0 is reserved as a special ID value to indicate a temporary WholeStageCodegenExec object is created, e.g. for special fallback handling when an existing WholeStageCodegenExec failed to generate/compile code.

Linear Supertypes
Serializable, Serializable, Product, Equals, Rule[SparkPlan], Logging, SQLConfHelper, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CollapseCodegenStages
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Rule
  7. Logging
  8. SQLConfHelper
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CollapseCodegenStages(codegenStageCounter: AtomicInteger = new AtomicInteger(0))

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def apply(plan: SparkPlan): SparkPlan
    Definition Classes
    CollapseCodegenStages → Rule
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. val codegenStageCounter: AtomicInteger
  8. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  11. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  12. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  15. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  16. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  23. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  28. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  29. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  30. lazy val ruleId: RuleId
    Attributes
    protected
    Definition Classes
    Rule
  31. val ruleName: String
    Definition Classes
    Rule
  32. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  33. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  35. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Rule[SparkPlan]

Inherited from Logging

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped