Packages

o

org.apache.spark.sql.catalyst.optimizer

NestedColumnAliasing

object NestedColumnAliasing

This aims to handle a nested column aliasing pattern inside the ColumnPruning optimizer rule. If: - A Project or its child references nested fields - Not all of the fields in a nested attribute are used Then: - Substitute the nested field references with alias attributes - Add grandchild Projects transforming the nested fields to aliases

Example 1: Project ------------------ Before: +- Project [concat_ws(s#0.a, s#0.b) AS concat_ws(s.a, s.b)#1] +- GlobalLimit 5 +- LocalLimit 5 +- LocalRelation <empty>, [s#0] After: +- Project [concat_ws(_extract_a#2, _extract_b#3) AS concat_ws(s.a, s.b)#1] +- GlobalLimit 5 +- LocalLimit 5 +- Project [s#0.a AS _extract_a#2, s#0.b AS _extract_b#3] +- LocalRelation <empty>, [s#0]

Example 2: Project above Filter ------------------------------- Before: +- Project [s#0.a AS s.a#1] +- Filter (length(s#0.b) > 2) +- GlobalLimit 5 +- LocalLimit 5 +- LocalRelation <empty>, [s#0] After: +- Project [_extract_a#2 AS s.a#1] +- Filter (length(_extract_b#3) > 2) +- GlobalLimit 5 +- LocalLimit 5 +- Project [s#0.a AS _extract_a#2, s#0.b AS _extract_b#3] +- LocalRelation <empty>, [s#0]

Example 3: Nested fields with referenced parents ------------------------------------------------ Before: +- Project [s#0.a AS s.a#1, s#0.a.a1 AS s.a.a1#2] +- GlobalLimit 5 +- LocalLimit 5 +- LocalRelation <empty>, [s#0] After: +- Project [_extract_a#3 AS s.a#1, _extract_a#3.name AS s.a.a1#2] +- GlobalLimit 5 +- LocalLimit 5 +- Project [s#0.a AS _extract_a#3] +- LocalRelation <empty>, [s#0]

The schema of the datasource relation will be pruned in the SchemaPruning optimizer rule.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. NestedColumnAliasing
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def getAttributeToExtractValues(exprList: Seq[Expression], exclusiveAttrs: Seq[Attribute]): Map[Attribute, Seq[ExtractValue]]

    Creates a map from root Attributes to non-redundant nested ExtractValues.

    Creates a map from root Attributes to non-redundant nested ExtractValues. Nested field accessors of exclusiveAttrs are not considered in nested fields aliasing.

  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  10. def getNewProjectList(projectList: Seq[NamedExpression], nestedFieldToAlias: Map[Expression, Alias]): Seq[NamedExpression]

    Replace the ExtractValues in a project list with aliased attributes.

  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  16. def replaceWithAliases(plan: LogicalPlan, nestedFieldToAlias: Map[Expression, Alias], attrToAliases: AttributeMap[Seq[Alias]]): LogicalPlan

    Replace the grandchildren of a plan with Projects of the nested fields as aliases, and replace the ExtractValue expressions with aliased attributes.

  17. def rewritePlanIfSubsetFieldsUsed(plan: LogicalPlan, exprList: Seq[Expression], exclusiveAttrs: Seq[Attribute]): Option[LogicalPlan]

    Rewrites a plan with aliases if only a subset of the nested fields are used.

  18. def rewritePlanWithAliases(plan: LogicalPlan, attributeToExtractValues: Map[Attribute, Seq[ExtractValue]]): LogicalPlan

    Replace nested columns to prune unused nested columns later.

  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. def unapply(plan: LogicalPlan): Option[LogicalPlan]
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped