Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Definition Classes

  apache

•  package sql

  Allows the execution of relational queries, including those expressed in SQL using Spark.

  Allows the execution of relational queries, including those expressed in SQL using Spark.

  Definition Classes

  spark

•  package execution

  The physical execution component of Spark SQL.

  The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

  Definition Classes

  sql

•  package adaptive

  Definition Classes

  execution

•  case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements) extends Rule[SparkPlan] with Product with Serializable

  A rule to optimize skewed joins to avoid straggler tasks whose share of data are significantly larger than those of the rest of the tasks.

  A rule to optimize skewed joins to avoid straggler tasks whose share of data are significantly larger than those of the rest of the tasks.

  The general idea is to divide each skew partition into smaller partitions and replicate its matching partition on the other side of the join so that they can run in parallel tasks. Note that when matching partitions from the left side and the right side both have skew, it will become a cartesian product of splits from left and right joining together.

  For example, assume the Sort-Merge join has 4 partitions: left: [L1, L2, L3, L4] right: [R1, R2, R3, R4]

  Let's say L2, L4 and R3, R4 are skewed, and each of them get split into 2 sub-partitions. This is scheduled to run 4 tasks at the beginning: (L1, R1), (L2, R2), (L3, R3), (L4, R4). This rule expands it to 9 tasks to increase parallelism: (L1, R1), (L2-1, R2), (L2-2, R2), (L3, R3-1), (L3, R3-2), (L4-1, R4-1), (L4-2, R4-1), (L4-1, R4-2), (L4-2, R4-2)

  Definition Classes

  adaptive

• ShuffleStage

org.apache.spark.sql.execution.adaptive.OptimizeSkewedJoin

ShuffleStage 