Packages

trait Partitioning extends AnyRef

Describes how an operator's output is split across partitions. It has 2 major properties:

  1. number of partitions. 2. if it can satisfy a given distribution.
Linear Supertypes
AnyRef, Any
Known Subclasses
BroadcastPartitioning, HashPartitioning, KeyGroupedPartitioning, PartitioningCollection, RangePartitioning, RoundRobinPartitioning, SinglePartition, UnknownPartitioning
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Partitioning
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract val numPartitions: Int

    Returns the number of partitions that the data is split across

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. def createShuffleSpec(distribution: ClusteredDistribution): ShuffleSpec

    Creates a shuffle spec for this partitioning and its required distribution.

    Creates a shuffle spec for this partitioning and its required distribution. The spec is used in the scenario where an operator has multiple children (e.g., join), and is used to decide whether this child is co-partitioned with others, therefore whether extra shuffle shall be introduced.

    distribution

    the required clustered distribution for this partitioning

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. final def satisfies(required: Distribution): Boolean

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e.

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e. the current dataset does not need to be re-partitioned for the required Distribution (it is possible that tuples within a partition need to be reorganized).

    A Partitioning can never satisfy a Distribution if its numPartitions doesn't match Distribution.requiredNumPartitions.

  16. def satisfies0(required: Distribution): Boolean

    The actual method that defines whether this Partitioning can satisfy the given Distribution, after the numPartitions check.

    The actual method that defines whether this Partitioning can satisfy the given Distribution, after the numPartitions check.

    By default a Partitioning can satisfy UnspecifiedDistribution, and AllTuples if the Partitioning only have one partition. Implementations can also overwrite this method with special logic.

    Attributes
    protected
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped