org.apache.mahout.sparkbindings.drm

CheckpointedDrmSpark

Related Doc: package drm

class CheckpointedDrmSpark[K] extends CheckpointedDrm[K]

Spark-specific optimizer-checkpointed DRM.

K

matrix key type (e.g. the keys of sequence files once persisted)

Linear Supertypes
CheckpointedDrm[K], DrmLike[K], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CheckpointedDrmSpark
  2. CheckpointedDrm
  3. DrmLike
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CheckpointedDrmSpark(rddInput: DrmRddInput[K], _nrow: Long = 1L, _ncol: Int = 1, cacheHint: CacheHint = CacheHint.NONE, partitioningTag: Long = Random.nextLong(), _canHaveMissingRows: Boolean = false)(implicit arg0: ClassTag[K])

    rddInput

    underlying rdd to wrap over.

    _nrow

    number of rows; if unspecified, we will compute with an inexpensive traversal.

    _ncol

    number of columns; if unspecified, we will try to guess with an inexpensive traversal.

    cacheHint

    cache level to use. (Implementors usually want to override the default!)

    partitioningTag

    unique partitioning tag. Used to detect identically partitioned operands.

    _canHaveMissingRows

    true if the matrix is int-keyed, and if it also may have missing rows (will require a lazy fix for some physical operations.

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  5. def cache(): CheckpointedDrmSpark[K]

  6. val cacheHint: CacheHint

    cache level to use.

    cache level to use. (Implementors usually want to override the default!)

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  7. lazy val canHaveMissingRows: Boolean

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  8. def checkpoint(cacheHint: CacheHint): CheckpointedDrm[K]

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def collect: Matrix

    Collecting DRM to fron-end in-core Matrix.

    Collecting DRM to fron-end in-core Matrix.

    If key in DRM is Int, then matrix is collected using key as row index. Otherwise, order of rows in result is undefined but key.toString is applied as rowLabelBindings of the in-core matrix .

    Note that this pre-allocates target matrix and then assigns collected RDD to it thus this likely would require about 2 times the RDD memory

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  11. def computeNCol: Int

    Attributes
    protected
  12. def computeNNonZero: Long

    Attributes
    protected
  13. def computeNRow: Long

    Attributes
    protected
  14. val context: DistributedContext

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  15. def dfsWrite(path: String): Unit

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    path

    output path to dump Matrix to

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  17. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  18. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  19. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  20. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  21. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  22. def keyClassTag: ClassTag[K]

    Explicit extraction of key class Tag

    Explicit extraction of key class Tag

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  23. lazy val ncol: Int

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  24. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  25. def newRowCardinality(n: Int): CheckpointedDrm[K]

    Changes the number of rows in the DRM without actually touching the underlying data.

    Changes the number of rows in the DRM without actually touching the underlying data. Used to redimension a DRM after it has been created, which implies some blank, non-existent rows.

    n

    new row dimension

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  26. final def notify(): Unit

    Definition Classes
    AnyRef
  27. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  28. lazy val nrow: Long

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  29. val partitioningTag: Long

    unique partitioning tag.

    unique partitioning tag. Used to detect identically partitioned operands.

    Attributes
    protected[org.apache.mahout]
    Definition Classes
    CheckpointedDrmSpark → DrmLike
  30. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  31. def toString(): String

    Definition Classes
    AnyRef → Any
  32. def uncache(): CheckpointedDrmSpark.this.type

    if matrix was previously persisted into cache, delete cached representation

    if matrix was previously persisted into cache, delete cached representation

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  33. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from CheckpointedDrm[K]

Inherited from DrmLike[K]

Inherited from AnyRef

Inherited from Any

Ungrouped