object ColumnarRdd
This provides a way to get back out GPU Columnar data RDD[Table]. Each Table will have the same
schema as the dataframe passed in. If the schema of the dataframe is something that Rapids does
not currently support an IllegalArgumentException will be thrown.
The size of each table will be determined by what is producing that table but typically will be about the number of bytes set by RapidsConf.GPU_BATCH_SIZE_BYTES.
Table is not a typical thing in an RDD so special care needs to be taken when working with it. By default it is not serializable so repartitioning the RDD or any other operator that involves a shuffle will not work. This is because it is very expensive to serialize and deserialize a GPU Table using a conventional spark shuffle. Also most of the memory associated with the Table is on the GPU itself, so each table must be closed when it is no longer needed to avoid running out of GPU memory. By convention it is the responsibility of the one consuming the data to close it when they no longer need it.
- Alphabetic
- By Inheritance
- ColumnarRdd
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def apply(df: DataFrame): RDD[Table]
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
- def convert(df: DataFrame): RDD[Table]
- lazy val convertMethod: Method
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )