Packages

class SparkReadConf extends AnyRef

A class for common Iceberg configs for Spark reads. Derived from Apache Iceberg's SparkReadConf class.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

  • Read options
  • Session configuration
  • Table metadata

The most specific value is set in read options and takes precedence over all other configs. If no read option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkReadConf
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkReadConf(spark: SparkSession, table: Table, readOptions: Map[String, String])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def asOfTimestamp(): Long
  6. def caseSensitive(): Boolean
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  8. def endSnapshotId(): Long
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def fileScanTaskSetId(): String
  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. def handleTimestampWithoutZone(): Boolean

    Enables reading a timestamp without time zone as a timestamp with time zone.

    Enables reading a timestamp without time zone as a timestamp with time zone.

    Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.

    When set to false (default), an exception must be thrown while reading a timestamp without time zone.

    returns

    boolean indicating if reading timestamps without timezone is allowed

  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. def localityEnabled(): Boolean
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  20. def orcBatchSize(): Int
  21. def orcVectorizationEnabled(): Boolean
  22. def parquetBatchSize(): Int
  23. def parquetVectorizationEnabled(): Boolean
  24. def snapshotId(): Long
  25. def splitLookback(): Int
  26. def splitLookbackOption(): Integer
  27. def splitOpenFileCost(): Long
  28. def splitOpenFileCostOption(): Long
  29. def splitSize(): Long
  30. def splitSizeOption(): Long
  31. def startSnapshotId(): Long
  32. def streamFromTimestamp(): Long
  33. def streamingSkipDeleteSnapshots(): Boolean
  34. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  35. def toString(): String
    Definition Classes
    AnyRef → Any
  36. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  38. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped