Packages

c

org.apache.spark.sql.execution.datasources

SingleDirectoryDataWriter

class SingleDirectoryDataWriter extends FileFormatDataWriter

Writes data to a single directory (used for non-dynamic-partition writes).

Linear Supertypes
FileFormatDataWriter, DataWriter[InternalRow], Closeable, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SingleDirectoryDataWriter
  2. FileFormatDataWriter
  3. DataWriter
  4. Closeable
  5. AutoCloseable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SingleDirectoryDataWriter(description: WriteJobDescription, taskAttemptContext: TaskAttemptContext, committer: FileCommitProtocol, customMetrics: Map[String, SQLMetric] = Map.empty)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val MAX_FILE_COUNTER: Int

    Max number of files a single task writes out due to file size.

    Max number of files a single task writes out due to file size. In most cases the number of files written should be very small. This is just a safe guard to protect some really bad settings, e.g. maxRecordsPerFile = 1.

    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  5. def abort(): Unit
    Definition Classes
    FileFormatDataWriter → DataWriter
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  8. def close(): Unit
    Definition Classes
    FileFormatDataWriter → Closeable → AutoCloseable
  9. def commit(): WriteTaskResult

    Returns the summary of relative information which includes the list of partition strings written out.

    Returns the summary of relative information which includes the list of partition strings written out. The list of partitions is sent back to the driver and used to update the catalog. Other information will be sent back to the driver too and used to e.g. update the metrics in UI.

    Definition Classes
    FileFormatDataWriter → DataWriter
  10. def currentMetricsValues(): Array[CustomTaskMetric]
    Definition Classes
    DataWriter
  11. var currentWriter: OutputWriter
    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  20. def releaseCurrentWriter(): Unit

    Release resources of currentWriter.

    Release resources of currentWriter.

    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  21. def releaseResources(): Unit

    Release all resources.

    Release all resources.

    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  22. val statsTrackers: Seq[WriteTaskStatsTracker]

    Trackers for computing various statistics on the data as it's being written out.

    Trackers for computing various statistics on the data as it's being written out.

    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. val updatedPartitions: Set[String]
    Attributes
    protected
    Definition Classes
    FileFormatDataWriter
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. def write(record: InternalRow): Unit

    Writes a record.

    Writes a record.

    Definition Classes
    SingleDirectoryDataWriter → FileFormatDataWriter → DataWriter
  30. def writeWithIterator(iterator: Iterator[InternalRow]): Unit

    Write an iterator of records.

    Write an iterator of records.

    Definition Classes
    FileFormatDataWriter
  31. def writeWithMetrics(record: InternalRow, count: Long): Unit
    Definition Classes
    FileFormatDataWriter

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from FileFormatDataWriter

Inherited from DataWriter[InternalRow]

Inherited from Closeable

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped