object DateTimeUtils

Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DateTimeUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final val JULIAN_DAY_OF_EPOCH: Int(2440588)
  5. val TIMEZONE_OPTION: String
  6. final val TimeZoneUTC: TimeZone
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def cleanLegacyTimestampStr(s: UTF8String): UTF8String
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  10. def convertSpecialDate(input: String, zoneId: ZoneId): Option[Int]

    Converts notational shorthands that are converted to ordinary dates.

    Converts notational shorthands that are converted to ordinary dates.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current date.

    returns

    Some of days since the epoch if the conversion completed successfully otherwise None.

  11. def convertSpecialTimestamp(input: String, zoneId: ZoneId): Option[Long]

    Converts notational shorthands that are converted to ordinary timestamps.

    Converts notational shorthands that are converted to ordinary timestamps.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current timestamp.

    returns

    Some of microseconds since the epoch if the conversion completed successfully otherwise None.

  12. def convertSpecialTimestampNTZ(input: String, zoneId: ZoneId): Option[Long]

    Converts notational shorthands that are converted to ordinary timestamps without time zone.

    Converts notational shorthands that are converted to ordinary timestamps without time zone.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current local timestamp.

    returns

    Some of microseconds since the epoch if the conversion completed successfully otherwise None.

  13. def convertTz(micros: Long, fromZone: ZoneId, toZone: ZoneId): Long

    Converts the timestamp micros from one timezone to another.

    Converts the timestamp micros from one timezone to another.

    Time-zone rules, such as daylight savings, mean that not every local date-time is valid for the toZone time zone, thus the local date-time may be adjusted.

  14. def currentDate(zoneId: ZoneId): Int

    Obtains the current date as days since the epoch in the specified time-zone.

  15. def currentTimestamp(): Long

    Obtains the current instant as microseconds since the epoch at the UTC time zone.

  16. def dateAddInterval(start: Int, interval: CalendarInterval): Int

    Adds the interval's months and days to a date expressed as days since the epoch.

    Adds the interval's months and days to a date expressed as days since the epoch.

    returns

    A date value, expressed in days since 1970-01-01.

    Exceptions thrown

    DateTimeException if the result exceeds the supported date range

    IllegalArgumentException if the interval has microseconds part

  17. def dateAddMonths(days: Int, months: Int): Int

    Adds an year-month interval to a date represented as days since 1970-01-01.

    Adds an year-month interval to a date represented as days since 1970-01-01.

    returns

    a date value, expressed in days since 1970-01-01.

  18. def daysToLocalDate(days: Int): LocalDate

    Obtains an instance of java.time.LocalDate from the epoch day count.

  19. def daysToMicros(days: Int, zoneId: ZoneId): Long

    Converts days since 1970-01-01 at the given zone ID to microseconds since 1970-01-01 00:00:00Z.

  20. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  22. def fromJavaDate(date: Date): Int

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part.

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part. The resulted days are rebased from the hybrid to Proleptic Gregorian calendar. The days rebasing is performed via UTC time zone for simplicity because the difference between two calendars is the same in any given time zone and UTC time zone.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions. The goal of the shift is to get a local date derived from the number of days that has the same date fields (year, month, day) as the original date at the default JVM time zone.

    date

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of days since the epoch in Proleptic Gregorian calendar.

  23. def fromJavaTimestamp(t: Timestamp): Long

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z. It extracts date-time fields from the input, builds a local timestamp in Proleptic Gregorian calendar from the fields, and binds the timestamp to the system time zone. The resulted instant is converted to microseconds since the epoch.

    The conversion is performed via the system time zone because it is used internally in java.sql.Timestamp while extracting date-time fields.

    The goal of the function is to have the same local date-time in the original calendar - the hybrid calendar (Julian + Gregorian) and in the target calendar which is Proleptic Gregorian calendar, see SPARK-26651.

    t

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of micros since epoch from java.sql.Timestamp.

  24. def fromJulianDay(days: Int, nanos: Long): Long

    Returns the number of microseconds since epoch from Julian day and nanoseconds in a day.

  25. def fromUTCTime(micros: Long, timeZone: String): Long

    Returns a timestamp of given timezone from UTC timestamp, with the same string representation in their timezone.

  26. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  27. def getDayInYear(days: Int): Int

    Returns the 'day in year' value for the given number of days since 1970-01-01.

  28. def getDayOfMonth(days: Int): Int

    Returns the 'day of month' value for the given number of days since 1970-01-01.

  29. def getDayOfWeek(days: Int): Int

    Returns the day of the week for the given number of days since 1970-01-01 (1 = Sunday, 2 = Monday, ..., 7 = Saturday).

  30. def getDayOfWeekFromString(string: UTF8String): Int

    Returns day of week from String.

    Returns day of week from String. Starting from Thursday, marked as 0. (Because 1970-01-01 is Thursday).

    Exceptions thrown

    IllegalArgumentException if the input is not a valid day of week.

  31. def getHours(micros: Long, zoneId: ZoneId): Int

    Returns the hour value of a given timestamp value.

    Returns the hour value of a given timestamp value. The timestamp is expressed in microseconds.

  32. def getLastDayOfMonth(days: Int): Int

    Returns last day of the month for the given number of days since 1970-01-01.

  33. def getMicroseconds(micros: Long, zoneId: ZoneId): Int

    Returns local seconds, including fractional parts, multiplied by 1000000.

    Returns local seconds, including fractional parts, multiplied by 1000000.

    micros

    The number of microseconds since the epoch.

    zoneId

    The time zone id which milliseconds should be obtained in.

  34. def getMinutes(micros: Long, zoneId: ZoneId): Int

    Returns the minute value of a given timestamp value.

    Returns the minute value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.

  35. def getMonth(days: Int): Int

    Returns the month value for the given number of days since 1970-01-01.

    Returns the month value for the given number of days since 1970-01-01. January is month 1.

  36. def getNextDateForDayOfWeek(startDay: Int, dayOfWeek: Int): Int

    Returns the first date which is later than startDate and is of the given dayOfWeek.

    Returns the first date which is later than startDate and is of the given dayOfWeek. dayOfWeek is an integer ranges in [0, 6], and 0 is Thu, 1 is Fri, etc,.

  37. def getQuarter(days: Int): Int

    Returns the quarter for the given number of days since 1970-01-01.

  38. def getSeconds(micros: Long, zoneId: ZoneId): Int

    Returns the second value of a given timestamp value.

    Returns the second value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.

  39. def getSecondsWithFraction(micros: Long, zoneId: ZoneId): Decimal

    Returns the seconds part and its fractional part with microseconds.

  40. def getTimeZone(timeZoneId: String): TimeZone
  41. def getWeekBasedYear(days: Int): Int

    Returns the year which conforms to ISO 8601.

    Returns the year which conforms to ISO 8601. Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January.

  42. def getWeekDay(days: Int): Int

    Returns the day of the week for the given number of days since 1970-01-01 (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).

  43. def getWeekOfYear(days: Int): Int

    Returns the week of the year of the given date expressed as the number of days from 1970-01-01.

    Returns the week of the year of the given date expressed as the number of days from 1970-01-01. A week is considered to start on a Monday and week 1 is the first week with > 3 days.

  44. def getYear(days: Int): Int

    Returns the year value for the given number of days since 1970-01-01.

  45. def getZoneId(timeZoneId: String): ZoneId
  46. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  47. def instantToMicros(instant: Instant): Long

    Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant.

    Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant. The epoch microsecond count is a simple incrementing count of microseconds where microsecond 0 is 1970-01-01 00:00:00Z.

  48. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  49. def localDateTimeToMicros(localDateTime: LocalDateTime): Long
  50. def localDateToDays(localDate: LocalDate): Int

    Converts the local date to the number of days since 1970-01-01.

  51. def microsToDays(micros: Long, zoneId: ZoneId): Int

    Converts microseconds since 1970-01-01 00:00:00Z to days since 1970-01-01 at the given zone ID.

  52. def microsToInstant(micros: Long): Instant

    Obtains an instance of java.time.Instant using microseconds from the epoch of 1970-01-01 00:00:00Z.

  53. def microsToLocalDateTime(micros: Long): LocalDateTime
  54. def microsToMillis(micros: Long): Long

    Converts the timestamp to milliseconds since epoch.

    Converts the timestamp to milliseconds since epoch. In Spark timestamp values have microseconds precision, so this conversion is lossy.

  55. def millisToMicros(millis: Long): Long

    Converts milliseconds since the epoch to microseconds.

  56. def monthsBetween(micros1: Long, micros2: Long, roundOff: Boolean, zoneId: ZoneId): Double

    Returns number of months between micros1 and micros2.

    Returns number of months between micros1 and micros2. micros1 and micros2 are expressed in microseconds since 1970-01-01. If micros1 is later than micros2, the result is positive.

    If micros1 and micros2 are on the same day of month, or both are the last day of month, returns, time of day will be ignored.

    Otherwise, the difference is calculated based on 31 days per month. The result is rounded to 8 decimal places if roundOff is set to true.

  57. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  58. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  59. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  60. def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId], Boolean)

    Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date.

    Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date. value. The return type is Option in order to distinguish between 0L and null. The following formats are allowed:

    [+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [+-]yyyy*-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] T[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]

    where zone_id should have one of the forms:

    • Z - Zulu time zone UTC+0
    • +|-[h]h:[m]m
    • A short id, see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html#SHORT_IDS
    • An id with one of the prefixes UTC+, UTC-, GMT+, GMT-, UT+ or UT-, and a suffix in the formats:
      • +|-h[h]
      • +|-hh[:]mm
      • +|-hh:mm:ss
      • +|-hhmmss
    • Region-based zone IDs in the form area/city, such as Europe/Paris
    returns

    timestamp segments, time zone id and whether the input is just time without a date. If the input string can't be parsed as timestamp, the result timestamp segments are empty.

  61. def parseTruncLevel(format: UTF8String): Int

    Returns the truncate level, could be from TRUNC_TO_MICROSECOND to TRUNC_TO_YEAR, or TRUNC_INVALID, TRUNC_INVALID means unsupported truncate level.

  62. def stringToDate(s: UTF8String): Option[Int]

    Trims and parses a given UTF8 date string to a corresponding Int value.

    Trims and parses a given UTF8 date string to a corresponding Int value. The return type is Option in order to distinguish between 0 and null. The following formats are allowed:

    [+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d * [+-]yyyy*-[m]m-[d]dT*

  63. def stringToDateAnsi(s: UTF8String): Int
  64. def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[Long]

    Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value.

    Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats

  65. def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId): Long
  66. def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long]

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch.

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch. The result is independent of time zones, which means that zone ID in the input string will be ignored. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats.

  67. def stringToTimestampWithoutTimeZoneAnsi(s: UTF8String): Long
  68. def subtractDates(endDay: Int, startDay: Int): CalendarInterval

    Subtracts two dates expressed as days since 1970-01-01.

    Subtracts two dates expressed as days since 1970-01-01.

    endDay

    The end date, exclusive

    startDay

    The start date, inclusive

    returns

    An interval between two dates. The interval can be negative if the end date is before the start date.

  69. def subtractTimestamps(endMicros: Long, startMicros: Long, zoneId: ZoneId): Long

    Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.

    Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.

    endMicros

    The end timestamp as microseconds since the epoch, exclusive

    startMicros

    The end timestamp as microseconds since the epoch, inclusive

    zoneId

    The time zone ID in which the subtraction is performed

    returns

    The difference in microseconds between local timestamps corresponded to the input instants end and start.

  70. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  71. def timestampAddDayTime(micros: Long, dayTime: Long, zoneId: ZoneId): Long

    Adds a day-time interval expressed in microseconds to a timestamp at the given time zone.

    Adds a day-time interval expressed in microseconds to a timestamp at the given time zone. It converts the input timestamp to a local timestamp, and adds the interval by:

    • Splitting the interval to days and microsecond adjustment in a day, and
    • First of all, it adds days and then the time part. The resulted local timestamp is converted back to an instant at the given time zone.
    micros

    The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

    dayTime

    The amount of microseconds to add. It can be positive or negative.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  72. def timestampAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long

    Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.

    Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  73. def timestampAddMonths(micros: Long, months: Int, zoneId: ZoneId): Long

    Adds months to a timestamp at the given time zone.

    Adds months to a timestamp at the given time zone. It converts the input timestamp to a local timestamp at the given time zone, adds months, and converts the resulted local timestamp back to a timestamp, expressed in microseconds since 1970-01-01 00:00:00Z.

    micros

    The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z

    months

    The amount of months to add. It can be positive or negative.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  74. def timestampNTZAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long

    Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.

    Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.

    returns

    A timestamp without time zone value, expressed in range [0001-01-01T00:00:00.000000, 9999-12-31T23:59:59.999999].

  75. def toJavaDate(days: Int): Date

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian).

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian). It rebases the given days from Proleptic Gregorian to the hybrid calendar at UTC time zone for simplicity because the difference between two calendars doesn't depend on any time zone. The result is shifted by the time zone offset in wall clock to have the same date fields (year, month, day) at the default JVM time zone as the input daysSinceEpoch in Proleptic Gregorian calendar.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions.

    days

    The number of days since 1970-01-01 in Proleptic Gregorian calendar.

    returns

    A local date in the hybrid calendar as java.sql.Date from number of days since epoch.

  76. def toJavaTimestamp(micros: Long): Timestamp

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    The conversion is based on the JVM system time zone because the java.sql.Timestamp uses the time zone internally.

    The method performs the conversion via local timestamp fields to have the same date-time representation as year, month, day, ..., seconds in the original calendar and in the target calendar.

    micros

    The number of microseconds since 1970-01-01T00:00:00.000000Z.

    returns

    A java.sql.Timestamp from number of micros since epoch.

  77. def toJulianDay(micros: Long): (Int, Long)

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Note: support timestamp since 4717 BC (without negative nanoseconds, compatible with Hive).

  78. def toString(): String
    Definition Classes
    AnyRef → Any
  79. def toUTCTime(micros: Long, timeZone: String): Long

    Returns a utc timestamp from a given timestamp from a given timezone, with the same string representation in their timezone.

  80. def truncDate(days: Int, level: Int): Int

    Returns the trunc date from original date and trunc level.

    Returns the trunc date from original date and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 6 and 9.

  81. def truncTimestamp(micros: Long, level: Int, zoneId: ZoneId): Long

    Returns the trunc date time from original date time and trunc level.

    Returns the trunc date time from original date time and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 0 and 9.

  82. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  84. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped