Class UsePreviousTimeOnInvalidTimestamp

  • All Implemented Interfaces:
    TimestampExtractor

    @Evolving
    public class UsePreviousTimeOnInvalidTimestamp
    extends java.lang.Object
    Retrieves embedded metadata timestamps from Kafka messages. If a record has a negative (invalid) timestamp, a new timestamp will be inferred from the current stream-time.

    Embedded metadata timestamp was introduced in "KIP-32: Add timestamps to Kafka message" for the new 0.10+ Kafka message format.

    Here, "embedded metadata" refers to the fact that compatible Kafka producer clients automatically and transparently embed such timestamps into message metadata they send to Kafka, which can then be retrieved via this timestamp extractor.

    If the embedded metadata timestamp represents CreateTime (cf. Kafka broker setting message.timestamp.type and Kafka topic setting log.message.timestamp.type), this extractor effectively provides event-time semantics. If LogAppendTime is used as broker/topic setting to define the embedded metadata timestamps, using this extractor effectively provides ingestion-time semantics.

    If you need processing-time semantics, use WallclockTimestampExtractor.

    See Also:
    FailOnInvalidTimestamp, LogAndSkipOnInvalidTimestamp, WallclockTimestampExtractor
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      long extract​(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Object,​java.lang.Object> record, long previousTimestamp)
      Extracts the embedded metadata timestamp from the given ConsumerRecord.
      long onInvalidTimestamp​(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Object,​java.lang.Object> record, long recordTimestamp, long previousTimestamp)
      Returns the current stream-time as new timestamp for the record.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • UsePreviousTimeOnInvalidTimestamp

        public UsePreviousTimeOnInvalidTimestamp()
    • Method Detail

      • onInvalidTimestamp

        public long onInvalidTimestamp​(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Object,​java.lang.Object> record,
                                       long recordTimestamp,
                                       long previousTimestamp)
                                throws StreamsException
        Returns the current stream-time as new timestamp for the record.
        Parameters:
        record - a data record
        recordTimestamp - the timestamp extractor from the record
        previousTimestamp - the latest extracted valid timestamp of the current record's partition˙ (could be -1 if unknown)
        Returns:
        the provided latest extracted valid timestamp as new timestamp for the record
        Throws:
        StreamsException - if latest extracted valid timestamp is unknown
      • extract

        public long extract​(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.Object,​java.lang.Object> record,
                            long previousTimestamp)
        Extracts the embedded metadata timestamp from the given ConsumerRecord.
        Specified by:
        extract in interface TimestampExtractor
        Parameters:
        record - a data record
        previousTimestamp - the latest extracted valid timestamp of the current record's partition˙ (could be -1 if unknown)
        Returns:
        the embedded metadata timestamp of the given ConsumerRecord