Class StreamTask

    • Method Detail

      • initializeStateStores

        public boolean initializeStateStores()
        Description copied from interface: Task
        Initialize the task and return true if the task is ready to run, i.e, it has not state stores
        Specified by:
        initializeStateStores in interface Task
        Returns:
        true if this task has no state stores that may need restoring.
      • initializeTopology

        public void initializeTopology()
         - (re-)initialize the topology of the task
         
        Specified by:
        initializeTopology in interface Task
        Throws:
        TaskMigratedException - if the task producer got fenced (EOS only)
      • resume

        public void resume()
         - resume the task
         
        Specified by:
        resume in interface Task
      • process

        public boolean process()
        Process one record.
        Returns:
        true if this method processes a record, false if it does not process a record.
        Throws:
        TaskMigratedException - if the task producer got fenced (EOS only)
      • commit

        public void commit()
         - flush state and producer
         - if(!eos) write checkpoint
         - commit offsets and start new transaction
         
        Specified by:
        commit in interface Task
        Throws:
        TaskMigratedException - if committing offsets failed (non-EOS) or if the task producer got fenced (EOS)
      • activeTaskCheckpointableOffsets

        protected java.util.Map<org.apache.kafka.common.TopicPartition,​java.lang.Long> activeTaskCheckpointableOffsets()
        Overrides:
        activeTaskCheckpointableOffsets in class AbstractTask
      • flushState

        protected void flushState()
      • suspend

        public void suspend()
         - close topology
         - commit()
           - flush state and producer
           - if (!eos) write checkpoint
           - commit offsets
         
        Specified by:
        suspend in interface Task
        Throws:
        TaskMigratedException - if committing offsets failed (non-EOS) or if the task producer got fenced (EOS)
      • closeSuspended

        public void closeSuspended​(boolean clean,
                                   boolean isZombie,
                                   java.lang.RuntimeException firstException)
        Specified by:
        closeSuspended in interface Task
      • close

        public void close​(boolean clean,
                          boolean isZombie)
         - suspend(clean)
           - close topology
           - if (clean) commit()
             - flush state and producer
             - commit offsets
         - close state
           - if (clean) write checkpoint
         - if (eos) close producer
         
        Specified by:
        close in interface Task
        Parameters:
        clean - shut down cleanly (ie, incl. flush and commit) if true -- otherwise, just close open resources
        isZombie - true is this task is a zombie or not (this will repress TaskMigratedException
        Throws:
        TaskMigratedException - if committing offsets failed (non-EOS) or if the task producer got fenced (EOS)
      • addRecords

        public void addRecords​(org.apache.kafka.common.TopicPartition partition,
                               java.lang.Iterable<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>> records)
        Adds records to queues. If a record has an invalid (i.e., negative) timestamp, the record is skipped and not added to the queue for processing
        Parameters:
        partition - the partition
        records - the records
      • schedule

        public Cancellable schedule​(long interval,
                                    PunctuationType type,
                                    Punctuator punctuator)
        Schedules a punctuation for the processor
        Parameters:
        interval - the interval in milliseconds
        type - the punctuation type
        Throws:
        java.lang.IllegalStateException - if the current node is not null
      • maybePunctuateStreamTime

        public boolean maybePunctuateStreamTime()
        Possibly trigger registered stream-time punctuation functions if current partition group timestamp has reached the defined stamp Note, this is only called in the presence of new records
        Throws:
        TaskMigratedException - if the task producer got fenced (EOS only)
      • maybePunctuateSystemTime

        public boolean maybePunctuateSystemTime()
        Possibly trigger registered system-time punctuation functions if current system timestamp has reached the defined stamp Note, this is called irrespective of the presence of new records
        Throws:
        TaskMigratedException - if the task producer got fenced (EOS only)