Package org.apache.kafka.streams.processor.internals

Class StreamTask

java.lang.Object

org.apache.kafka.streams.processor.internals.AbstractTask

org.apache.kafka.streams.processor.internals.StreamTask

All Implemented Interfaces:

ProcessorNodePunctuator, Task

public class StreamTask
extends AbstractTask
implements ProcessorNodePunctuator, Task

A StreamTask is associated with a PartitionGroup, and is assigned to a StreamThread for processing.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.kafka.streams.processor.internals.Task

Task.State, Task.TaskType

Field Summary

Fields inherited from class org.apache.kafka.streams.processor.internals.AbstractTask

id, inputPartitions, stateDirectory, stateMgr, topology

Fields inherited from interface org.apache.kafka.streams.processor.internals.Task

LATEST_OFFSET

Constructor Summary

Constructors

Constructor

Description

StreamTask(TaskId id, java.util.Set<org.apache.kafka.common.TopicPartition> partitions, ProcessorTopology topology, org.apache.kafka.clients.consumer.Consumer<byte[],byte[]> mainConsumer, StreamsConfig config, StreamsMetricsImpl streamsMetrics, StateDirectory stateDirectory, ThreadCache cache, org.apache.kafka.common.utils.Time time, ProcessorStateManager stateMgr, RecordCollector recordCollector, InternalProcessorContext processorContext)

Method Summary

Modifier and Type	Method	Description
void	addRecords(org.apache.kafka.common.TopicPartition partition, java.lang.Iterable<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> records)	Adds records to queues.
java.util.Map<org.apache.kafka.common.TopicPartition,java.lang.Long>	changelogOffsets()
void	closeClean()	Must be idempotent.
void	closeCleanAndRecycleState()	Attempt a clean close but do not close the underlying state
void	closeDirty()	Must be idempotent.
boolean	commitNeeded()
boolean	commitRequested()	Whether or not a request has been made to commit the current state
void	completeRestoration()
boolean	hasRecordsQueued()
void	initializeIfNeeded()
boolean	isActive()
boolean	isProcessable(long wallClockTime)	An active task is processable if its buffer contains data for all of its input source topic partitions, or if it is enforced to be processable
boolean	maybePunctuateStreamTime()	Possibly trigger registered stream-time punctuation functions if current partition group timestamp has reached the defined stamp Note, this is only called in the presence of new records
boolean	maybePunctuateSystemTime()	Possibly trigger registered system-time punctuation functions if current system timestamp has reached the defined stamp Note, this is called irrespective of the presence of new records
void	postCommit()	This should only be called if the attempted commit succeeded for this task
java.util.Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>	prepareCommit()
boolean	process(long wallClockTime)	Process one record.
InternalProcessorContext	processorContext()
void	punctuate(ProcessorNode<?,?> node, long timestamp, PunctuationType type, Punctuator punctuator)
java.util.Map<org.apache.kafka.common.TopicPartition,java.lang.Long>	purgeableOffsets()
void	recordProcessBatchTime(long processBatchTime)
void	recordProcessTimeRatioAndBufferSize(long allTaskProcessMs, long now)
void	resume()	- resume the task
Cancellable	schedule(long interval, PunctuationType type, Punctuator punctuator)	Schedules a punctuation for the processor
void	suspend()
java.lang.String	toString()	Produces a string representation containing useful information about a Task.
java.lang.String	toString(java.lang.String indent)	Produces a string representation containing useful information about a Task starting with the given indent.
void	update(java.util.Set<org.apache.kafka.common.TopicPartition> topicPartitions, java.util.Map<java.lang.String,java.util.List<java.lang.String>> allTopologyNodesToSourceTopics)	Updates input partitions and topology after rebalance

Methods inherited from class org.apache.kafka.streams.processor.internals.AbstractTask

changelogPartitions, getStore, id, inputPartitions, isClosed, markChangelogAsCorrupted, revive, state

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.kafka.streams.processor.internals.Task

changelogPartitions, getStore, id, inputPartitions, isClosed, markChangelogAsCorrupted, needsInitializationOrRestoration, revive, state

Constructor Detail

StreamTask

public StreamTask(TaskId id,
                  java.util.Set<org.apache.kafka.common.TopicPartition> partitions,
                  ProcessorTopology topology,
                  org.apache.kafka.clients.consumer.Consumer<byte[],byte[]> mainConsumer,
                  StreamsConfig config,
                  StreamsMetricsImpl streamsMetrics,
                  StateDirectory stateDirectory,
                  ThreadCache cache,
                  org.apache.kafka.common.utils.Time time,
                  ProcessorStateManager stateMgr,
                  RecordCollector recordCollector,
                  InternalProcessorContext processorContext)

Method Detail

isActive

public boolean isActive()

Specified by:

isActive in interface Task

initializeIfNeeded

public void initializeIfNeeded()

Specified by:

initializeIfNeeded in interface Task

Throws:

LockException - could happen when multi-threads within the single instance, could retry

org.apache.kafka.common.errors.TimeoutException - if initializing record collector timed out

StreamsException - fatal error, should close the thread

completeRestoration

public void completeRestoration()

Specified by:

completeRestoration in interface Task

Throws:

org.apache.kafka.common.errors.TimeoutException - if fetching committed offsets timed out

suspend

public void suspend()

Specified by:

suspend in interface Task

resume

public void resume()

 - resume the task
 

Specified by:

resume in interface Task

prepareCommit

public java.util.Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> prepareCommit()

Specified by:

prepareCommit in interface Task

Returns:

offsets that should be committed for this task

postCommit

public void postCommit()

This should only be called if the attempted commit succeeded for this task

Specified by:

postCommit in interface Task

closeClean

public void closeClean()

Description copied from interface: Task

Must be idempotent.

Specified by:

closeClean in interface Task

closeDirty

public void closeDirty()

Description copied from interface: Task

Must be idempotent.

Specified by:

closeDirty in interface Task

update

public void update(java.util.Set<org.apache.kafka.common.TopicPartition> topicPartitions,
                   java.util.Map<java.lang.String,java.util.List<java.lang.String>> allTopologyNodesToSourceTopics)

Description copied from interface: Task

Updates input partitions and topology after rebalance

Specified by:

update in interface Task

Overrides:

update in class AbstractTask

closeCleanAndRecycleState

public void closeCleanAndRecycleState()

Description copied from interface: Task

Attempt a clean close but do not close the underlying state

Specified by:

closeCleanAndRecycleState in interface Task

isProcessable

public boolean isProcessable(long wallClockTime)

An active task is processable if its buffer contains data for all of its input source topic partitions, or if it is enforced to be processable

process

public boolean process(long wallClockTime)

Process one record.

Specified by:

process in interface Task

Returns:

true if this method processes a record, false if it does not process a record.

Throws:

TaskMigratedException - if the task producer got fenced (EOS only)

recordProcessBatchTime

public void recordProcessBatchTime(long processBatchTime)

Specified by:

recordProcessBatchTime in interface Task

recordProcessTimeRatioAndBufferSize

public void recordProcessTimeRatioAndBufferSize(long allTaskProcessMs,
                                                long now)

Specified by:

recordProcessTimeRatioAndBufferSize in interface Task

punctuate

public void punctuate(ProcessorNode<?,?> node,
                      long timestamp,
                      PunctuationType type,
                      Punctuator punctuator)

Specified by:

punctuate in interface ProcessorNodePunctuator

Throws:

java.lang.IllegalStateException - if the current node is not null

TaskMigratedException - if the task producer got fenced (EOS only)

purgeableOffsets

public java.util.Map<org.apache.kafka.common.TopicPartition,java.lang.Long> purgeableOffsets()

Specified by:

purgeableOffsets in interface Task

addRecords

public void addRecords(org.apache.kafka.common.TopicPartition partition,
                       java.lang.Iterable<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> records)

Adds records to queues. If a record has an invalid (i.e., negative) timestamp, the record is skipped and not added to the queue for processing

Specified by:

addRecords in interface Task

Parameters:

partition - the partition

records - the records

schedule

public Cancellable schedule(long interval,
                            PunctuationType type,
                            Punctuator punctuator)

Schedules a punctuation for the processor

Parameters:

interval - the interval in milliseconds

type - the punctuation type

Throws:

java.lang.IllegalStateException - if the current node is not null

maybePunctuateStreamTime

public boolean maybePunctuateStreamTime()

Possibly trigger registered stream-time punctuation functions if current partition group timestamp has reached the defined stamp Note, this is only called in the presence of new records

Specified by:

maybePunctuateStreamTime in interface Task

Throws:

TaskMigratedException - if the task producer got fenced (EOS only)

maybePunctuateSystemTime

public boolean maybePunctuateSystemTime()

Possibly trigger registered system-time punctuation functions if current system timestamp has reached the defined stamp Note, this is called irrespective of the presence of new records

Specified by:

maybePunctuateSystemTime in interface Task

Throws:

TaskMigratedException - if the task producer got fenced (EOS only)

commitRequested

public boolean commitRequested()

Whether or not a request has been made to commit the current state

Specified by:

commitRequested in interface Task

processorContext

public InternalProcessorContext processorContext()

toString

public java.lang.String toString()

Produces a string representation containing useful information about a Task. This is useful in debugging scenarios.

Overrides:

toString in class java.lang.Object

Returns:

A string representation of the StreamTask instance.

toString

public java.lang.String toString(java.lang.String indent)

Produces a string representation containing useful information about a Task starting with the given indent. This is useful in debugging scenarios.

Returns:

A string representation of the Task instance.

commitNeeded

public boolean commitNeeded()

Specified by:

commitNeeded in interface Task

changelogOffsets

public java.util.Map<org.apache.kafka.common.TopicPartition,java.lang.Long> changelogOffsets()

Specified by:

changelogOffsets in interface Task

Returns:

the offsets of all the changelog partitions associated with this task, indicating the current positions of the logged state stores of the task.

hasRecordsQueued

public boolean hasRecordsQueued()