Package org.apache.kafka.streams

Class StreamsBuilder

  • public class StreamsBuilder
extends java.lang.Object
    StreamsBuilder provide the high-level Kafka Streams DSL to specify a Kafka Streams topology.
    See Also:
    Topology, KStream, KTable, GlobalKTable

    • Constructor Detail

      • StreamsBuilder

        public StreamsBuilder()

    • Method Detail

      • stream

        public <K,​V> KStream<K,​V> stream​(java.lang.String topic)
        Create a KStream from the specified topics. The default "auto.offset.reset" strategy, default TimestampExtractor, and default key and value deserializers as specified in the config are used.

        If multiple topics are specified there is no ordering guarantee for records from different topics.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topic - the topic name; cannot be null
        Returns:
        a KStream for the specified topics

      • stream

        public <K,​V> KStream<K,​V> stream​(java.lang.String topic,
                                             Consumed<K,​V> consumed)
        Create a KStream from the specified topics. The "auto.offset.reset" strategy, TimestampExtractor, key and value deserializers are defined by the options in Consumed are used.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topic - the topic names; cannot be null
        consumed - the instance of Consumed used to define optional parameters
        Returns:
        a KStream for the specified topics

      • stream

        public <K,​V> KStream<K,​V> stream​(java.util.Collection<java.lang.String> topics)
        Create a KStream from the specified topics. The default "auto.offset.reset" strategy, default TimestampExtractor, and default key and value deserializers as specified in the config are used.

        If multiple topics are specified there is no ordering guarantee for records from different topics.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topics - the topic names; must contain at least one topic name
        Returns:
        a KStream for the specified topics

      • stream

        public <K,​V> KStream<K,​V> stream​(java.util.Collection<java.lang.String> topics,
                                             Consumed<K,​V> consumed)
        Create a KStream from the specified topics. The "auto.offset.reset" strategy, TimestampExtractor, key and value deserializers are defined by the options in Consumed are used.

        If multiple topics are specified there is no ordering guarantee for records from different topics.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topics - the topic names; must contain at least one topic name
        consumed - the instance of Consumed used to define optional parameters
        Returns:
        a KStream for the specified topics

      • stream

        public <K,​V> KStream<K,​V> stream​(java.util.regex.Pattern topicPattern)
        Create a KStream from the specified topic pattern. The default "auto.offset.reset" strategy, default TimestampExtractor, and default key and value deserializers as specified in the config are used.

        If multiple topics are matched by the specified pattern, the created KStream will read data from all of them and there is no ordering guarantee between records from different topics.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topicPattern - the pattern to match for topic names
        Returns:
        a KStream for topics matching the regex pattern.

      • stream

        public <K,​V> KStream<K,​V> stream​(java.util.regex.Pattern topicPattern,
                                             Consumed<K,​V> consumed)
        Create a KStream from the specified topic pattern. The "auto.offset.reset" strategy, TimestampExtractor, key and value deserializers are defined by the options in Consumed are used.

        If multiple topics are matched by the specified pattern, the created KStream will read data from all of them and there is no ordering guarantee between records from different topics.

        Note that the specified input topics must be partitioned by key. If this is not the case it is the user's responsibility to repartition the data before any key based operation (like aggregation or join) is applied to the returned KStream.

        Parameters:
        topicPattern - the pattern to match for topic names
        consumed - the instance of Consumed used to define optional parameters
        Returns:
        a KStream for topics matching the regex pattern.

      • table

        public <K,​V> KTable<K,​V> table​(java.lang.String topic,
                                           Consumed<K,​V> consumed,
                                           Materialized<K,​V,​KeyValueStore<org.apache.kafka.common.utils.Bytes,​byte[]>> materialized)
        Create a KTable for the specified topic. The "auto.offset.reset" strategy, TimestampExtractor, key and value deserializers are defined by the options in Consumed are used. Input records with null key will be dropped.

        Note that the specified input topic must be partitioned by key. If this is not the case the returned KTable will be corrupted.

        The resulting KTable will be materialized in a local KeyValueStore using the given Materialized instance. However, no internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        You should only specify serdes in the Consumed instance as these will also be used to overwrite the serdes in Materialized, i.e., 

         
 streamBuilder.table(topic, Consumed.with(Serde.String(), Serde.String(), Materialized.<String, String, KeyValueStore<Bytes, byte[]>as(storeName))
        To query the local KeyValueStore it must be obtained via KafkaStreams#store(...): 
        
 KafkaStreams streams = ...
 ReadOnlyKeyValueStore<String, Long> localStore = streams.store(queryableStoreName, QueryableStoreTypes.<String, Long>keyValueStore());
 String key = "some-key";
 Long valueForKey = localStore.get(key); // key must be local (application state is shared over all running Kafka Streams instances)
        For non-local keys, a custom RPC mechanism must be implemented using KafkaStreams.allMetadata() to query the value of the key on a parallel running instance of your Kafka Streams application.
        Parameters:
        topic - the topic name; cannot be null
        consumed - the instance of Consumed used to define optional parameters; cannot be null
        materialized - the instance of Materialized used to materialize a state store; cannot be null
        Returns:
        a KTable for the specified topic

      • table

        public <K,​V> KTable<K,​V> table​(java.lang.String topic)
        Create a KTable for the specified topic. The default "auto.offset.reset" strategy and default key and value deserializers as specified in the config are used. Input records with null key will be dropped.

        Note that the specified input topics must be partitioned by key. If this is not the case the returned KTable will be corrupted.

        The resulting KTable will be materialized in a local KeyValueStore with an internal store name. Note that store name may not be queriable through Interactive Queries. No internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        Parameters:
        topic - the topic name; cannot be null
        Returns:
        a KTable for the specified topic

      • table

        public <K,​V> KTable<K,​V> table​(java.lang.String topic,
                                           Consumed<K,​V> consumed)
        Create a KTable for the specified topic. The "auto.offset.reset" strategy, TimestampExtractor, key and value deserializers are defined by the options in Consumed are used. Input records with null key will be dropped.

        Note that the specified input topics must be partitioned by key. If this is not the case the returned KTable will be corrupted.

        The resulting KTable will be materialized in a local KeyValueStore with an internal store name. Note that store name may not be queriable through Interactive Queries. No internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        Parameters:
        topic - the topic name; cannot be null
        consumed - the instance of Consumed used to define optional parameters; cannot be null
        Returns:
        a KTable for the specified topic

      • table

        public <K,​V> KTable<K,​V> table​(java.lang.String topic,
                                           Materialized<K,​V,​KeyValueStore<org.apache.kafka.common.utils.Bytes,​byte[]>> materialized)
        Create a KTable for the specified topic. The default "auto.offset.reset" strategy as specified in the config are used. Key and value deserializers as defined by the options in Materialized are used. Input records with null key will be dropped.

        Note that the specified input topics must be partitioned by key. If this is not the case the returned KTable will be corrupted.

        The resulting KTable will be materialized in a local KeyValueStore using the Materialized instance. No internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        Parameters:
        topic - the topic name; cannot be null
        materialized - the instance of Materialized used to materialize a state store; cannot be null
        Returns:
        a KTable for the specified topic

      • globalTable

        public <K,​V> GlobalKTable<K,​V> globalTable​(java.lang.String topic,
                                                       Consumed<K,​V> consumed)
        Create a GlobalKTable for the specified topic. Input records with null key will be dropped.

        The resulting GlobalKTable will be materialized in a local KeyValueStore with an internal store name. Note that store name may not be queriable through Interactive Queries. No internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        Note that GlobalKTable always applies "auto.offset.reset" strategy "earliest" regardless of the specified value in StreamsConfig or Consumed.

        Parameters:
        topic - the topic name; cannot be null
        consumed - the instance of Consumed used to define optional parameters
        Returns:
        a GlobalKTable for the specified topic

      • globalTable

        public <K,​V> GlobalKTable<K,​V> globalTable​(java.lang.String topic)
        Create a GlobalKTable for the specified topic. The default key and value deserializers as specified in the config are used. Input records with null key will be dropped.

        The resulting GlobalKTable will be materialized in a local KeyValueStore with an internal store name. Note that store name may not be queriable through Interactive Queries. No internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        Note that GlobalKTable always applies "auto.offset.reset" strategy "earliest" regardless of the specified value in StreamsConfig.

        Parameters:
        topic - the topic name; cannot be null
        Returns:
        a GlobalKTable for the specified topic

      • globalTable

        public <K,​V> GlobalKTable<K,​V> globalTable​(java.lang.String topic,
                                                       Consumed<K,​V> consumed,
                                                       Materialized<K,​V,​KeyValueStore<org.apache.kafka.common.utils.Bytes,​byte[]>> materialized)
        Create a GlobalKTable for the specified topic. Input KeyValue pairs with null key will be dropped.

        The resulting GlobalKTable will be materialized in a local KeyValueStore configured with the provided instance of Materialized. However, no internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        You should only specify serdes in the Consumed instance as these will also be used to overwrite the serdes in Materialized, i.e., 

         
 streamBuilder.globalTable(topic, Consumed.with(Serde.String(), Serde.String(), Materialized.<String, String, KeyValueStore<Bytes, byte[]>as(storeName))
        To query the local KeyValueStore it must be obtained via KafkaStreams#store(...): 
        
 KafkaStreams streams = ...
 ReadOnlyKeyValueStore<String, Long> localStore = streams.store(queryableStoreName, QueryableStoreTypes.<String, Long>keyValueStore());
 String key = "some-key";
 Long valueForKey = localStore.get(key);
        Note that GlobalKTable always applies "auto.offset.reset" strategy "earliest" regardless of the specified value in StreamsConfig or Consumed.
        Parameters:
        topic - the topic name; cannot be null
        consumed - the instance of Consumed used to define optional parameters; can't be null
        materialized - the instance of Materialized used to materialize a state store; cannot be null
        Returns:
        a GlobalKTable for the specified topic

      • globalTable

        public <K,​V> GlobalKTable<K,​V> globalTable​(java.lang.String topic,
                                                       Materialized<K,​V,​KeyValueStore<org.apache.kafka.common.utils.Bytes,​byte[]>> materialized)
        Create a GlobalKTable for the specified topic. Input KeyValue pairs with null key will be dropped.

        The resulting GlobalKTable will be materialized in a local KeyValueStore configured with the provided instance of Materialized. However, no internal changelog topic is created since the original input topic can be used for recovery (cf. methods of KGroupedStream and KGroupedTable that return a KTable).

        To query the local KeyValueStore it must be obtained via KafkaStreams#store(...): 

        
 KafkaStreams streams = ...
 ReadOnlyKeyValueStore<String, Long> localStore = streams.store(queryableStoreName, QueryableStoreTypes.<String, Long>keyValueStore());
 String key = "some-key";
 Long valueForKey = localStore.get(key);
        Note that GlobalKTable always applies "auto.offset.reset" strategy "earliest" regardless of the specified value in StreamsConfig.
        Parameters:
        topic - the topic name; cannot be null
        materialized - the instance of Materialized used to materialize a state store; cannot be null
        Returns:
        a GlobalKTable for the specified topic

      • addGlobalStore

        public StreamsBuilder addGlobalStore​(StoreBuilder storeBuilder,
                                     java.lang.String topic,
                                     Consumed consumed,
                                     ProcessorSupplier stateUpdateSupplier)
        Adds a global StateStore to the topology. The StateStore sources its data from all partitions of the provided input topic. There will be exactly one instance of this StateStore per Kafka Streams instance.

        A SourceNode with the provided sourceName will be added to consume the data arriving from the partitions of the input topic.

        The provided ProcessorSupplier will be used to create an ProcessorNode that will receive all records forwarded from the SourceNode. This ProcessorNode should be used to keep the StateStore up-to-date. The default TimestampExtractor as specified in the config is used.

        Parameters:
        storeBuilder - user defined StoreBuilder; can't be null
        topic - the topic to source the data from
        consumed - the instance of Consumed used to define optional parameters; can't be null
        stateUpdateSupplier - the instance of ProcessorSupplier
        Returns:
        itself
        Throws:
        TopologyException - if the processor of state is already registered

      • build

        public Topology build()
        Returns the Topology that represents the specified processing logic.
        Returns:
        the Topology that represents the specified processing logic