diff --git a/docs/streams/developer-guide.html b/docs/streams/developer-guide.html index ab5a823f355..3368757bdeb 100644 --- a/docs/streams/developer-guide.html +++ b/docs/streams/developer-guide.html @@ -2904,6 +2904,16 @@ Note that in the WordCountProcessor implementation, users need to r ); +

+ To retrieve information about the local running threads, you can use the localThreadsMetadata() method after you start the application. +

+ + 
+    // For instance, use this method to print/monitor the partitions assigned to each local tasks.
+    Set<ThreadMetadata> threads = streams.localThreadsMetadata();
+    ...
+
+

To stop the application instance call the close() method:

diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index 96c5941d1b2..c4ae54f5f98 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -49,7 +49,7 @@

With 1.0 a major API refactoring was accomplished and the new API is cleaner and easier to use. - This change includes the five main classes KafkaStreams, KStreamBuilder, + This change includes the five main classes KafkaStreams, KStreamBuilder, KStream, KTable, and TopologyBuilder (and some more others). All changes are fully backward compatible as old API is only deprecated but not removed. We recommend to move to the new API as soon as you can. @@ -59,7 +59,7 @@

The two main classes to specify a topology via the DSL (KStreamBuilder) or the Processor API (TopologyBuilder) were deprecated and replaced by - StreamsBuilder and Topology (both new classes are located in + StreamsBuilder and Topology (both new classes are located in package org.apache.kafka.streams). Note, that StreamsBuilder does not extend Topology, i.e., the class hierarchy is different now. @@ -74,7 +74,7 @@

- Changing how a topology is specified also affects KafkaStreams constructors, + Changing how a topology is specified also affects KafkaStreams constructors, that now only accept a Topology. Using the DSL builder class StreamsBuilder one can get the constructed Topology via StreamsBuilder#build(). @@ -86,33 +86,61 @@

- With the introduction of KIP-182 - you should no longer pass in Serde to KStream#print operations. - If you can't rely on using toString to print your keys an values, you should instead you provide a custom KeyValueMapper via the Printed#withKeyValueMapper call. + New methods in KafkaStreams:

- +
    +
  • retrieve the current runtime information about the local threads via #localThreadsMetadata()
    • +
+

+ Deprecated methods in KafkaStreams: +

+
    +
  • toString()
    • +
  • toString(final String indent)
    • +

- Windowed aggregations have moved from KGroupedStream to WindowedKStream. - You can now perform a windowed aggregation by, for example, using KGroupedStream#windowedBy(Windows)#reduce(Reducer). - Note: the previous aggregate functions on KGroupedStream still work, but have been deprecated. + Previously the above methods were used to return static and runtime information. + They have been deprecated in favor of using the new classes/methods #localThreadsMetadata() / ThreadMetadata (returning runtime information) and + TopologyDescription / Topology#describe() (returning static information).

- The Processor API was extended to allow users to schedule punctuate functions either based on data-driven stream time or wall-clock time. - As a result, the original ProcessorContext#schedule is deprecated with a new overloaded function that accepts a user customizable Punctuator callback interface, which triggers its punctuate API method periodically based on the PunctuationType. - The PunctuationType determines what notion of time is used for the punctuation scheduling: either stream time or wall-clock time (by default, stream time is configured to represent event time via TimestampExtractor). - In addition, the punctuate function inside Processor is also deprecated. + More deprecated methods in KafkaStreams:

+
    +
  • With the introduction of KIP-182 + you should no longer pass in Serde to KStream#print operations. + If you can't rely on using toString to print your keys an values, you should instead you provide a custom KeyValueMapper via the Printed#withKeyValueMapper call. +
    • +
  • + Windowed aggregations have moved from KGroupedStream to WindowedKStream. + You can now perform a windowed aggregation by, for example, using KGroupedStream#windowedBy(Windows)#reduce(Reducer). + Note: the previous aggregate functions on KGroupedStream still work, but have been deprecated. +
    • +

- Before this, users could only schedule based on stream time (i.e. PunctuationType.STREAM_TIME) and hence the punctuate function was data-driven only because stream time is determined (and advanced forward) by the timestamps derived from the input data. - If there is no data arriving at the processor, the stream time would not advance and hence punctuation will not be triggered. - On the other hand, When wall-clock time (i.e. PunctuationType.WALL_CLOCK_TIME) is used, punctuate will be triggered purely based on wall-clock time. - So for example if the Punctuator function is scheduled based on PunctuationType.WALL_CLOCK_TIME, if these 60 records were processed within 20 seconds, - punctuate would be called 2 times (one time every 10 seconds); - if these 60 records were processed within 5 seconds, then no punctuate would be called at all. - Users can schedule multiple Punctuator callbacks with different PunctuationTypes within the same processor by simply calling ProcessorContext#schedule multiple times inside processor's init() method. + Modified methods in Processor:

+
    +
  • +

    + The Processor API was extended to allow users to schedule punctuate functions either based on data-driven stream time or wall-clock time. + As a result, the original ProcessorContext#schedule is deprecated with a new overloaded function that accepts a user customizable Punctuator callback interface, which triggers its punctuate API method periodically based on the PunctuationType. + The PunctuationType determines what notion of time is used for the punctuation scheduling: either stream time or wall-clock time (by default, stream time is configured to represent event time via TimestampExtractor). + In addition, the punctuate function inside Processor is also deprecated. +

    +

    + Before this, users could only schedule based on stream time (i.e. PunctuationType.STREAM_TIME) and hence the punctuate function was data-driven only because stream time is determined (and advanced forward) by the timestamps derived from the input data. + If there is no data arriving at the processor, the stream time would not advance and hence punctuation will not be triggered. + On the other hand, When wall-clock time (i.e. PunctuationType.WALL_CLOCK_TIME) is used, punctuate will be triggered purely based on wall-clock time. + So for example if the Punctuator function is scheduled based on PunctuationType.WALL_CLOCK_TIME, if these 60 records were processed within 20 seconds, + punctuate would be called 2 times (one time every 10 seconds); + if these 60 records were processed within 5 seconds, then no punctuate would be called at all. + Users can schedule multiple Punctuator callbacks with different PunctuationTypes within the same processor by simply calling ProcessorContext#schedule multiple times inside processor's init() method. +

    +
    • +

If you are monitoring on task level or processor-node / state store level Streams metrics, please note that the metrics sensor name and hierarchy was changed: