Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3994 from bbejeck/KAFKA-5985_document_need_to_close_iterators
When logging is disabled and there are state stores the task never transitions from restoring to running. This is because we only ever check if the task has state stores and return false on initialization if it does. The check should be if we have changelog partitions, i.e., we need to restore.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, tedyu <yuzhihong@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3983 from dguy/restore-test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3978 from bbejeck/KAFKA-5932_no_fetch_previous_when_no_old_values_returned
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Closes#3943 from dguy/count-materialized
- compare KAFKA-5958
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3986 from mjsax/kafka-5949-exceptions-user-callbacks-KIP-161-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3948 from mjsax/kafka-4593-illegal-state-exception-in-restore
The `metric.reporters` description in the documentation says to implement the `MetricReporter` class, but the actual class is `MetricsReporter`. [MetricsReporter.java](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/MetricsReporter.java)
The configurations documentation is also inconsistent as some references to classes do not have the full package name while others do.
ijuma
Author: Kevin Lu <kelu@paypal.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3875 from KevinLiLu/trunk
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3979 from mjsax/kafka-5979-kip-120-regression
- catch user exception in user callback (TimestampExtractor, DeserializationHandler, StateRestoreListener) and wrap with StreamsException
Additional cleanup:
- rename globalRestoreListener to userRestoreListener
- remove unnecessary interface -> collapse SourceNodeRecordDeserializer and RecordDeserializer
- removed unused parameter loggingEnabled from ProcessorContext#register
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3939 from mjsax/kafka-5949-exceptions-user-callbacks
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3973 from bbejeck/KAFKA-5958_global_stores_access_state_restore_listener
This is the polished version.
1. The old merge() method in StreamsBuilder has been removed,
2. The merge() method in KStreamBuilder was changed so that it would use the single variable argument
rather than several variable arguments in the KStreamImpl implementation
3. The merge() method in KStream has been declared as final and tests have been added to test correctness.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3916 from ConcurrencyPractitioner/trunk
The new overloads `StreamBuilder.table(String, Materialized)` and `StreamsBuilder.globalTable(String, Materialized)` need to set the serdes from `Materialized` on the internal `Consumed` instance that is created, otherwise the defaults will be used and may result in serialization errors
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3936 from dguy/table-materialized
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ted Yu <yuzhihong@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3913 from mjsax/kafka-5937-exceptions-processor-state-manager
Add `SessionWindowedKStream` and implementation. Deprecate existing `SessionWindow` `aggregate` methods on `KGroupedStream`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3902 from dguy/kafka-5922
In `AssignedTasks` log at debug all task ids that are yet to be initialized.
In `StreamsTask` log at trace when the task is initialized.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3905 from dguy/minor-task-init-logging
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3903 from dguy/deprectate-to-through
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3884 from mjsax/minor-fixed-discoverd-via-exception-handling-investigation
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3892 from dguy/cleanup-state-stores
Add `Materialized` overloads to `WindowedKStream`. Deprecate existing methods on `KGroupedStream`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3889 from dguy/kafka-5921
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Bill Bejeck <bill@confluent.io>
Closes#3893 from mjsax/kafka-5893-reset-integration-test
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ted Yu <yuzhihong@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3841 from mjsax/kafka-5833-interrupts
Add overloads for `table` and `globalTable` that use `Materialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3837 from dguy/kafka-5873
The parameter is already called `mapper` in the KStreamImpl class. I think it was probably named `processor` here because it was copy/pasted from some other signature. This sees trivial enough to not require a jira as per the contribution guidelines.
Author: Andy Chambers <andy.chambers@fundingcircle.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3888 from cddr/fix-kstream-flatMapValues-signature
Remove date formatting from `Segments` and use the `segementId` instead.
Add tests to make sure can load old segments.
Rename old segment dirs to new formatting at load time.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3783 from dguy/kafka-5515
I'm doing this in my spare time, so don't let reviewing this PR take away actual work time. This is just me going over the code with the Intellij analyzer and implementing the most easily implementable fixes.
This PR is focused only on seemingly erronous log statements.
1: A log statement that has 4 arguments supplied but only 3 `{}` statements
2: A log statement that checks is debug is enabled, but then logs on `info` level.
Author: coscale_kdegroot <koen.degroote@coscale.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3886 from KoenDG/loggingErrors
Add overloads of `count`, `reduce`, and `aggregate` that are `Materialized` to `KGroupedStream`.
Refactor common parts between `KGroupedStream` and `WindowedKStream`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3827 from dguy/kafka-5654
This PR utilizes `org.apache.kafka.common.utils.LogContext` for logging in `KafkaStreams`. hachikuji, ijuma please review this and let me know your thoughts.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3727 from umesh9794/KAFKA-5754
The `NextIteratorFunction` in `CompositeReadOnlyWindowStore` was incorrectly using the `timeFrom` as the `timeTo`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3868 from dguy/window-store-range-scan
The contribution is my original work and I license the work to the project under the project's open source licence.
Author: lperry <lperry@simplemachines.com.au>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3843 from leigh-perry/trunk
I have decided to use the following approach to fixing this bug:
1) Since the Window Size in WindowedDeserializer was originally unknown, I have initialized
a field _windowSize_ and created a constructor to allow it to be instantiated
2) The default size for __windowSize__ is _Long.MAX_VALUE_. If that is the case, then the
deserialize method will return an Unlimited Window, or else will return Timed one.
3) Temperature Demo was modified to demonstrate how to use this new constructor, given
that the window size is known.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3745 from ConcurrencyPractitioner/trunk
Add overloads of `count`, `aggregate`, `reduce` using `Materialized` to `KGroupedTable`
deprecate other overloads
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3829 from dguy/kafka-5655
Add `join`, `leftJoin`, `outerJoin` overloads that use `Materialized` to `KTable`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3826 from dguy/kafka-5653
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3828 from bbejeck/MINOR_update_processor_topology_test_driver
Add `SerializedInternal` class and remove getters from `Serialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3825 from dguy/kafka-5817-follow-up
Create `ProducedInternal` and remove getters from `Produced`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3810 from dguy/kafka-5816-follow-up
1. Now instead of just generic `Exception` methods declare more concrete
exceptions throwing or don't declare any throwing at all, if not needed.
2. `SimpleBenchmark.run()` throws `RuntimeException`
3. `SimpleBenchmark.produce()` throws `IllegalArgumentException`
4. Expect `ProcessorStateException` in
`StandbyTaskTest.testUpdateNonPersistentStore()`
/cc enothereska
Author: Evgeny Veretennikov <evg.veretennikov@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3485 from evis/5531-throw-concrete-exceptions
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3816 from dguy/consumed-ctor
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3817 from dguy/printed-ctor-protected
Add overloads of `filter`, `filterNot`, `mapValues` that take `Materialized` as a param to `KTable`. Deprecate overloads using `storeName` and `storeSupplier`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3807 from dguy/ktable-filter-map
Part of KIP-182
- Add `Printed` class and `KStream#print(Printed)`
- deprecate all other `print` and `writeAsText` methods
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3768 from dguy/kafka-5652-printed
Add the `WindowedKStream` interface and implementation of methods that don't require `Materialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3809 from dguy/kgrouped-stream-windowed-by
I removed synchronized keyword from 3 methods.
I ran the change thru streams module where test suite passed.
Author: tedyu <yuzhihong@gmail.com>
Reviewers: Viktor Somogyi <viktorsomogyi@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3777 from tedyu/trunk
1. Sort processor nodes within a sub-topology by its sub-tree size: nodes with largest sizes are source nodes and hence printed earlier.
2. Sort sub-topologies by ids; sort global stores by the source topic names.
3. Open for discussion: start newlines for predecessor and successor.
4. Minor: space between processor nodes and stores / topics; maintain `[]` for the topic names.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ted Yu <yuzhihong@gmail.com>
Closes#3618 from guozhangwang/K5698-topology-description-sorting
add `KTable#groupBy(KeyValueMapper, Serialized)` and deprecate the overload with `Serde` params
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Bill Bejeck <bill@confluent.io>
Closes#3802 from dguy/kip-182-ktable-groupby
Bill Bejeck
Damian Guy
Matthias J. Sax
Kevin Lu
Richard Yu
Andy Chambers
coscale_kdegroot
umesh chaudhary
Rajini Sivaram
lperry
Andrey Dyachkov
Evgeny Veretennikov
Ted Yu
Guozhang Wang