Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#4168 from bbejeck/MINOR_update_streams_produer_timeout_in_system_test
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4128 from mjsax/minor-cleanup
minor fix
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#4110 from ConcurrencyPractitioner/trunk
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
Closes#4136 from guozhangwang/K6100-rocksdb-580-regression
Even though this class is internal, it's widely
used by other projects and it's better to avoid
breaking them until we have a publicly supported
test library.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4138 from ijuma/revert-embedded-zookeeper-rename
- kafka.controller.ZookeeperClient -> kafka.zookeeper.ZooKeeperClient
- kafka.controller.ControllerZkUtils -> kafka.zk.KafkaZkClient
- kafka.controller.ZkData -> kafka.zk.ZkData
- Renamed various fields to match new names and for consistency
- A few clean-ups in ZkData
- Document intent
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#4112 from ijuma/rename-zookeeper-client-and-move-to-zookeper-package
1. After the poll call, re-check if the state has been changed or not; if yes, initialize the tasks again.
2. Minor log4j improvements.
Author: Guozhang Wang <wangguoz@gmail.com>
Author: Damian Guy <damian.guy@gmail.com>
Author: Jason Gustafson <jason@confluent.io>
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ted Yu <yuzhihong@gmail.com>
Closes#4096 from guozhangwang/KHotfix-restore-only
A couple of root causes of this flaky test is fixed:
1. The MockTime was incorrectly used across multiple test methods within the class, as a class rule. Instead we set it on each test case; also remove the scala MockTime dependency.
2. List topics may not contain the deleted topics while their ZK paths are yet to be deleted; so the delete-check-recreate pattern may fail to successfully recreate the topic at all. Change the checking to read from zk path directly instead.
Another minor fix is to remove the misleading wait condition error message as the accumData is always empty.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#4095 from guozhangwang/KMinor-reset-integration-test
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4104 from mjsax/minor-uncaught-exception-handler
Author: Tommy Becker <tobecker@tivo.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4081 from twbecker/KAFKA-6069
long sizeBytes() {
long sizeInBytes = 0;
for (final NamedCache namedCache : caches.values()) {
sizeInBytes += namedCache.sizeInBytes();
}
return sizeInBytes;
}
The summation w.r.t. sizeInBytes may overflow.
Check similar to what is done in size() should be performed.
Author: siva santhalingam <siva.santhalingam@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4041 from shivsantham/kafka-6023
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax<matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
Closes#4074 from dguy/minor-session-window-equals
1. Added missing Javadocs in public interfaces.
2. Added missing upgrade web docs.
3. Minor improvements on exception messages.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Antony Stubbs <antony.stubbs@gmail.com>
Closes#4071 from guozhangwang/KMinor-javadoc-gaps
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4064 from mjsax/minor-add-state-serdes-test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax<matthias@confluent.io>
Closes#4068 from bbejeck/MINOR_fix_java_doc_example_for_1_0_API
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4063 from mjsax/minor-improve-store-parameter-checks
I found this by running the tests while I happened to
have a kafka broker running.
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4065 from tombentley/MINOR-random-port
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4046 from mjsax/kafka-5541-minor-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4051 from mjsax/minor-kip-182-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4048 from mjsax/fix-eos-test-race-condition
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4037 from mjsax/kafka-5541-dont-rethrow-on-suspend-or-close-2
- improve tests to get rid of calls to `sleep` in Python
- fixed some flaky test conditions
- improve debugging
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3542 from mjsax/failing-eos-system-tests
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Xavier Léauté <xavier@confluent.io>, Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
Closes#4031 from guozhangwang/KMinor-assigned-task-log4j
Add a `with(Serde keySerde, Serde valSerde)` to `Materialized` for cases where people don't care about the state store name.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Matthias J. Sax <matthias@confluent.io>
Closes#4009 from dguy/materialized
This change allows for testing custom Processors and Transformers that call `schedule` and `commit` using KStreamTestDriver, by _not_ throwing `UnsupportedOperationException`.
This PR is my original work.
Author: Mats Julian Olsen <mats@plysjbyen.net>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3992 from mewwts/allow-schedule-and-commit
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3819 from guozhangwang/KMinor-rocksDB-573
Stores where logging is disabled where never consumed as the partitions were paused, but never resumed.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4002 from dguy/restore
- fixes examples with regard to new API
- fixes `Topology#addGlobalStore` parameters
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#4003 from mjsax/minor-deprecated
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4006 from mjsax/minor-javadoc-improvments-for-stores
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3984 from dguy/quickstart-update
package name: org.apache.kafka.streams.state.internals
Minor change to approximateNumEntries() method in CompositeReadOnlyKeyValueStore class.
long total = 0;
for (ReadOnlyKeyValueStore<K, V> store : stores) {
total += store.approximateNumEntries();
}
return total < 0 ? Long.MAX_VALUE : total;
The check for negative value seems to account for wrapping. However, wrapping can happen within the for loop. So the check should be performed inside the loop.
Author: siva santhalingam <ssanthalingam@netskope.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3988 from shivsantham/trunk
- add some missing annotations for deprecated methods
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4005 from mjsax/minor-fix-javadoc-warnings
Author: Matthias J. Sax <matthias@confluent.io>
Author: Bharat Viswanadham <bharatv@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3970 from mjsax/kafka-5225-streams-resetter-properties
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3994 from bbejeck/KAFKA-5985_document_need_to_close_iterators
When logging is disabled and there are state stores the task never transitions from restoring to running. This is because we only ever check if the task has state stores and return false on initialization if it does. The check should be if we have changelog partitions, i.e., we need to restore.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, tedyu <yuzhihong@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3983 from dguy/restore-test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3978 from bbejeck/KAFKA-5932_no_fetch_previous_when_no_old_values_returned
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Closes#3943 from dguy/count-materialized
- compare KAFKA-5958
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3986 from mjsax/kafka-5949-exceptions-user-callbacks-KIP-161-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3948 from mjsax/kafka-4593-illegal-state-exception-in-restore
The `metric.reporters` description in the documentation says to implement the `MetricReporter` class, but the actual class is `MetricsReporter`. [MetricsReporter.java](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/MetricsReporter.java)
The configurations documentation is also inconsistent as some references to classes do not have the full package name while others do.
ijuma
Author: Kevin Lu <kelu@paypal.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3875 from KevinLiLu/trunk
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3979 from mjsax/kafka-5979-kip-120-regression
- catch user exception in user callback (TimestampExtractor, DeserializationHandler, StateRestoreListener) and wrap with StreamsException
Additional cleanup:
- rename globalRestoreListener to userRestoreListener
- remove unnecessary interface -> collapse SourceNodeRecordDeserializer and RecordDeserializer
- removed unused parameter loggingEnabled from ProcessorContext#register
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3939 from mjsax/kafka-5949-exceptions-user-callbacks
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3973 from bbejeck/KAFKA-5958_global_stores_access_state_restore_listener
This is the polished version.
1. The old merge() method in StreamsBuilder has been removed,
2. The merge() method in KStreamBuilder was changed so that it would use the single variable argument
rather than several variable arguments in the KStreamImpl implementation
3. The merge() method in KStream has been declared as final and tests have been added to test correctness.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3916 from ConcurrencyPractitioner/trunk
The new overloads `StreamBuilder.table(String, Materialized)` and `StreamsBuilder.globalTable(String, Materialized)` need to set the serdes from `Materialized` on the internal `Consumed` instance that is created, otherwise the defaults will be used and may result in serialization errors
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3936 from dguy/table-materialized
Bill Bejeck
Matthias J. Sax
Richard Yu
Guozhang Wang
Ismael Juma
Tommy Becker
siva santhalingam
Damian Guy
Tom Bentley
Mats Julian Olsen
siva santhalingam
Kevin Lu