Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Apurva Mehta <apurva@confluent.io>
Closes#3251 from guozhangwang/KMinor-remove-ignored-streams-integration-test
Expose the streams commit interval config for different testing scenarios.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3244 from bbejeck/MINOR_add_commit_config_benchmark
This resolved the issue with Kafka Streams skipped records sensor reporting wrong values.
Jira ticket: https://issues.apache.org/jira/browse/KAFKA-5368
The contribution is my original work and I license the work to the project under the project's open source license.
Author: Hamidreza Afzali <hrafzali@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3206 from hrafzali/KAFKA-5368_skipped-records-sensor-bug
Author: Dale Peakall <dale@peakall.net>
Reviewers: Michael André Pearce <michael.andre.pearce@me.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3199 from subnova/streams-extendeddeserializer
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3195 from rajinisivaram/KAFKA-5345
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
subscribed stream may not be detected by all followers until
onJoinComplete returns.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3157 from bbejeck/KAFKA-5226_null_pointer_source_node_deserialize
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3172 from guozhangwang/K5350-compatibility-annotations
- Added a boolean `allow_auto_topic_creation` to MetadataRequest and
bumped the protocol version to V4.
- When connecting to brokers older than 0.11.0.0, the `allow_auto_topic_creation`
field won't be considered, so we send a metadata request for all topics
to keep the behavior consistent.
- Set `allow_auto_topic_creation` to false in the new AdminClient and
StreamsKafkaClient (which exists for the purpose of creating topics
manually); set it to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.
- Add `allowAutoTopicCreation` field to `Metadata`, which is used by
`DefaultMetadataUpdater`. This is not strictly needed for the new
`AdminClient`, but it avoids surprises if it ever adds a topic to `Metadata`
via `setTopics` or `addTopic`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3098 from ijuma/kafka-5291-admin-client-no-auto-topic-creation
Adjust "importance level" and add explanation to the docs.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#2855 from mjsax/minor-improve-streams-config-parameters
Handle` rocksdb.config.setter` being set as a class name or class
instance.
Author: Tommy Becker <tobecker@tivo.com>
Author: Tommy Becker <twbecker@gmail.com>
Reviewers: Matthias J. Sax, Damian Guy, Guozhang Wang
Closes#3155 from twbecker/KAFKA-5334
KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error
Change all the OptionParser constructor in Kafka into "new OptionParser(false)"
Author: xinlihua <xin.lihua1@zte.com.cn>
Author: unknown <00067310@A23338408.zte.intra>
Author: auroraxlh <xin.lihua1@zte.com.cn>
Author: xin <xin.lihua1@zte.com.cn>
Reviewers: Damian Guy, Guozhang Wang
Closes#2349 from auroraxlh/fix_OptionParser_bug
There is a Misspell in Annotations of ResetIntegrationTest.
Author: hejiefang <he.jiefang@zte.com.cn>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#3159 from hejiefang/KAFKA-5338
dguy , mjsax Please review the PR and let me know your comments.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Bill Bejeck, Matthias J. Sax, Guozhang Wang
Closes#3099 from umesh9794/mylocal
- introduces a new thread state DEAD
- ignores DEAD threads when querying
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3140 from mjsax/kafka-5309-stores-not-queryable
A follow up RP to fix [issue](2cd0b87bc8 (commitcomment-22200864))
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang
Closes#3109 from jeyhunkarimov/KAFKA-4144-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang
Closes#3104 from mjsax/minor-improve-embedded-kafka-cluster
- add broker compatibility system tests
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
Summary:
- add `reconnect.backoff.max.ms` common client configuration parameter
- if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an exponential backoff policy
- apply +/- 20% random jitter to smooth cluster reconnects
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1523 from dpkp/exp_backoff
Implements range scan for keys in windowed and session stores
Modifies caching session and windowed stores to use segmented cache keys.
Cache keys are internally prefixed with their segment id to ensure key ordering in the cache matches the ordering in the underlying store for keys spread across multiple segments.
This should also result in fewer cache keys getting scanned for queries spanning only some segments.
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#3027 from xvrl/windowstore-range-scan
Don't produce messages on a separate thread continuosly. Just produce one of each value and stop.
Close the producer once finished.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3080 from dguy/qs-test
I remove `KeyValuePrinter` and `KStreamForeach` two class, then implements them by `KStreamPeek`.
So, now `KStreamPeek` can do `KeyValuePrinter` and `KStreamForeach` job.
Author: jameschien <jameschien@staff.ruten.com.tw>
Author: JamesChien <jedichien@users.noreply.github.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2955 from jedichien/trunk
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Apurva Metha, Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2945 from mjsax/kafka-4923-add-eos-to-streams
Ensure checkpointable offsets for GlobalKTables are always written on close.
Author: Tommy Becker <tobecker@tivo.com>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3054 from twbecker/KAFKA-5241
CachingSessionStore wasn't properly using the default keySerde if no Serde was supplied. I saw the below error in the logs for one of my test cases.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2963 from KyleWinkelman/CachingSessionStore-fix-keySerde-use
RocksDBSessionStore wasn't properly using the default aggSerde if no Serde was supplied.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2971 from KyleWinkelman/RocksDBSessionStore-fix-aggSerde-use
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Damian Guy, Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2466 from jeyhunkarimov/KAFKA-4144
…it is accessed from multiple threads
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3000 from cmccabe/KAFKA-5198
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2931 from mjsax/kafka-5140-flaky-reset-integration-test
a KStream.to() sink is also a topic
... so the KStreamTestDriver to fetch it when required
Author: Wim Van Leuven <wim.vanleuven@bigboards.io>
Author: Wim Van Leuven <wim.vanleuven@highestpoint.biz>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2716 from wimvanleuven/KAFKA-4927
This fix needs to be backported to 0.10.2 as well.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy, Ismael Juma, Guozhang Wang
Closes#2982 from enothereska/KAFKA-5174-1-core
Change fetchPrevious to use findSessions with the proper key and timestamps rather than using fetch.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2972 from KyleWinkelman/CachingSessionStore-fetchPrevious
The code was correct since the method is only called from
one thread, but the change is worthwhile anyway.
Author: Amit Daga <adaga@adobe.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2966 from amitdaga/findbugs-streams-multithread
Add new broker config, `group.initial.rebalance.delay.ms`, with a default of 3 seconds.
When a consumer creates a new group, set the group's state to InitialRebalance and delay the rebalance until `min(group.initial.rebalance.delay.ms, rebalanceTimeout)`. As other members join the group further delay the rebalance by `min(group.initial.rebalance.delay.ms, remainingRebalanceTimeout)`. Once `rebalanceTimeout` is hit or no new members join the group within the delay, complete the rebalance.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang
Closes#2758 from dguy/kafka-4925
…dSource
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2783 from bbejeck/HOTFIX_potentially_hanging_test_in_RegexSourceIntegrationTest
This is the implementation of KIP-114: KTable state stores and improved semantics:
- Allow for decoupling between querying and materialisation
- consistent APIs, overloads with queryableName and without
- depreciated several KTable calls
- new unit and integration tests
In this implementation, state stores are materialized if the user desires them to be queryable. In subsequent versions we can offer a second option, to have a view-like state store. The tradeoff then would be between storage space (materialize) and re-computation (view). That tradeoff can be exploited by later query optimizers.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Guozhang Wang
Closes#2832 from enothereska/KAFKA-5045-ktable
The descendingSubsequence is a misnomer. The linked list is actually arranged so that the lowest timestamp is first and larger timestamps are added to the end, therefore renamed to ascendingSubsequence.
The minElem variable was also misnamed. It's actually the current maximum element as it's taken from the end of the list.
Added comment to get() to make it clear it's returning the lowest timestamp.
Author: mihbor <mihbor@users.noreply.github.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2948 from mihbor/patch-4
- addressing open Github comments from #2773
- test clean-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#2854 from mjsax/kafka-4986-producer-per-task-follow-up
1. Added a flag to indicate if it is restoring or not in the LRU Store; since we only have a restore callback we have to set it each time applying the change.
2. Fixed the corresponding unit test, plus some minor cleaning up.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2908 from guozhangwang/K4379-remove-listener
- mainly moving methods
- also improved logging
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2917 from mjsax/kafka-5111-code-cleanup-follow-up
* Producer and Consumer `close` calls were not handled via `try-with-resources`
* `cleanRun` unused field removed
* Refactored handling of Consumer configuration in `IntegrationTestUtils` to ensure auto-committing of offsets and starting from `earliest`
* As a result reverted https://github.com/apache/kafka/pull/2921 since it's redundant now
Author: Armin Braun <me@obrown.io>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2920 from original-brownbear/cleanup-it-utils-closing
Fixes `org.apache.kafka.streams.integration.utils.IntegrationTestUtils#readKeyValues` potentially starting to `poll` for stream output after the stream finished sending the test data and hence missing it when working with `latest` offsets.
Author: Armin Braun <me@obrown.io>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2921 from original-brownbear/KAFKA-5124
Refactors Task with proper interface methods `init()`, `resume()`, `commit()`, `suspend()`, and `close()`. All other methods for task handling are internal now. This allows to simplify `StreamThread` code, avoid code duplication and allows for easier reasoning of control flow.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2895 from mjsax/kafka-5111-cleanup-task-code
Guozhang Wang
Bill Bejeck
Hamidreza Afzali
Dale Peakall
Rajini Sivaram
Colin P. Mccabe
Matthias J. Sax
Ismael Juma
Tommy Becker
xinlihua
hejiefang
Umesh Chaudhary
Jeyhun Karimov
Dana Powers
Xavier Léauté
Damian Guy
James Chien
Kyle Winkelman
Wim Van Leuven
Eno Thereska
Amit Daga
Bill Bejeck
Davor Poldrugo
Michal Borowiecki
Armin Braun