The issue of transiently having duplicates is due to the bad design of the left join itself: in order to ignore the partial joined results such as `A:null`, it lets the producer to potentially send twice to source stream one and rely on all the following conditions to be true in order to pass the test:
1. `receiveMessages` happen to have fetched all the produced results and have committed offsets.
2. streams app happen to have completed sending all result data.
3. consumer used in `receiveMessages` will complete getting all messages in a single poll().
If any of the above is not true, the test fails.
Fixed this test to add a filter right after left join to filter out partial joined results. Minor cleanup on integration test utils.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Ewen Cheslack-Postava
Closes#2485 from guozhangwang/K3896-duplicate-join-results
the toString method prints the topology, but had no tests making sure it works and/or doesn't cause exceptions
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Guozhang Wang
Closes#2444 from dguy/KAFKA-4645
Add a test to ensure a `StreamsException` is thrown when an exception other than `StreamsException` is caught
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2450 from dguy/KAFKA-4647
Most of the exception paths weren't covered. Now they are.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes#2442 from dguy/KAFKA-4642
Kafka brokers have a config called "offsets.topic.replication.factor" that specify the replication factor for the "__consumer_offsets" topic. The problem is that this config isn't being enforced. If an attempt to create the internal topic is made when there are fewer brokers than "offsets.topic.replication.factor", the topic ends up getting created anyway with the current number of live brokers. The current behavior is pretty surprising when you have clients or tooling running as the cluster is getting setup. Even if your cluster ends up being huge, you'll find out much later that __consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement on the internal topic is another way of saying the cluster isn't fully setup yet.
The right behavior should be for "offsets.topic.replication.factor" to be enforced. Topic creation of the internal topic should fail with GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor" requirement is met. This closely resembles the behavior of regular topic creation when the requested replication factor exceeds the current size of the cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2177 from onurkaraman/KAFKA-3959
Makes task assignment more sticky by preferring to assign tasks to clients that had previously had the task as active task. If there are no clients with the task previously active, then search for a standby. Finally falling back to the least loaded client.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2429 from dguy/kafka-4677
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2469 from hachikuji/improve-consumer-logging
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Damian Guy, Guozhang Wang
Closes#2420 from jeyhunkarimov/KAFKA-4613
ewencp ignore this PR if you are already started to work on this ticket.
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2423 from baluchicken/KAFKA-4679
(cherry picked from commit 1434b61d5d)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2414 from cmccabe/KAFKA-4635
- also deprecate ZK config for Streams
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Guozhang Wang
Closes#2459 from mjsax/javaDocImprovements8
When iterating deque for expiring record batches, delay the
invocation of the callback and deallocation until iteration is
complete since callbacks invoked during expiry may send more
records, modifying the deque, resulting in a
ConcurrentModificationException in the iterator.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2449 from rajinisivaram/KAFKA-4557
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Gwen Shapira <cshapi@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2387 from cmccabe/KAFKA-4664
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2455 from hachikuji/KAFKA-4704
In some Places for the loop was used but it can be replaced by the for each.
In One file if else if else was used so I replaced the same with match.
Author: Akash Sethi <akash.sethi@knoldus.in>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2435 from akashsethi24/trunk
Some exception paths not previously covered. Extracted `ensureCopartitioning` into a static class.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2448 from dguy/KAFKA-4644
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2345 from Kamal15/trunk
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Michael G. Noll
Closes#2445 from ijuma/kafka-4578-upgrade-notes-0.10.2
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2403 from mjsax/addStreamsClientCompatibilityTest
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2443 from ijuma/close-create-topics-policy-during-shutdown
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#2406 from ijuma/kafka-4636-per-listener-security-settings
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Xavier Léauté <xavier@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2454 from guozhangwang/KMinor-trace-logging-add-metrics-twice
This was missing from [an earlier PR](https://github.com/apache/kafka/pull/2247) that escaped these symbols in another section of the doc.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2453 from vahidhashemian/doc/escape_lt_gt_in_streams_code
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska, Damian Guy, Guozhang Wang
Closes#2441 from ijuma/streams-kafka-client-drops-security-configs
handle existing classpath with spaces
list each lib dependency individually
Author: Jon Freedman <jon.freedman@zoho.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2433 from jonfreedman/trunk
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2436 from hachikuji/hotfix-offset-deletion
This behaviour was changed in 8b3c6c0, but it caused interceptor
test failures (which rely on callbacks) and since we’re so close to
code freeze, it’s better to be conservative.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2440 from ijuma/kafka-4699-callbacks-invoked-before-future-is-completed
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2390 from cmccabe/KAFKA-4630
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2416 from hachikuji/refactor-partition-lag-cleanup
`bin-kafka-console-producer.sh` should be `bin/kafka-console-producer.sh`.
Author: Will Marshall <wcm214@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2410 from wmarshall484/typo-fix
There is a slight change of behaviour: we now complete the `Future` returned from `send`
before the callbacks are invoked. This seems OK and perhaps a little better as the `Future`
can make progress sooner (as it would typically be blocked on a different thread than the
I/O thread that invokes the callbacks).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2318 from ijuma/kafka-4597-record-metadata-log-append-time
This is the HOTFIX PR for any issues detected with KIP-104 until code freeze. Note: do not merge until close to code freeze.
The name changes reflect feedback received while writing the documentation.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Dan Norwood, Guozhang Wang
Closes#2398 from enothereska/hotfix-streams-metrics
Found a few recently added unit tests did not close KStreamTestDriver after the test itself is closed; this can cause RocksDB virtual function called if the contained topology has some persistent store since they will be initialized but not closed in time.
MINOR fix: found that when closing KStreamTestDriver, we need to first flushing all stores before closing any of them; this is triggered from the `KTableKTableLeftJoin.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions`.
MINOR fix: in CachingXXXStore, the `name` field is actually used as the cache's namespace, not really the store name or its corresponding topic name. Fixed it by renaming it to `cacheName` and use `this.name()` elsewhere which will call the underlying store's name.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ewen Cheslack-Postava, Eno Thereska, Damian Guy
Closes#2432 from guozhangwang/K3502-kstream-builder-test
Author: Manikumar Reddy O <manikumar.reddy@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#1850 from omkreddy/KAFKA-2700-DELETE
Fixes a logic error in the Reassignment process which throws an exception
if you don't rebalance all partitions.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2399 from benstopford/KAFKA-4596
Guozhang Wang
Damian Guy
Onur Karaman
Ewen Cheslack-Postava
Jason Gustafson
Manikumar reddy O
Matthias J. Sax
Jeyhun Karimov
Balint Molnar
Colin P. Mccabe
Rajini Sivaram
Akash Sethi
Kamal C
Ismael Juma
Vahid Hashemian
Jon Freedman
Will Marshall
Gwen Shapira
Xi Hu
Eno Thereska
Ben Stopford