For topics that support timestamp search, if no offset is found for a partition, the partition should still be included in the result with a `null` offset value. This `KafkaConsumer` method currently excludes such partitions from the result.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3460 from vahidhashemian/KAFKA-5534
Before this patch, we would instead return the non-retriable `UNSUPPORTED_FOR_MESSAGE_FORMAT` causing markers to be lost.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3550 from apurvam/KAFKA-5610-handleWriteTxnMarker-should-handle-emigration
Author: Ewen Cheslack-Postava <ewen@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Eno Thereska <eno.thereska@gmail.com>
Closes#3546 from ewencp/fix-rsync
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>
Closes#2301 from mjsax/kafka-3856-topology-builder-API
This PR makes `commitOffsetsSync` method check whether future is completed after client's poll or not.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3489 from umesh9794/KAFKA-5556
I was investigating an exception in this code and found a few
opportunities for making it clearer.
I also added the `out` folder to `.gitignore` as IntelliJ sometimes
uses that as the build folder.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3552 from ijuma/minor-quota-improvements
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3249 from jklukas/docs-no-more-yammer-metrics-in-client
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3551 from ppatierno/admin_config_entry
Clean up includes:
- Switching try-catch-finally blocks to try-with-resources when possible
- Removing some seemingly unnecessary `SuppressWarnings` annotations
- Resolving some Java warnings
- Closing unclosed Closable objects
- Removing unused code
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Balint Molnar <balintmolnar91@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3222 from vahidhashemian/minor/code_cleanup_1706
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3415 from vahidhashemian/doc/make_topic_config_a_section
Instead of redefining the constants.
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3349 from Kamal15/log4j2
This seems to fail a lot in Jenkins although it always passes locally
for me. Removing the assertion to restore Jenkins stability while
we investigate in more detail.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3543 from ijuma/remove-selector-test-assertion
Waiting for the first line of output was added in KAFKA-2527 when JmxMixin was originally added as a heuristic to
determine when the process was ready. We've since determined this is not good enough given JmxTool's limitations
and now include a separate, more reliable check before starting JmxTool. This check is also dangerous since a
consumer that is started before data is available in the topic, it won't output anything to stdout and only logs
errors to a separate log file. This means we may have a long delay between starting the process and starting JMX
monitoring.
Since we have a more reliable check for liveness via JMX now (and in cases that need it, partition assignment
metrics via JMX), we should no longer need to wait for the first line of output.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>
Closes#3447 from ewencp/dont-wait-first-line-console-consumer
When idle connections are closed, ensure that channels with staged
receives are retained in `closingChannels` until all staged receives
are completed. Also ensure that only one staged receive is completed
in each poll, even when channels are closed.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3526 from rajinisivaram/KAFKA-5587
the while loop was too big and need to be closed earlier
to see the fix, ignore whitespace since most of it is indentation
this bug was introduced by commit
5bd06f1d54
Author: Jan Burkhardt <jan.burkhardt@just.social>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3538 from bjrke/trunk
Also add connection id to KafkaChannel exception message
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3529 from rajinisivaram/MINOR-log-connection-id
Currently, the output of `ConsumerPerformance` looks strange. The `header` format as follow:
```
"time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec"
```
while the `body` as follow:
```
println("%s, %d, %.4f, %.4f, %d, %.4f".format(dateFormat.format(endMs), id, totalMBRead,
1000.0 * (mbRead / elapsedMs), messagesRead, ((messagesRead - lastMessagesRead) / elapsedMs) * 1000.0))
```
So we get the follow result:
```
time, data.consumeed.in.MB, MB.sec, data.consumeed.in.nMsg, nMsg.Sec
09:52:00, 0, 1100.3086, 220.0177, 563358, 112649.0702
```
So the `header` and `body` mismatching.
And also, this pr makes the functions more readable.
Author: Xianyang Liu <xianyang.liu@intel.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3417 from ConeyLiu/consumertest
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3523 from guozhangwang/KMinor-group-coordinator-member-failure-info-log4j
Update the code where possible to fix the warnings. The unused
warning introduced in Scala 2.12 is quite handy and provides
a reason to compile with Scala 2.12.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3464 from ijuma/scala-xlint
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2919 from baluchicken/KAFKA-5127
It may lead to wrong metrics and it may break
size-based retention.
Author: Gregor Uhlenheuer <kongo2002@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3521 from kongo2002/KAFKA-5584
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3520 from Kamal15/stream_doc
Previously, the code mistakenly consumed from inputTopic, which
worked, but didn't actually verify that the messages were correctly
copied from inputTopic to outputTopic.
Author: Joel Dice <jdice@mersive.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3522 from dicej/trunk
…s until removed
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3461 from bbejeck/MINOR_make_sure_deprecated_streams_configs_still_usable
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3510 from guozhangwang/KHotfix-streams-quickstart
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3494 from enothereska/minor-quickstart-docs
rename section name and bold font for section names
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3505 from enothereska/minor-less-logs
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3496 from mjsax/KAFKA-5464-follow-up
Also fix quickstart.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3501 from ijuma/introduce-full-version-variable-in-docs
Ensure only one thread dies, not both.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3500 from enothereska/hotfix-queryable-state
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3497 from mjsax/disable-flaky-system-tests
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3493 from dguy/hotfix-test-failure
Operations documentation should guide user to employ `kafka-configs.sh` to add/remove configs for a topic.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3429 from huxihx/KAFKA-5508_Doc_for_altering_topics
Several fixes to state transition logic:
- Kafka streams will now be in ERROR when all threads are DEAD or when global thread stops unexpectedly
- Fixed transition logic in corner cases when thread is already dead or Kafka Streams is already closed
- Fixed incorrect transition diagram in StreamThread
- Unit tests to verify transitions
Also:
- re-enabled throwing an exception when an unexpected state change happens
- fixed a whole bunch of EoS tests that did not start a thread
- added more comments.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3432 from enothereska/KAFKA-5372-state-transitions
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3439 from mjsax/kafka-5464-streamskafkaclient-poll
Added logPrefix for avoiding stream thread name formatting replicated more times
Author: ppatierno <ppatierno@live.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3488 from ppatierno/kafka-5557
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3449 from mjsax/kafka-5167-streams-task-gets-stuck-after-re-balance-due-to-LockException
Vahid Hashemian
Apurva Mehta
Ewen Cheslack-Postava
Matthias J. Sax
umesh chaudhary
jameschien
Ismael Juma
Balint Molnar
Jeff Klukas
Paolo Patierno
Jeff Widman
Ewen Cheslack-Postava
Gwen Shapira
Kamal C
Rajini Sivaram
Jan Burkhardt
Jeremy Custenborder
Xianyang Liu
Guozhang Wang
Gregor Uhlenheuer
Evgeny Veretennikov
Joel Dice
Eno Thereska
Manikumar Reddy O
Bill Bejeck
Damian Guy
huxihx