Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3565 from lindong28/KAFKA-5627
1. Make the WordCountDemo application to not stop automatically but via "ctrl-C".
2. Update the quickstart html file to let users type input messages one-by-one, and observe added output in an interactive manner.
3. Some minor fixes on the parent documentation page pointing to streams sub-pages, added a new recommended Scala version number.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3515 from guozhangwang/KMinor-interactive-quickstart
- extract InternalTopologyBuilder from TopologyBuilder
- deprecate all "leaking" methods from public TopologyBuilder API
- changed TopologyDescription and all nested classed into interfaces
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3536 from mjsax/kafka-3856-extract-internal-topology-builder
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Onur Karaman <okaraman@linkedin.com>
Closes#2929 from lindong28/KAFKA-4763
Content and assets for the updated Streams API landing page
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3540 from derrickdoo/streams-landing-page
Author: Stephane Roset <stephane@roset.me>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3442 from rosets/KAFKA-5512
Allow users to specify default topic configs for streams internal topics by supplying properties from `TopicConfig` with a prefix.
Supplied defaults are used when creating the internal topics. They are overridden by the configs supplied along with the `InternalTopicConfig`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3459 from dguy/kafka-3741
For a compacted topic with preallocate enabled, during log cleaning, LogCleaner.cleanSegments does not have to pre-allocate the underlying file size since we only want to store the cleaned data in the file.
It's believed that this fix should also solve KAFKA-5582.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3525 from huxihx/log_compact_test
For topics that support timestamp search, if no offset is found for a partition, the partition should still be included in the result with a `null` offset value. This `KafkaConsumer` method currently excludes such partitions from the result.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3460 from vahidhashemian/KAFKA-5534
Before this patch, we would instead return the non-retriable `UNSUPPORTED_FOR_MESSAGE_FORMAT` causing markers to be lost.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3550 from apurvam/KAFKA-5610-handleWriteTxnMarker-should-handle-emigration
Author: Ewen Cheslack-Postava <ewen@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Eno Thereska <eno.thereska@gmail.com>
Closes#3546 from ewencp/fix-rsync
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>
Closes#2301 from mjsax/kafka-3856-topology-builder-API
This PR makes `commitOffsetsSync` method check whether future is completed after client's poll or not.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3489 from umesh9794/KAFKA-5556
I was investigating an exception in this code and found a few
opportunities for making it clearer.
I also added the `out` folder to `.gitignore` as IntelliJ sometimes
uses that as the build folder.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3552 from ijuma/minor-quota-improvements
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3249 from jklukas/docs-no-more-yammer-metrics-in-client
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3551 from ppatierno/admin_config_entry
Clean up includes:
- Switching try-catch-finally blocks to try-with-resources when possible
- Removing some seemingly unnecessary `SuppressWarnings` annotations
- Resolving some Java warnings
- Closing unclosed Closable objects
- Removing unused code
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Balint Molnar <balintmolnar91@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3222 from vahidhashemian/minor/code_cleanup_1706
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3415 from vahidhashemian/doc/make_topic_config_a_section
Instead of redefining the constants.
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3349 from Kamal15/log4j2
This seems to fail a lot in Jenkins although it always passes locally
for me. Removing the assertion to restore Jenkins stability while
we investigate in more detail.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3543 from ijuma/remove-selector-test-assertion
Waiting for the first line of output was added in KAFKA-2527 when JmxMixin was originally added as a heuristic to
determine when the process was ready. We've since determined this is not good enough given JmxTool's limitations
and now include a separate, more reliable check before starting JmxTool. This check is also dangerous since a
consumer that is started before data is available in the topic, it won't output anything to stdout and only logs
errors to a separate log file. This means we may have a long delay between starting the process and starting JMX
monitoring.
Since we have a more reliable check for liveness via JMX now (and in cases that need it, partition assignment
metrics via JMX), we should no longer need to wait for the first line of output.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>
Closes#3447 from ewencp/dont-wait-first-line-console-consumer
When idle connections are closed, ensure that channels with staged
receives are retained in `closingChannels` until all staged receives
are completed. Also ensure that only one staged receive is completed
in each poll, even when channels are closed.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3526 from rajinisivaram/KAFKA-5587
the while loop was too big and need to be closed earlier
to see the fix, ignore whitespace since most of it is indentation
this bug was introduced by commit
5bd06f1d54
Author: Jan Burkhardt <jan.burkhardt@just.social>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3538 from bjrke/trunk
Also add connection id to KafkaChannel exception message
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3529 from rajinisivaram/MINOR-log-connection-id
Currently, the output of `ConsumerPerformance` looks strange. The `header` format as follow:
```
"time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec"
```
while the `body` as follow:
```
println("%s, %d, %.4f, %.4f, %d, %.4f".format(dateFormat.format(endMs), id, totalMBRead,
1000.0 * (mbRead / elapsedMs), messagesRead, ((messagesRead - lastMessagesRead) / elapsedMs) * 1000.0))
```
So we get the follow result:
```
time, data.consumeed.in.MB, MB.sec, data.consumeed.in.nMsg, nMsg.Sec
09:52:00, 0, 1100.3086, 220.0177, 563358, 112649.0702
```
So the `header` and `body` mismatching.
And also, this pr makes the functions more readable.
Author: Xianyang Liu <xianyang.liu@intel.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3417 from ConeyLiu/consumertest
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3523 from guozhangwang/KMinor-group-coordinator-member-failure-info-log4j
Update the code where possible to fix the warnings. The unused
warning introduced in Scala 2.12 is quite handy and provides
a reason to compile with Scala 2.12.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3464 from ijuma/scala-xlint
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2919 from baluchicken/KAFKA-5127
It may lead to wrong metrics and it may break
size-based retention.
Author: Gregor Uhlenheuer <kongo2002@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3521 from kongo2002/KAFKA-5584
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3520 from Kamal15/stream_doc
Previously, the code mistakenly consumed from inputTopic, which
worked, but didn't actually verify that the messages were correctly
copied from inputTopic to outputTopic.
Author: Joel Dice <jdice@mersive.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3522 from dicej/trunk
Dong Lin
Guozhang Wang
Matthias J. Sax
Ismael Juma
Ewen Cheslack-Postava
Derrick Or
ppatierno
Hooman Broujerdi
Stephane Roset
Damian Guy
Vahid Hashemian
Stephane Maarek
huxihx
Colin P. Mccabe
Apurva Mehta
Ewen Cheslack-Postava
umesh chaudhary
jameschien
Balint Molnar
Jeff Klukas
Jeff Widman
Gwen Shapira
Kamal C
Rajini Sivaram
Jan Burkhardt
Jeremy Custenborder
Xianyang Liu
Gregor Uhlenheuer
Evgeny Veretennikov
Joel Dice