guozhangwang enothereska mjsax miguno please take a look. A few things that need to be clarified
1. I've added StreamsConfig.USER_ENDPOINT_CONFIG, but should we have separate configs for host and port or is this one config ok?
2. `HostState` in the KIP has a byte[] field - not sure why and what it would be populated with
3. I've changed the API to return `List<KafkaStreamsInstance>` as opposed to `Map<HostInfo, Set<TaskMetadata>>` as i find this far more intuitive to work with.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Michael G. Noll, Eno Thereska, Guozhang Wang
Closes#1576 from dguy/kafka-3914v2
Added non null checks to parameters supplied via the DSL and `TopologyBuilder`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Edward Ribeiro <edward.ribeiro@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1711 from dguy/kafka-3936
kafka-run-class.sh now runs under Cygwin; paths and classpath are set up properly.
**WARNING:** The script was not tested on a Linux machine, only under Cygwin. Prior to merge it into trunk, if accepted, please run a quick test to ensure nothing broke. From my own code review, there should not be any problem, but we can never be too sure.
I do not have the environment to test it under Linux at this moment.
Author: deragon <32nx9812masakjds>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1487 from deragon/trunk
It previously hardcoded it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1613 from ijuma/kafka-3954-consumer-internal-topics-from-broker
Also include a few minor clean-ups.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1623 from ijuma/fix-zk-inconsistent-security-check
ijuma
Author: dan norwood <norwood@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1713 from norwood/add-security-protocol-option-for-fetch
added new consumer metrics section
refactored common metrics into new section
updated TOC
Author: Kaufman Ng <kaufman@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1361 from coughman/KAFKA-3479-consumer-metrics-doc
ewencp I went down the list of connect configs and it looks like only the internal converter configs are mismarked. It looks like the `cluster` config that is present in the current docs is already gone. The only other values I can see arguing to change importance on are the ssl configs (marked high) but they are consistent with the producer/consumer config docs so that's at least consistent. Everything else marked high looks either mandatory or requires consideration in a production deployment to me.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1653 from cotedm/KAFKA-2932
ijuma said that it would make sense to split out this work from KAFKA-3234, since KAFKA-3234 had both a mechanical change (generating docs) as well as a change requiring discussion (deprecating/renaming config options).
jjkoshy, I hope you don't mind that I took over this work. It's been 3 months since the last activity on KAFKA-3234, so I thought it would be okay to take over.
This work is essentially is the first 5-6 commits from Joel's https://github.com/apache/kafka/pull/907. However, since I'm not very experienced with git, I didn't do a direct merge/rebase, but instead largely hand-merged it. I did some minor cleanup. All credit goes to Joel, all blame goes to me. :)
For reference, I attached the auto-generated configuration.html file (as a PDF, because github won't let me attache html).
[configuration.pdf](https://github.com/apache/kafka/files/323901/configuration.pdf)
This is my first time writing Scala, so let me know if there are any changes needed.
I don't know who is the right person to review this. ijuma, can you help me redirect this to the appropriate person? Thanks.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1527 from wushujames/generate_topic_docs
I affirm that the contribution is my original work and that I license the work to the project under the project's open source license.
This cleans up misbehaviour that was introduce while fixing KAFKA-3817. It is impossible for a non-count aggregate to be build, when the addition happens before the removal. IMHO making sure that these details are correct is very important.
This PR has local test errors. It somehow fails the ResetIntegrationTest. It doesn't quite appear to me why but it looks like this PR breaks it, especially because the error appears with the ordering of the events. Still I am unable to find where I could have broken it. Maybe not seems to fail on trunk aswell.
Author: jfilipiak <Jan.Filipiak@trivago.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1705 from Kaiserchen/KAFKA-3817-preserve-order-for-aggreagators
1. The IllegalStateException is actually thrown from testCloseWithZeroTimeoutFromSenderThread() due to a bug. We call producer.close() in the callback. Once the first callback is called, producing records in the callback will hit the IllegalStateException. This only pollutes the output, but doesn't fail the test. I fixed this by only calling producer.send() in the first callback.
2. It's not clear which test throws TimeoutException and it's not reproducible locally. One thing is that the error message in TimeoutException is mis-leading since the timeout is not necessarily due to metadata. Improved this by making the error message in TimeoutException clearer.
3. It's not clear what actually failed testSendNonCompressedMessageWithCreateTime(). One thing I found is that since we set the linger time to MAX_LONG and are sending small messages, those produced messages won't be drained until we call producer.close(10000L, TimeUnit.MILLISECONDS). Normally, 10 secs should be enough for the records to be sent. My only hypothesis is that since SSL is more expensive, occasionally, 10 secs is still not enough. So, I bumped up the timeout from 10 secs to 20 secs.
Author: Jun Rao <junrao@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1703 from junrao/kafka-3875
This is a regression caused by 0bb1d3ae.
After that commit, Streams no longer has a direct dependency on slf4j-log4j12, but zkclient
has a dependency on an older version of slf4j-log4j12, so we get a transitive dependency on
the older version.
The fix is to simply exclude the undesired dependencies from the zkclient dependency.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1704 from ijuma/kafka-4018-streams-duplicate-slf4j-log4j
Rename StateStoreProvider.getStores(...) to StateStoreProvider.stores(...) as this is consistent with the naming of other 'getters' in the public API.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1699 from dguy/minor-method-rename
We are not joining in a window here.
Author: Jendrik Poloczek <jendrik.poloczek@hivestreaming.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1692 from jpzk/trunk
The `acks` config that is provided to console producer with `request-required-acks` comes with `all`, `-1`, `0`, `1` as valid options (`all` and `-1` being interchangeable). Currently, the console producer expects an integer for this input and that makes `all` to become an invalid input. This PR fixes this issue by changing the input type to String.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Grant Henke <granthenke@gmail.com, Ismael Juma <ismael@juma.me.uk>
Closes#1618 from vahidhashemian/KAFKA-3945
By default Kafka is configured to allow ssl communication without hostname verification. This docs has been amended to include instructions on how to set that up in the event clients would like to take a more conservative approach.
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1384 from rnpridgeon/KAFKA-3667
The latter has been deprecated.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1690 from ijuma/rocks-db-dispose-methods-deprecated
The StreamTask is owned by a specific thread, so it doesn't seem necessary to synchronized the processing of the records as discussed with guozhangwang on the dev mailing list
Author: PierreCoquentin <pierre.coquentin@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1688 from PierreCoquentin/trunk
Add prefixes for consumer and producer configs to StreamsConfig, but be backward compatible.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes#1649 from dguy/kafka-3929
The KafkaStreamsTest can occasionally hang if the test doesn't run fast enough. This is due to there being no brokers available on the broker.urls provided to the StreamsConfig. The KafkaConsumer does a poll and blocks causing the test to never complete.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1693 from dguy/kafka-streams-test
(cherry picked from commit ce34614a43)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
moved streams application reset tool from tools to core
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1685 from mjsax/moveResetTool
(cherry picked from commit f2405a73ea2dd4b636832b7f8729fb06a04de1d5)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
When resetting the first dirty offset to the log start offset, we currently log an ERROR which makes users think the log cleaner has a problem and maybe has exited. We should log a WARN instead to avoid alarming the users.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Gwen Shapira
Closes#1691 from cotedm/minorlogcleanerlogging
- Removed an unnecessary annotation
- Parameterized a couple of raw types
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1675 from mimaison/warnings
…nly be 0, 1, or -1
Rephrased the documentation string for the Produce request
Updated the acks configuration docs to state that -1, 0, and 1 are the only allowed values
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Gwen Shapira
Closes#1680 from mimaison/KAFKA-3946
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1673 from mjsax/hotfix
(cherry picked from commit ad1dab9c3d3ae14746ee5d94434ef98ef4889023)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
InvalidRecordException is not part of the public API so go back
to the behaviour before ff557f02 for now.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1676 from hachikuji/raise-kafka-exception-on-invalid-crc
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>
Closes#1656 from hachikuji/KAFKA-3977
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1670 from ewencp/kafka-3851-automate-release-notes
…ceful shutdown
The patch is pretty simple and the justification is explained in https://issues.apache.org/jira/browse/KAFKA-3924
I could not find Andrew Olson, who seems to be the contributor of this part of the code, in github so I am not sure whom I should ask to review the patch.
the contribution is my original work and that i license the work to the project under the project's open source license.
Author: Maysam Yabandeh <myabandeh@dropbox.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Andrew Olson <andrew.olson@cerner.com>, Jun Rao <junrao@gmail.com>
Closes#1634 from maysamyabandeh/KAFKA-3924
Also:
* Introduce a blocking variant to be used by `FileMessageSet.append`
* Add tests
* Minor clean-ups
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1669 from ijuma/kafka-3996-byte-buffer-message-set-write-to-non-blocking
They are now consistent with `waitUntilTrue`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1665 from ijuma/increase-default-wait-until-time
ijuma
As discussed in https://github.com/apache/kafka/pull/1645, this patch removes an extraneous line from several __init__.py files, and a few others as well
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1659 from granders/minor-cleanup-init-files
Fix the test by using a more liberal timeout and forcing more frequent SinkTask.put() calls. Also add some logging to aid future debugging.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1663 from ewencp/kafka-3935-fix-restart-system-test
Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.
Seeeing as `messageFormat > 0` always reads the full compressed message
set and is the default going forwards, we can use that behaviour to
always close the compressor when calling `deepIterator`
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1660 from tcrayford/dont_leak_native_memory_round_2
I’ve implemented my own custom Deserializer and been using it with `KStream.reduceByKey`; I observed that `reduceByKey` was passing null to my implementation, but it wasn’t clear to me what my implementation was expected to do in this case. So this attempts to clarify it.
This is my original work and I license this work to the Kafka project under Kafka’s open source license (the Apache License 2.0).
Author: Avi Flax <avi@aviflax.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1503 from aviflax/patch-1
(cherry picked from commit 6b1a6d9551fc44d80c7cffc0b4c21427da9d8fda)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Author: Alexey Romanchuk <al.romanchuk@2gis.ru>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1629 from 13h3r/kafka-3960
Add additional information to Acceptor debug message upon connection acceptance
Author: rnpridgeon <ryan.n.pridgeon@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1648 from rnpridgeon/trunk
5 seconds is probably enough when running tests locally, but
doesn't seem to be so for Jenkins when it is overloaded.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1589 from ijuma/increase-default-wait-time-for-wait-until-true
Better performance is always welcome:
"The Gradle build itself has seen a 50% reduction in configuration time. You'll see the biggest impact on multi-project builds"
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1644 from ijuma/update-gradle
Without this file the benchmark does not run nightly.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Geoff Anderson <geoff@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1645 from enothereska/hotfix-streams-test
Damian Guy
sahil kharb
Grant Henke
deragon
Ismael Juma
Bryan Baugher
dan norwood
Kaufman Ng
Dustin Cote
Liquan Pei
James Cheng
Jan Filipiak
Jun Rao
Jendrik Poloczek
Vahid Hashemian
Ryan P
Eno Thereska
Pierre Coquentin
Matthias J. Sax
Mickael Maison
Gwen Shapira
Jason Gustafson
Ewen Cheslack-Postava
Maysam Yabandeh
Geoff Anderson
Tom Crayford
Avi Flax
Alexey Romanchuk