Closed the last PR I made for this because I accidentally borked it with my other PR. Small error; I figure this is from copy-pasting the above doc
Author: Nikki Thean <nthean@etsy.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2336 from nixsticks/trunk
When creating the source node in TopologyBuilder, we need to decorate its input topics if they are inner (i.e. repartition) topics with the prefix.
Also did some minor cleanup in the printing function for better visualization in debugging.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska, Damian Guy, Eno Thereska, Jun Rao
Closes#2320 from guozhangwang/KMinor-source-topic-fix
Remove use of TestTimestampExtractor as it causes the logs to roll and segments get deleted.
Remove the wcnt example as it is dependent on the TestTimestampExtractor - windowed counting is covered elsewhere.
Change all aggregate operations to use TimeWindow as use of UnlimitedWindow was causing logs to roll and segments being deleted.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang, Eno Thereska
Closes#2319 from dguy/smoke-test
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Vahid Hashemian <vahidhashemian@us.ibm.com>, Konstantin <konstantin@tubemogul.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2128 from yaojuncn/KAFKA-4402-client-producer-round-robin-fix
Add support for SessionWindows based on design detailed in https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows.
This includes refactoring of the RocksDBWindowStore such that functionality common with the RocksDBSessionStore isn't duplicated.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#2166 from dguy/kafka-3452-session-merge
Adds a bunch of tests to unit tests to the assignment command.
Moves the Rack aware test into its own class as it makes use of ZooKeeperTestHarness and slows everything else down.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1950 from benstopford/os-rebalance-extra-unit-testing
Otherwise in this test the sink task goes through the pause/resume cycle with 0 assigned partitions, since the default metadata refresh interval is quite long
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2313 from shikhar/kafka-4575
Shut down the group coordinator before shutting down the log manager to
ensure that any delayed operations are completed before the logs are closed.
Author: steve <sniemitz@twitter.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2311 from steveniemitz/KAFKA-4523
Removed the extra ',' character while printing the replicas / in-sync replicas
array.
Author: Kamal <kamal@nmsworks.co.in>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2306 from Kamal15/trunk
In reality, we’ll only test older brokers after KAFKA-4462 is fully implemented.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2263 from cmccabe/KAFKA-4508
ProducerConfig calls AbstractConfig.init where does the logs. KafkaProducer init will inovoke ProducerConfig.init twice that leads to logging twice.
Author: huxi <huxi@zhenrongbao.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2307 from amethystic/kafka-4434_Kafkaproducer_log_twice
Jason recently cleaned things up significantly by consolidating the Message/Record classes
into the common Java code in the clients module. While reviewing that, I noticed a few things
that could be improved a little more. To make reviewing easier, there will be multiple PRs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Jason Gustafson <jason@confluent.io>
Closes#2271 from ijuma/records-minor-fixes
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Jason Gustafson <jason@confluent.io>
Closes#2155 from lindong28/KAFKA-4429
I was able to reproduce the failure in less than 10 runs before the change. With the change,
the test passed 70 times consecutively.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang
Closes#2298 from ijuma/kafka-4528-fix-test-async-send-timeout
In `shutdownTasksAndState` and `suspendTasksAndState` we commit offsets BEFORE we flush any state. This is wrong as if an exception occurs during a flush, we may violate the at-least-once guarantees, that is we would have committed some offsets but NOT sent the processed data on to other Sinks.
Also during suspend and shutdown, we should try and complete all tasks even when exceptions occur. We should just keep track of the exception and rethrow it at the end if necessary. This helps with ensuring that StateStores etc are closed.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2281 from dguy/kafka-4561
This makes it consistent with MirrorMaker with the old consumer.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2072 from amethystic/kafka-4351_Regex_behavior_change_for_new_consumer
The original Javadoc description for `ConsumerRecord` is slightly confusing in that it can be read in a way such that an object is a key value pair received from Kafka, but (only) consists of the metadata associated with the record. This PR makes it clearer that the metadata is _included_ with the record, and moves the comma so that the phrase "topic name and partition number" in the sentence is more closely associated with the phrase "from which the record is being received".
Author: LoneRifle <LoneRifle@users.noreply.github.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2290 from LoneRifle/patch-1
Mx4jLoader.scala should explicitly `return true` if the class is successfully loaded and started, otherwise it will return false even if the class is loaded.
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2295 from eribeiro/mx4jloader-bug
Some of the recent changes to `kafka-run-class.sh` have not been applied to `kafka-run-class.bat`.
These recent changes include setting proper streams or connect classpaths. So any streams or connect use case that leverages `kafka-run-class.bat` would fail with an error like
```
Error: Could not find or load main class org.apache.kafka.streams.???
```
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2238 from vahidhashemian/minor/sync_up_kafka-run-class.bat
During Acceptor initialization, if "Address already in use" error is encountered,
the shutdown latch in each Processor is never counted down. Consequently,
the Kafka server hangs when `Processor.shutdown` is called.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2156 from amethystic/kafka-4428_Kafka_noexit_for_port_already_use
Author: MURAKAMI Masahiko <fossamagna2@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2265 from fossamagna/fix-lz4outputstream-close
adding a LogConfig value validator. gwenshap or junrao would you mind taking a look?
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1796 from cotedm/retentionbytesvalidation
h/t ewencp for pointing out the issue
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2277 from shikhar/kafka-4527
4 release cycles (0.9.0.0, 0.10.0.0, 0.10.1.0, 0.10.2.0) should be enough
to remove the beta label.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2286 from ijuma/kafka-3284-security-beta-label
The current output for the `--generate` option looks like this
```
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[0]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[1]}]}
```
This PR simply changes it to
```
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[0]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[1]}]}
```
to make it more readable.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1868 from vahidhashemian/minor/improve_output_format_of_reassign_partitions
Based on: https://github.com/apache/kafka-site/pull/27
I recommend also merging into 10.1.0.0 branch to avoid mishaps when re-publishing docs.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2077 from gwenshap/protocol_doc_fix
I spent a bit of time tracking down why files were being deleted before they reached log.retention.hours of age. It turns out that the time and size log retention schemes function independently, and not as the original comment "The minimum age of a log file to be eligible for deletion" might indicate to a new user.
Author: Mark Rose <markrose@markrose.ca>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#28 from MarkRose/fix_misleading_configuration_file_for_trunk
This was changed in b58b6a1bef and caused the `ReplicaVerificationToolTest.test_replica_lags`
system test to start failing.
I also added a unit test and a couple of other minor clean-ups.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2280 from ijuma/kafka-4554-fix-replica-buffer-verify-checksum
Nikki Thean
Magnus Reftel
Guozhang Wang
Matthias J. Sax
Damian Guy
yaojuncn
Raghav Kumar Gautam
Ben Stopford
Shikhar Bhushan
steve
Kamal
Colin P. Mccabe
Rajini Sivaram
huxi
Vahid Hashemian
Ismael Juma
fpj
Dong Lin
Ewen Cheslack-Postava
Andrew Olson
Balint Molnar
Himani Arora
LoneRifle
Edward Ribeiro
MURAKAMI Masahiko
Jeff Widman
Dustin Cote
Gwen Shapira
Manikumar Reddy O
Scott Ferguson
Mark Rose
Jason Gustafson