This field doesn't seem to be used and the value for
`AwaitingSync` seems to be wrong (it seems like it
should have been `2` instead of `5`).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3572 from ijuma/remove-unused-group-state-field
Added deprecation and warning message when the --new-consumer
option is used with the ConsumerGroupCommand and
the ConsumerPerformance tools.
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3555 from ppatierno/kafka-5619
Marked --new-consumer as deprecated in the help of ConsoleConsumer
and a warning will be printed if it's used.
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3537 from ppatierno/kafka-5599
Also fix potential NPE.
Author: Manikumar Reddy <manikumar.reddy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3615 from omkreddy/zk-notif-duplicates
Author: Abhishek Mendhekar <amendhekar@linkedin.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3549 from abhishekmendhekar/KAFKA-5461
This patch replaces the legacy ControlledShutdown objects in `kafka.api` with the alternatives in `org.apache.kafka.common.requests`. Since this was the last API that needed updating, we have also dropped the reference in `RequestChannel.Request` to the legacy object type.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3612 from hachikuji/remove-old-controlled-shutdown-objects
The previous code kept two references to `Buffer` and only nulled
out one of them.
As part of this, I removed the `case` modifier from
`RequestChannel.{Request, Response}`. They don't seem to be good
matches given the types of fields they contain (mutable buffers and
opaque `Send` instances).
Also removed a couple of unused files in `kafka.network`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Radai Rosenblatt <radai.rosenblatt@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3596 from ijuma/release-buffer-in-request
At some point, we lost the ability to output request
logging at debug level (which is a little less
verbose than at trace level).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3595 from ijuma/fix-debug-request-log
This patch changes the segment deletion behavior to take the high watermark of the partition into account. In particular, segments containing offsets equal to or larger than the high watermark are no longer eligible for deletion. This is needed to ensure that the log start offset reported in fetch responses does not get ahead of the high watermark.
Impact: segment deletion may be delayed compared to existing behavior since the broker must await advancement of the high watermark. For topics with heavy load, this may make the active segment effectively ineligible for deletion since the high watermark may never catch up to the log end offset.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3575 from hachikuji/KAFKA-5634
In a test by onurkaraman involving 3066 topics and 95895 partitions,
Controller initialisation time spent on JSON parsing would be reduced from
37.1 seconds to 0.7 seconds by switching from the current JSON parser to
Jackson. See the following JIRA comment for more details:
https://issues.apache.org/jira/browse/KAFKA-5328?focusedCommentId=16027086
I tested that we only use Jackson methods introduced in 2.0 in the main
codebase by compiling it with the older version locally. We use a
constructor introduced in 2.4 in one test, but I didn't remove it as it
seemed harmless. The reasoning for this is explained in the mailing list
thread:
http://search-hadoop.com/m/uyzND1FWbWw1qUbWe
Finally, this PR only handles the parsing side. It would be good to use Jackson
for serialising to JSON as well. I filed KAFKA-5631 for that.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#83 from ijuma/kafka-1595-remove-deprecated-json-parser-jackson
this is the initial implementation.
Author: radai-rosenblatt <radai.rosenblatt@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Jun Rao <junrao@gmail.com>
Closes#2330 from radai-rosenblatt/broker-memory-pool-with-muting
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3565 from lindong28/KAFKA-5627
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Onur Karaman <okaraman@linkedin.com>
Closes#2929 from lindong28/KAFKA-4763
For a compacted topic with preallocate enabled, during log cleaning, LogCleaner.cleanSegments does not have to pre-allocate the underlying file size since we only want to store the cleaned data in the file.
It's believed that this fix should also solve KAFKA-5582.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3525 from huxihx/log_compact_test
For topics that support timestamp search, if no offset is found for a partition, the partition should still be included in the result with a `null` offset value. This `KafkaConsumer` method currently excludes such partitions from the result.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3460 from vahidhashemian/KAFKA-5534
Before this patch, we would instead return the non-retriable `UNSUPPORTED_FOR_MESSAGE_FORMAT` causing markers to be lost.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3550 from apurvam/KAFKA-5610-handleWriteTxnMarker-should-handle-emigration
I was investigating an exception in this code and found a few
opportunities for making it clearer.
I also added the `out` folder to `.gitignore` as IntelliJ sometimes
uses that as the build folder.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3552 from ijuma/minor-quota-improvements
the while loop was too big and need to be closed earlier
to see the fix, ignore whitespace since most of it is indentation
this bug was introduced by commit
5bd06f1d542e6b588a1d402d059bc24690017d32
Author: Jan Burkhardt <jan.burkhardt@just.social>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3538 from bjrke/trunk
Also add connection id to KafkaChannel exception message
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3529 from rajinisivaram/MINOR-log-connection-id
Currently, the output of `ConsumerPerformance` looks strange. The `header` format as follow:
```
"time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec"
```
while the `body` as follow:
```
println("%s, %d, %.4f, %.4f, %d, %.4f".format(dateFormat.format(endMs), id, totalMBRead,
1000.0 * (mbRead / elapsedMs), messagesRead, ((messagesRead - lastMessagesRead) / elapsedMs) * 1000.0))
```
So we get the follow result:
```
time, data.consumeed.in.MB, MB.sec, data.consumeed.in.nMsg, nMsg.Sec
09:52:00, 0, 1100.3086, 220.0177, 563358, 112649.0702
```
So the `header` and `body` mismatching.
And also, this pr makes the functions more readable.
Author: Xianyang Liu <xianyang.liu@intel.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3417 from ConeyLiu/consumertest
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3523 from guozhangwang/KMinor-group-coordinator-member-failure-info-log4j
Update the code where possible to fix the warnings. The unused
warning introduced in Scala 2.12 is quite handy and provides
a reason to compile with Scala 2.12.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3464 from ijuma/scala-xlint
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2919 from baluchicken/KAFKA-5127
It may lead to wrong metrics and it may break
size-based retention.
Author: Gregor Uhlenheuer <kongo2002@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3521 from kongo2002/KAFKA-5584
Fixed console output to be consistent with/without dry-run option
Author: ppatierno <ppatierno@live.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3443 from ppatierno/kafka-5525
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3456 from hachikuji/KAFKA-5522
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3457 from hachikuji/rename-base-timestamp
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3446 from ijuma/improve-transaction-index-exception-sanity-check
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3303 from rajinisivaram/KAFKA-5402
This is used in the generated config table. Also fix a couple
of typos in the process.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3451 from ijuma/fix-doc-typos
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#3406 from hachikuji/KAFKA-5490
I'll let this have multiple runs on the branch builder to see if it fails, and investigate if so.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3441 from apurvam/MINOR-enable-transactions-bounce-test
We intended to make `KafkaConsumer.close()` idempotent,
but due to the fact that the `closed` variable is
checked without a lock prior to beginning close logic,
it is possible for two or more threads to see
`closed=false` and attempt to close.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3426 from hachikuji/minor-fix-consumer-idempotent-close
Ismael Juma
Balint Molnar
Paolo Patierno
Manikumar Reddy
Abhishek Mendhekar
Jason Gustafson
Dong Lin
Onur Karaman
radai-rosenblatt
Ewen Cheslack-Postava
huxihx
Vahid Hashemian
Apurva Mehta
Gwen Shapira
Jan Burkhardt
Rajini Sivaram
Xianyang Liu
Guozhang Wang
Gregor Uhlenheuer
Ben Stopford
Matthias J. Sax