Also:
* Make all implementations of `Time` thread-safe as they are accessed from multiple threads in some cases.
* Change default implementation of `MockTime` to use two separate variables for `nanoTime` and `currentTimeMillis` as they have different `origins`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Shikhar Bhushan <shikhar@confluent.io>, Jason Gustafson <jason@confluent.io>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#2095 from ijuma/kafka-2247-consolidate-time-interfaces
Author: Alexey Ozeritsky <aozeritsky@yandex-team.ru>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2125 from resetius/KAFKA-4399
Removed stale comment left behind, minor fixes (UpdateMetadataRequest instead of MetadataUpdateRequest) and remove redundant comments.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jiangjie (Becket) Qin <becket.qin@gmail.com>, Dong Lin <lindong28@gmail.com>
Closes#2194 from ijuma/kafka-4443-minor-follow-up
The last patch submitted by MayureshGharat (back in Dec 15) has been rebased to the latest trunk. I took care of a couple of test failures (MetricsTest) along the way. jjkoshy , granders , avianey , you may be interested in this PR.
Author: Sumant Tambe <sutambe@yahoo.com>
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>
Closes#1664 from sutambe/async-delete-topic
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2168 from lindong28/KAFKA-4443
Author: Jun He <jun.he@airbnb.com>
Reviewers: Jiangjie (Becket) Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2127 from jun-he/KAFKA-4384
- bug-fix follow up
- Resetter fails if no intermediate topic is used because seekToEnd() commit ALL partitions to EOL
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll, Roger Hoover, Guozhang Wang
Closes#2138 from mjsax/kafka-4331-streams-resetter-bugfix
Fixes static initialization order dependency between KafkaConfig and LogConfig. jjkoshy please take a look.
Author: Sumant Tambe <sutambe@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2120 from sutambe/logconfig-static-init
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2116 from MayureshGharat/KAFKA-4362
Process requests received from channels before they were closed. For consumers, wait for coordinator requests to complete before returning from close.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1836 from rajinisivaram/KAFKA-3703
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2148 from lindong28/KAFKA-4420
JavaConversions are deprecated in 2.12 in favour of JavaConverters.
Author: Bernard Leach <leachbj@bouncycastle.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2101 from leachbj/4377-java-converters
I think we should add a space here, otherwise the two words will join together.
And replace the host string with a constant, otherwise when I need to modify the host, I need to modify several files.
Author: 郑谦00117553 <00117553@zte.intra>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2132 from ZhengQian1/trunk
The consumer can deadlock on shutdown if a topic event fires during shutdown. The shutdown acquires the rebalance lock and then the topic-event-watcher lock. The topic event watcher acquires these in the reverse order. Shutdown should not need to acquire the topic-event-watcher’s lock - all it does is unsubscribes from topic events.
Author: Joel Koshy <jjkoshy@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>
Closes#2129 from jjkoshy/KAFKA-4409
This bug was recently introduced via 2959bc2ad.
Author: Himani Arora <1himani.arora@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2091 from himani1/minor_fix
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Jason Gustafson <jason@confluent.io>
Closes#2075 from vahidhashemian/KAFKA-4357
This test fails locally when I run it, but somehow Jenkins builds are passed. Not clear how.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2073 from ijuma/async-producer-test-port-config
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2053 from hachikuji/KAFKA-4326
The edge case where consumer group state is `PreparingRebalance` or `AwaitingSync` will be separately handled as the group assignment is not yet determined.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2070 from vahidhashemian/KAFKA-4349
We had already made this change to the client created in `ZooKeeperTestHarness`.
I last saw this failure when `SaslPlaintextTopicMetadataTest.testAliveBrokerListWithNoTopics`
was executed in Jenkins.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2066 from ijuma/increase-zk-timeout-in-kafka-server
Also use named parameters in KafkaServer for clarity (even though it was correct previously).
Author: Matt <wangm92@163.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1646 from wangzzu/wangzzu
There should be only one cases where these clean-ups have a functional impact: replaced repeated identical logs with a single log for the stale controller epoch case.
The rest should just make the code easier to read and make it a bit less wasteful. I did this exercise because unused variables sometimes mask bugs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1985 from ijuma/remove-unused
- reworked to use a sinlge KafkaConsumer and subscribe only once
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2049 from mjsax/improveResetTool
- fixed consumer group dead condition
- disabled state store cache
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2056 from mjsax/KAFKA-4058-instableResetToolTest
This PR makes a couple of enhancements to the `--describe` option of `ConsumerGroupCommand`:
1. Listing members with no assigned partitions.
2. Showing the member id along with the owner of each partition (owner is supposed to be the logical application id and all members in the same group are supposed to set the same owner).
3. Printing a warning indicating whether ZooKeeper based or new consumer API based information is being reported.
It also adds unit tests to verify the added functionality.
Note: The third request on the corresponding JIRA (listing active offsets for empty groups of new consumers) is not implemented as part of this PR, and has been moved to its own JIRA (KAFKA-3853).
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jun Rao <junrao@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1336 from vahidhashemian/KAFKA-3144
Author: Alexey Ozeritsky <aozeritsky@yandex-team.ru>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2023 from resetius/AbstractFetcherManager-shutdown-speedup
Author: Ishita Mandhan <imandha@us.ibm.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#2013 from imandhan/KAFKA-refactor
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2019 from hachikuji/KAFKA-4298
Author: Konstantine Karantasis <konstantine@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1995 from kkonstantine/KAFKA-4254-Update-producers-metadata-before-failing-on-non-existent-partition
Remove topics under /admin/delete_topics path in zk if deleteTopic is disabled. The topic should never be enqueued for deletion.
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#846 from MayureshGharat/kafka-3175
Test expects all records to be published successfully, which cannot be guaranteed with acks=0 since failures are not retried.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ben Stopford <benstopford@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1991 from rajinisivaram/KAFKA-4265
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1993 from becketqin/KAFKA-4274
We recently switched `SystemTimer` to use `hiResClockMs` (based on `nanoTime`), but we
were still using `System.currentTimeMillis` in the test. That would sometimes mean
that we would measure elapsed time as lower than expected.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>
Closes#1890 from ijuma/fix-test-request-satisfaction-transient-failure
To prevent test from completing without throttling before config change takes effect, produce more messages in the test.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ben Stopford <benstopford@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1982 from rajinisivaram/KAFKA-4262
Grow read and write buffers of cleaner up to the maximum message size of the log being cleaned if the topic has larger max message size than the default config of the broker.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1758 from rajinisivaram/KAFKA-4019
Ismael Juma
Alexey Ozeritsky
Mayuresh Gharat
Dong Lin
Jun He
Matthias J. Sax
Sumant Tambe
Rajini Sivaram
Bernard Leach
Qian Zheng
Jason Gustafson
Joel Koshy
Tu Yang
Himani Arora
Vahid Hashemian
wangzzu
Jun Rao
Ishita Mandhan
Konstantine Karantasis
Radai Rosenblatt
Jiangjie Qin
Ben Stopford