Write txn markers and complete the commit/abort for transactions in PrepareXX
state during partition immigration.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2926 from dguy/kafka-5059
KAFKA-5028 moves the controller to a single-threaded model, so we would no longer have work interleaved between preferred replica leader election, meaning we don't need to keep its state.
This patch additionally addresses a bug from KAFKA-5028 where it made onPreferredReplicaElection keep the line calling topicDeletionManager.markTopicIneligibleForDeletion but removes the line calling topicDeletionManager.resumeDeletionForTopics
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2927 from onurkaraman/KAFKA-5107
Moving the coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry will generate fewer broker send requests
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#2925 from dguy/tc-write-txn-request-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2933 from hachikuji/minor-transactional-client-cleanup
remove broken locking. fix handleAddPartitions after complete commit/abort
respond with CONCURRENT_TRANSACTIONS in initPid
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2934 from dguy/follow-up-tc-work
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Dan Norwood <norwood@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2472 from cmccabe/KAFKA-3265
ReassignPartitionsCommand should protect against empty replica list assignment.
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2904 from amethystic/kafka-5901_ReassignPartitionsCommand_protect_against_empty_replica_list
As per KIP-82
Adding record headers api to ProducerRecord, ConsumerRecord
Support to convert from protocol to api added Kafka Producer, Kafka Fetcher (Consumer)
Updated MirrorMaker, ConsoleConsumer and scala BaseConsumer
Add RecordHeaders and RecordHeader implementation of the interfaces Headers and Header
Some bits using are reverted to being Java 7 compatible, for the moment until KIP-118 is implemented.
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Radai Rosenblatt <radai.rosenblatt@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2772 from michaelandrepearce/KIP-82
I verified that the test would trigger an `IllegalStateException` if the
`position` call was added back.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska <eno@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2887 from ijuma/kafka-5097-unit-test
- addressing open Github comments from #2773
- test clean-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#2854 from mjsax/kafka-4986-producer-per-task-follow-up
1. Added a flag to indicate if it is restoring or not in the LRU Store; since we only have a restore callback we have to set it each time applying the change.
2. Fixed the corresponding unit test, plus some minor cleaning up.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2908 from guozhangwang/K4379-remove-listener
- mainly moving methods
- also improved logging
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2917 from mjsax/kafka-5111-code-cleanup-follow-up
* Producer and Consumer `close` calls were not handled via `try-with-resources`
* `cleanRun` unused field removed
* Refactored handling of Consumer configuration in `IntegrationTestUtils` to ensure auto-committing of offsets and starting from `earliest`
* As a result reverted https://github.com/apache/kafka/pull/2921 since it's redundant now
Author: Armin Braun <me@obrown.io>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2920 from original-brownbear/cleanup-it-utils-closing
Depending on the test execution order, the global registry would
contain some metrics causing the test to fail.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
Closes#2915 from ijuma/socket-server-test-metric-collection-after-shutdown
Update topic expiry time after every metadata update to handle max.block.ms greater than 5 minutes
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2869 from lindong28/KAFKA-5086
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2840 from apurvam/exactly-once-transactional-clients
The goal of this ticket is to improve controller maintainability by simplifying the controller's concurrency semantics. The controller code has a lot of shared state between several threads using several concurrency primitives. This makes the code hard to reason about.
This ticket proposes we convert the controller to a single-threaded event queue model. We add a new controller thread which processes events held in an event queue. Note that this does not mean we get rid of all threads used by the controller. We merely delegate all work that interacts with controller local state to this single thread. With only a single thread accessing and modifying the controller local state, we no longer need to worry about concurrent access, which means we can get rid of the various concurrency primitives used throughout the controller.
Performance is expected to match existing behavior since the bulk of the existing controller work today already happens sequentially in the ZkClient’s single ZkEventThread.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2816 from onurkaraman/KAFKA-5028
Author: Damian Guy <damian.guy@gmail.com>
Author: Guozhang Wang <wangguoz@gmail.com>
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang, Jason Gustafson, Apurva Mehta, Jun Rao
Closes#2849 from dguy/exactly-once-tc
Fixes `org.apache.kafka.streams.integration.utils.IntegrationTestUtils#readKeyValues` potentially starting to `poll` for stream output after the stream finished sending the test data and hence missing it when working with `latest` offsets.
Author: Armin Braun <me@obrown.io>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2921 from original-brownbear/KAFKA-5124
Refactors Task with proper interface methods `init()`, `resume()`, `commit()`, `suspend()`, and `close()`. All other methods for task handling are internal now. This allows to simplify `StreamThread` code, avoid code duplication and allows for easier reasoning of control flow.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2895 from mjsax/kafka-5111-cleanup-task-code
Author: Sean McCauliff <smccauliff@linkedin.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2659 from smccauliff/kafka-4840
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2745 from hachikuji/add-replication-testcase-for-compression
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2897 from mjsax/KAFKA-4564-follow-up
The `common` package is public and this class is
internal.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2759 from ijuma/move-os-to-utils
- call close() on Metrics to join created threads
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska, Damian Guy, Guozhang Wang
Closes#2788 from mjsax/minor-improve-test-metric-cleanup
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2837 from mjsax/kafka-4564-fail-fast-test-stream-compatibility
If `partition==null` and `partitioner!=null` we should not fall back to default partitioner (as we do before the patch if `producer.partitionsFor(...)` returns empty list. Falling back to default partitioner might corrupt hash partitioning.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska, Damian Guy, Guozhang Wang
Closes#2868 from mjsax/minor-fix-RecordCollector
- typo error corrected (spelling)
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2885 from Kamal15/log
fix some spelling errors
Author: xinlihua <xin.lihua1@zte.com.cn>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2871 from auroraxlh/fix_spellingerror
Skip null keys when initializing GlobalKTables. This is inline with what happens during normal processing.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2834 from dguy/kafka-5047
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2789 from mjsax/minor-improve-integration-test
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2813 from ijuma/kafka-5014-least-loaded-node-should-check-if-channel-is-ready
Damian Guy
Onur Karaman
sharad-develop
dan norwood
Guozhang Wang
Jason Gustafson
Colin P. Mccabe
amethystic
Rajini Sivaram
Michael Andre Pearce
Ismael Juma
Matthias J. Sax
Balint Molnar
Armin Braun
Dong Lin
Apurva Mehta
Sean McCauliff
Jun Rao
Michal Borowiecki
Kamal C
Lihua Xin
Eno Thereska