Added kafka-broker-api-versions.bat for Windows platform
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian, Ismael Juma, Guozhang Wang
Closes#2943 from amethystic/kafka-5143_offer_broker_api_versions_bat
Added code to check existence of the brokers in the proposed plan.
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2962 from amethystic/kafka5161_reassign_check_invalid_brokerID
Provide correct config details in the log message.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Dustin Cote <dustin@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2932 from umesh9794/local
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>
Closes#2958 from cmccabe/KAFKA-5162
The code was correct since the method is only called from
one thread, but the change is worthwhile anyway.
Author: Amit Daga <adaga@adobe.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2966 from amitdaga/findbugs-streams-multithread
Add a parameter 'writable' for AbstractIndex and set its default value to true for its children classes.
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2905 from amethystic/kafka-5104_DumpLogSegments_should_not_open_index_files_with_rw
Explicitly throwing clear exceptions when starting up a Kafka with some non-Kafka data in data.dirs.
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2907 from amethystic/kafka-5118_improve_msg__for_failed_startup_with_nonKafka_data
Add new broker config, `group.initial.rebalance.delay.ms`, with a default of 3 seconds.
When a consumer creates a new group, set the group's state to InitialRebalance and delay the rebalance until `min(group.initial.rebalance.delay.ms, rebalanceTimeout)`. As other members join the group further delay the rebalance by `min(group.initial.rebalance.delay.ms, remainingRebalanceTimeout)`. Once `rebalanceTimeout` is hit or no new members join the group within the delay, complete the rebalance.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang
Closes#2758 from dguy/kafka-4925
…dSource
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2783 from bbejeck/HOTFIX_potentially_hanging_test_in_RegexSourceIntegrationTest
This is the implementation of KIP-114: KTable state stores and improved semantics:
- Allow for decoupling between querying and materialisation
- consistent APIs, overloads with queryableName and without
- depreciated several KTable calls
- new unit and integration tests
In this implementation, state stores are materialized if the user desires them to be queryable. In subsequent versions we can offer a second option, to have a view-like state store. The tradeoff then would be between storage space (materialize) and re-computation (view). That tradeoff can be exploited by later query optimizers.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Guozhang Wang
Closes#2832 from enothereska/KAFKA-5045-ktable
The descendingSubsequence is a misnomer. The linked list is actually arranged so that the lowest timestamp is first and larger timestamps are added to the end, therefore renamed to ascendingSubsequence.
The minElem variable was also misnamed. It's actually the current maximum element as it's taken from the end of the list.
Added comment to get() to make it clear it's returning the lowest timestamp.
Author: mihbor <mihbor@users.noreply.github.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2948 from mihbor/patch-4
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2954 from ijuma/fix-error-message-if-transactions-topic-replication-factor-too-low
Tests broker with multiple SASL mechanisms with different endpoints for different mechanisms. Each endpoint uses its own JAAS context.
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Rajini Sivaram, Ismael Juma
Closes#2506 from baluchicken/KAFKA-4703
Add a default log retention policy to keep GC logs from growing too large
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1431 from rnpridgeon/KAFKA-3754
If we pass in 0 futures to an AllOfAdapter, we should complete immediately
Author: dan norwood <norwood@confluent.io>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2953 from norwood/handle-all-of-0
Write txn markers and complete the commit/abort for transactions in PrepareXX
state during partition immigration.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2926 from dguy/kafka-5059
KAFKA-5028 moves the controller to a single-threaded model, so we would no longer have work interleaved between preferred replica leader election, meaning we don't need to keep its state.
This patch additionally addresses a bug from KAFKA-5028 where it made onPreferredReplicaElection keep the line calling topicDeletionManager.markTopicIneligibleForDeletion but removes the line calling topicDeletionManager.resumeDeletionForTopics
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2927 from onurkaraman/KAFKA-5107
Moving the coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry will generate fewer broker send requests
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#2925 from dguy/tc-write-txn-request-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2933 from hachikuji/minor-transactional-client-cleanup
remove broken locking. fix handleAddPartitions after complete commit/abort
respond with CONCURRENT_TRANSACTIONS in initPid
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2934 from dguy/follow-up-tc-work
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Dan Norwood <norwood@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2472 from cmccabe/KAFKA-3265
ReassignPartitionsCommand should protect against empty replica list assignment.
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2904 from amethystic/kafka-5901_ReassignPartitionsCommand_protect_against_empty_replica_list
As per KIP-82
Adding record headers api to ProducerRecord, ConsumerRecord
Support to convert from protocol to api added Kafka Producer, Kafka Fetcher (Consumer)
Updated MirrorMaker, ConsoleConsumer and scala BaseConsumer
Add RecordHeaders and RecordHeader implementation of the interfaces Headers and Header
Some bits using are reverted to being Java 7 compatible, for the moment until KIP-118 is implemented.
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Radai Rosenblatt <radai.rosenblatt@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2772 from michaelandrepearce/KIP-82
I verified that the test would trigger an `IllegalStateException` if the
`position` call was added back.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska <eno@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2887 from ijuma/kafka-5097-unit-test
- addressing open Github comments from #2773
- test clean-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#2854 from mjsax/kafka-4986-producer-per-task-follow-up
1. Added a flag to indicate if it is restoring or not in the LRU Store; since we only have a restore callback we have to set it each time applying the change.
2. Fixed the corresponding unit test, plus some minor cleaning up.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2908 from guozhangwang/K4379-remove-listener
- mainly moving methods
- also improved logging
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2917 from mjsax/kafka-5111-code-cleanup-follow-up
* Producer and Consumer `close` calls were not handled via `try-with-resources`
* `cleanRun` unused field removed
* Refactored handling of Consumer configuration in `IntegrationTestUtils` to ensure auto-committing of offsets and starting from `earliest`
* As a result reverted https://github.com/apache/kafka/pull/2921 since it's redundant now
Author: Armin Braun <me@obrown.io>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2920 from original-brownbear/cleanup-it-utils-closing
Depending on the test execution order, the global registry would
contain some metrics causing the test to fail.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
Closes#2915 from ijuma/socket-server-test-metric-collection-after-shutdown
Update topic expiry time after every metadata update to handle max.block.ms greater than 5 minutes
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2869 from lindong28/KAFKA-5086
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2840 from apurvam/exactly-once-transactional-clients
The goal of this ticket is to improve controller maintainability by simplifying the controller's concurrency semantics. The controller code has a lot of shared state between several threads using several concurrency primitives. This makes the code hard to reason about.
This ticket proposes we convert the controller to a single-threaded event queue model. We add a new controller thread which processes events held in an event queue. Note that this does not mean we get rid of all threads used by the controller. We merely delegate all work that interacts with controller local state to this single thread. With only a single thread accessing and modifying the controller local state, we no longer need to worry about concurrent access, which means we can get rid of the various concurrency primitives used throughout the controller.
Performance is expected to match existing behavior since the bulk of the existing controller work today already happens sequentially in the ZkClient’s single ZkEventThread.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2816 from onurkaraman/KAFKA-5028
Author: Damian Guy <damian.guy@gmail.com>
Author: Guozhang Wang <wangguoz@gmail.com>
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang, Jason Gustafson, Apurva Mehta, Jun Rao
Closes#2849 from dguy/exactly-once-tc
Fixes `org.apache.kafka.streams.integration.utils.IntegrationTestUtils#readKeyValues` potentially starting to `poll` for stream output after the stream finished sending the test data and hence missing it when working with `latest` offsets.
Author: Armin Braun <me@obrown.io>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2921 from original-brownbear/KAFKA-5124
Refactors Task with proper interface methods `init()`, `resume()`, `commit()`, `suspend()`, and `close()`. All other methods for task handling are internal now. This allows to simplify `StreamThread` code, avoid code duplication and allows for easier reasoning of control flow.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2895 from mjsax/kafka-5111-cleanup-task-code
amethystic
Matthias J. Sax
Colin P. Mccabe
umesh chaudhary
Amit Daga
Ismael Juma
Damian Guy
Bill Bejeck
Eno Thereska
Davor Poldrugo
Michal Borowiecki
Balint Molnar
Ryan P
Bernard Leach
dan norwood
Apurva Mehta
Onur Karaman
sharad-develop
Guozhang Wang
Jason Gustafson
Rajini Sivaram
Michael Andre Pearce
Armin Braun
Dong Lin