As a follow-up from [KIP-482](https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields), this PR bumps the version for several
RPC's to enable tagged fields via the flexible versioning mechanism.
Additionally, a new IBP version `KAFKA_2_8_IV0` is introduced to
allow replication to take advantage of these new RPC versions for
OffsetForLeaderEpoch and ListOffset.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Followup to PR #9582, need to restrict DSL so only KStreams can be created from the same set of topic(s)s but not KTables, which can be tackled as followup work
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bruno Cadonna <cadonna@confluent.io>
A few small cleanups in `Partition` handling of `AlterIsr`:
- Factor state update and log message into `sendAlterIsrRequest`
- Ensure illegal state error gets raised if a retry fails to be enqueued
- Always check the proposed state against the current state in `handleAlterIsrResponse`
- Add `toString` implementations to `IsrState` case classes
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The unit tests for method ProcessorTopology#updateSourceTopics() did not cover all
code paths.
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
SSH outputs in system tests originating from paramiko are bytes. However, the logger in the system tests does not accept bytes and instead throws an exception. That means, the bytes returned as SSH output from paramiko need to converted to a type that the logger (or other objects) can process.
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Compacted topics can accumulate a large number of empty transaction markers as the data from the transactions gets cleaned. For each transaction, there is some bookkeeping that leaders and followers must do to keep the transaction index up to date. The cost of this overhead can degrade performance when a replica needs to catch up if the log has mostly empty or small transactions. This patch improves the cost by skipping over empty transactions since these will have no effect on the last stable offset and do not need to be reflected in the transaction index.
Reviewers: Lucas Bradstreet <lucas@confluent.io>, Jun Rao <junrao@gmail.com>
We should ignore any source nodes that aren't part of the ProcessorTopology's subtopology when updating its source topics after a change in the topic metadata.
Reviewers: Bruno Cadonna <cadonna@confluent.io>, Matthias J. Sax <mjsax@confluent.io>
The flaky test is because we didn't wait for the streams become RUNNING before verifying the state becoming ERROR state. This fix explicitly wait for the streams become RUNNING state. Also, put the 2nd stream into try resource block so it will be closed after the test.
Reviewers: Walker Carlson <wcarlson@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
Scala 2.13.4 restores default global `ExecutionContext` to 2.12 behavior
(to fix a perf regression in some use cases) and improves pattern matching
(especially exhaustiveness checking). Most of the changes are related
to the latter as I have enabled the newly introduced `-Xlint:strict-unsealed-patmat`.
More details on the code changes:
* Don't swallow exception in `ReassignPartitionsCommand.topicDescriptionFutureToState`.
* `RequestChannel.Response` should be `sealed`.
* Introduce sealed ClientQuotaManager.BaseUserEntity to avoid false positive
exhaustiveness warning.
* Handle a number of cases where pattern matches were not exhaustive:
either by marking them with @unchecked or by adding a catch-all clause.
* Workaround scalac bug related to exhaustiveness warnings in ZooKeeperClient
* Remove warning suppression annotations related to the optimizer that are no
longer needed in ConsumerGroupCommand and AclAuthorizer.
* Use `forKeyValue` in `AclAuthorizer.acls` as the scala bug preventing us from
using it seems to be fixed.
* Also update scalaCollectionCompat to 2.3.0, which includes minor improvements.
Full release notes:
* https://github.com/scala/scala/releases/tag/v2.13.4
* https://github.com/scala/scala-collection-compat/releases/tag/v2.3.0
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
We use a background thread for Kerberos to perform re-login before tickets expire. The thread performs logout() followed by login(), relying on the Java library to clear and then populate credentials in Subject. This leaves a timing window where clients fail to authenticate because credentials are not available. We cannot introduce any form of locking since authentication is performed on the network thread. So this commit treats NO_CRED as a transient failure rather than a fatal authentication exception in clients.
Reviewers: Ron Dagostino <rdagostino@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>
This patch fixes a liveness bug which prevents follower truncation from completing after a leader election. If there are consecutive leader elections without writing any data entries, then the leader and follower may have conflicting epoch entries at the end of the log.
The problem is the shortcut return in `Log.truncateTo` when the truncation offset is larger than or equal to the end offset, which prevents the conflicting entries from being resolved. Here we change this case to ensure `LeaderEpochFileCache.truncateFromEnd` is still called.
Reviewers: Jun Rao <junrao@gmail.com>
This patch factors out some common parsing logic from `NetworkClient.parseResponse` and `AbstractResponse.parseResponse`. As a result of this refactor, we are now verifying the correlationId in forwarded requests. This patch also adds a test case to verify handling in this case.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Boyang Chen <boyang@confluent.io>
The `testLogCleanerStats` test in `LogCleanerTest.scala` was not implemented but not enabled. This PR adds the `@Test` annotation, and also gives it a larger map to allow the test to pass as intended. Also fixes a few misleading comments.
Reviewers: Jason Gustafson <jason@confluent.io>
This patch changes the grouping of `Send` objects created by `SendBuilder` in order to reduce the number of generated `Send` objects and thereby the number of system writes.
Reviewers: David Jacot <djacot@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
The asynchronous shutdown in LogManager has the shortcoming that if during shutdown any of the internal futures fail, then we do not always ensure that all futures are completed before LogManager.shutdown returns. This is because, this line in the finally clause shuts down the thread pools asynchronously. As a result, despite the shut down completed message from KafkaServer is seen in the error logs, some futures continue to run from inside LogManager attempting to close some logs. This is misleading during debugging. Also sometimes it introduces an avoidable post-shutdown activity where resources (such as file handles) are released or persistent state is checkpointed in the Broker.
In this PR, we fix the above behavior such that we prevent leakage of threads. If any of the futures throw an error, we skip creating of checkpoint and clean shutdown file only for the affected log directory. We continue to wait for all futures to complete for all the directories.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
This PR implements the part of KIP-612 for adding IP throttling enforcement, and a ZK entity for configuring dynamic IP throttles.
Reviewers: Anna Povzner <anna@confluent.io>, David Jacot <djacot@confluent.io>
This PR migrates the OffsetsForLeaderEpoch request/response to the automated protocol. It also refactors the OffsetsForLeaderEpochClient to use directly the internal structs generated by the automated protocol. It relies on the existing tests.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Jason Gustafson <jason@confluent.io>
- part of KIP-572
- when KafkaStreams commits a task, a TimeoutException should not kill
the thread but `task.timeout.ms` should be triggered and the commit
should be retried in the next loop
Reviewer: John Roesler <john@confluent.io>
Topic IDs must be created for all new topics and all existing topics that do not yet have a topic ID. In ZooKeeper, the ID is written to the TopicZNode, and in the controller, it is stored in a map.
This is a preliminary change before the second part, which will propagate these IDs to brokers.
Reviewers: Lucas Bradstreet <lucas@confluent.io>, dengziming <dengziming1993@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Montyleo
Chia-Ping Tsai
Gardner Vickers
A. Sophie Blee-Goldman
Jason Gustafson
Bruno Cadonna
ArunParthiban-ST
fml2
Tom Bentley
Luke Chen
abc863377
Ismael Juma
Rajini Sivaram
dengziming
Matthias J. Sax
Prithvi
Hamza Slama
Matthew Wong
Kowshik Prakasam
Stanislav Kozlovski
David Mao
Samuel Cantero
Liju John
David Jacot
Walker Carlson
Justine Olshan