Add check in `KafkaApis` that the inter broker protocol version is at least `KAFKA_0_11_0_IV0`, i.e., supporting transactions
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3103 from dguy/kafka-5128
- introduces a new thread state DEAD
- ignores DEAD threads when querying
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3140 from mjsax/kafka-5309-stores-not-queryable
The previous code did not handle this correctly if a batch was
compacted more than once.
Also add test case for duplicate check after log cleaning and
improve various comments.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3145 from hachikuji/minor-improve-base-sequence-docs
The basic idea is that exactly three collections, ie. `pendingRequests`, `newPartitionsToBeAddedToTransaction`, and `partitionsInTransaction` are accessed from the context of application threads. The first two are modified from the application threads, and the last is read from those threads.
So to make the `TransactionManager` truly thread safe, we have to ensure that all accesses to these three members are done in a synchronized block. I inspected the code, and I believe this patch puts the synchronization in all the correct places.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3132 from apurvam/KAFKA-5147-transaction-manager-synchronization-fixes
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3137 from rajinisivaram/KAFKA-5320
remove transactions that have not been updated for at least `transactional.id.expiration.ms`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Apurva Mehta, Guozhang Wang
Closes#3101 from dguy/kafka-5279
For consumers with manual partition assignment, await metadata when there are no ready nodes to avoid busy polling.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3124 from rajinisivaram/KAFKA-5263
This call to isCompletedExceptionally introduced a race condition
because the future might not have been completed. assertFutureError
checks that the exception is present and of the correct type in any
case, so the call was not necessary.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3139 from cmccabe/fix-test-deleteacls
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3123 from hachikuji/KAFKA-4935
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3133 from hachikuji/minor-replica-manager-append-refactor
A follow up RP to fix [issue](2cd0b87bc8 (commitcomment-22200864))
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang
Closes#3109 from jeyhunkarimov/KAFKA-4144-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jun Rao <junrao@gmail.com>
Closes#3075 from hachikuji/KAFKA-5259-FIXED
Clarify the consumer group command help message around `zookeeper`, `bootstrap-server`, and `new-consumer` options.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2046 from vahidhashemian/minor/improve_consumer_group_command_doc
Before this patch the consumer would return the cached offsets for partitions in its current assignment. This worked when all the offset commits went through the consumer.
With KIP-98, offsets can be committed transactionally through the producer. This means that relying on cached positions in the consumer returns incorrect information: since commits go through the producer, the cache is never updated.
Hence we need to update the `KafkaConsumer.committed` method to always lookup the server for the last committed offset to ensure it gets the correct information every time.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#3119 from apurvam/KAFKA-5273-kafkaconsumer-committed-should-always-hit-server
**JIRA ticket:** [KAFKA-5081 two versions of jackson-annotations-xxx.jar in distribution tgz](https://issues.apache.org/jira/browse/KAFKA-5081)
**Solutions:**
1. accept this merge request **_OR_**
2. upgrade jackson libraries to version **_2.9.x_** (currently available as a pre-release only)
**Related jackson issue:** [Add explicit \`jackson-annotations\` dependency version for \`jackson-databind\`](https://github.com/FasterXML/jackson-databind/issues/1545)
**Note:** previous (equivalent) merge request #2900 ended up deep in the sand with swarm of messages due to flaky test, so I opted to close it and to open this one.
ijuma: FYI
Author: dejan2609 <dejan2609@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3116 from dejan2609/KAFKA-5081
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Onur Karaman <okaraman@linkedin.com>
Closes#2983 from ijuma/kafka-5135-controller-health-metrics-kip-143
This ticket is all about ControllerContext initialization and teardown. The key points are:
1. we should teardown ControllerContext during resignation instead of waiting on election to fix it up. A heapdump shows that the former controller keeps pretty much all of its ControllerContext state laying around.
2. we don't properly teardown/reset ControllerContext.partitionsBeingReassigned. This can cause problems when the former controller becomes re-elected as controller at a later point in time.
Suppose a partition assignment is initially R0. Now suppose a reassignment R1 gets stuck during controller C0 and an admin tries to "undo" R1 (by deleting /admin/partitions_reassigned, deleting /controller, and submitting another reassignment specifying R0). The new controller C1 may succeed with R0. If the controller moves back to C0, it will then reattempt R1 even though that partition reassignment has been cleared from zookeeper prior to shifting the controller back to C0. This results in the actual partition reassignment in zookeeper being unexpectedly changed back to R1.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3122 from onurkaraman/KAFKA-5310
Two major changes plus one minor change:
0. change stateLock to a read-write lock.
1. Put the check of "isCoordinator" and "coordinatorLoading" together with the return of the metadata, under one read lock block, since otherwise we can get incorrect behavior if there is a change in the metadata cache after the check but before the accessing of the metadata.
2. Grab the read lock right before trying to append to local txn log, and until the local append returns; this is to avoid the scenario that the epoch has actually changed when we are appending to local log (e.g. emigration followed by immigration).
3. only watch on txnId instead of txnId and txnPartitionId in the txn marker purgatory, and disable reaper thread, as we can now safely clear all the delayed operations by traversing the marker queues.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson, Jun Rao
Closes#3082 from guozhangwang/K5231-read-write-lock
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang
Closes#3104 from mjsax/minor-improve-embedded-kafka-cluster
`TransactionCoordinatorIntegrationTest` is not covering anything that isn't already covered by the more complete `TransactionsTest`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3128 from dguy/minor-remove-test
With this patch, offset commits are always materialized according to the order of the commit records in the offsets topic.
Before this patch, transactional offset commits were materialized in transaction order. However, the log cleaner will always preserve the record with the greatest offset. This meant that if there was a mix of offset commits from a consumer and a transactional producer, then it we would switch from transactional order to offset order after cleaning, resulting in an inconsistent state.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3108 from apurvam/KAFKA-5247-materialize-committed-offsets-in-offset-order
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3118 from hachikuji/disallow-transactional-idempotent-downconversion
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3121 from ewencp/hotfix-vagrant-provisioning
... plus some minor cleanup
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3092 from vahidhashemian/KAFKA-5277
`shutdownIdleFetcherThreads()` can throw InterruptedException
for example.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3096 from ijuma/kafka-5289-stop-replica-should-not-send-two-responses
If props is null, use POLICY_NOPLAINTEXT default value: false
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3105 from mimaison/KAFKA-5294
- add broker compatibility system tests
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
## Fixes race condition in TestVerifiableProducer sanity test:
The test starts a producer, waits for at least 5 acks, and then
logs in to the worker to grep for the producer process to figure
out what version it is running.
The problem was that the producer was set up to produce 1000 messages
at a rate of 1000 msgs/s and then exit. This means it will have a
typical runtime slightly above 1 second.
Logging in to the vagrant instance might take longer than that thus
resulting in the process grep to fail, failing the test.
This commit doesn't really fix the issue - a proper fix would be to tell
the producer to stick around until explicitly killed - but it increases
the chances of the test passing, at the expense of a slightly longer
runtime.
## Improves error reporting when is_version() fails
Author: Magnus Edenhill <magnus@edenhill.se>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2765 from edenhill/trunk
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2638 from becketqin/KAFKA-3995
Update upgrade.html
Raising this now, as KIP-118 is pulled from release as such submitting this without java 8 changes.
As per remaining review comment from https://github.com/apache/kafka/pull/2772, updating the upgrade notes.
Author: Michael André Pearce <michael.andre.pearce@me.com>
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2991 from michaelandrepearce/KIP-82
We should retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when receiving an UNKNOWN_TOPIC_OR_PARTITION error.
As described in the JIRA: It turns out that the `UNKNOWN_TOPIC_OR_PARTITION` is returned from the request handler in KafkaAPis for the AddPartitionsToTxn and the TxnOffsetCommitRequest when the broker's metadata doesn't contain one or more partitions in the request. This can happen for instance when the broker is bounced and has not received the cluster metadata yet.
We should retry in these cases, as this is the model followed by the consumer when committing offsets, and by the producer with a ProduceRequest.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3094 from apurvam/KAFKA-5269-handle-unknown-topic-partition-in-transaction-manager
Summary:
- add `reconnect.backoff.max.ms` common client configuration parameter
- if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an exponential backoff policy
- apply +/- 20% random jitter to smooth cluster reconnects
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1523 from dpkp/exp_backoff
Implements range scan for keys in windowed and session stores
Modifies caching session and windowed stores to use segmented cache keys.
Cache keys are internally prefixed with their segment id to ensure key ordering in the cache matches the ordering in the underlying store for keys spread across multiple segments.
This should also result in fewer cache keys getting scanned for queries spanning only some segments.
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#3027 from xvrl/windowstore-range-scan
- Avoid unnecessary inner methods
- Remove redundant parameter in `sendResponseExemptThrottle`
- Go through `sendResponseExemptThrottle` for produce requests with acks=0
- Tighten how we handle cases where there’s no response
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3087 from ijuma/kafka-apis-improvements
The backing store for offsets, status, and configs now attempts to use the new AdminClient to look up the internal topics and create them if they don’t yet exist. If the necessary APIs are not available in the connected broker, the stores fall back to the old behavior of relying upon auto-created topics. Kafka Connect requires a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with all versions since 0.10.0.0.
All three of Connect’s internal topics are created as compacted topics, and new distributed worker configuration properties control the replication factor for all three topics and the number of partitions for the offsets and status topics; the config topic requires a single partition and does not allow it to be set via configuration. All of these new configuration properties have sensible defaults, meaning users can upgrade without having to change any of the existing configurations. In most situations, existing Connect deployments will have already created the storage topics prior to upgrading.
The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes than 3 will receive an error unless they explicitly set the replication factor for the three internal topics. This is actually desired behavior, since it signals the users that they should be aware they are not using sufficient replication for production use.
The integration tests use a cluster with a single broker, so they were changed to explicitly specify a replication factor of 1 and a single partition.
The `KafkaAdminClientTest` was refactored to extract a utility for setting up a `KafkaAdminClient` with a `MockClient` for unit tests.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2984 from rhauch/kafka-4667
Perform down conversion after throttling to avoid retaining
messages in memory during throttling since this could result
in OOM. Also update bytesOut metrics after throttling.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3068 from rajinisivaram/KAFKA-5250
Use IBM Kerberos module for SASL tests if running on IBM JDK
Developed with edoardocomar
Based on https://github.com/apache/kafka/pull/738 by rajinisivaram
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Edoardo Comar <ecomar@uk.ibm.com>
Closes#2878 from mimaison/KAFKA-3070
This is a rebase version of [PR#2973](https://github.com/apache/kafka/pull/2973).
guozhangwang , please review this updated PR.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3086 from umesh9794/mylocal
Today PartitionStateMachine and ReplicaStateMachine define and assert the
valid state transitions inline for each state. It's cleaner to move the
transition rules into ReplicaState/PartitionState and do the assertion at
the top of the handleStateChange.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3071 from onurkaraman/KAFKA-5258
Damian Guy
Matthias J. Sax
Jason Gustafson
Apurva Mehta
Rajini Sivaram
Colin P. Mccabe
Jeyhun Karimov
Vahid Hashemian
dejan2609
Ismael Juma
Onur Karaman
Guozhang Wang
Ewen Cheslack-Postava
Mickael Maison
Magnus Edenhill
Jiangjie Qin
Michael André Pearce
Dana Powers
Xavier Léauté
Randall Hauch
umesh chaudhary
Konstantine Karantasis