Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2638 from becketqin/KAFKA-3995
Update upgrade.html
Raising this now, as KIP-118 is pulled from release as such submitting this without java 8 changes.
As per remaining review comment from https://github.com/apache/kafka/pull/2772, updating the upgrade notes.
Author: Michael André Pearce <michael.andre.pearce@me.com>
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2991 from michaelandrepearce/KIP-82
We should retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when receiving an UNKNOWN_TOPIC_OR_PARTITION error.
As described in the JIRA: It turns out that the `UNKNOWN_TOPIC_OR_PARTITION` is returned from the request handler in KafkaAPis for the AddPartitionsToTxn and the TxnOffsetCommitRequest when the broker's metadata doesn't contain one or more partitions in the request. This can happen for instance when the broker is bounced and has not received the cluster metadata yet.
We should retry in these cases, as this is the model followed by the consumer when committing offsets, and by the producer with a ProduceRequest.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3094 from apurvam/KAFKA-5269-handle-unknown-topic-partition-in-transaction-manager
Summary:
- add `reconnect.backoff.max.ms` common client configuration parameter
- if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an exponential backoff policy
- apply +/- 20% random jitter to smooth cluster reconnects
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1523 from dpkp/exp_backoff
Implements range scan for keys in windowed and session stores
Modifies caching session and windowed stores to use segmented cache keys.
Cache keys are internally prefixed with their segment id to ensure key ordering in the cache matches the ordering in the underlying store for keys spread across multiple segments.
This should also result in fewer cache keys getting scanned for queries spanning only some segments.
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#3027 from xvrl/windowstore-range-scan
- Avoid unnecessary inner methods
- Remove redundant parameter in `sendResponseExemptThrottle`
- Go through `sendResponseExemptThrottle` for produce requests with acks=0
- Tighten how we handle cases where there’s no response
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3087 from ijuma/kafka-apis-improvements
The backing store for offsets, status, and configs now attempts to use the new AdminClient to look up the internal topics and create them if they don’t yet exist. If the necessary APIs are not available in the connected broker, the stores fall back to the old behavior of relying upon auto-created topics. Kafka Connect requires a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with all versions since 0.10.0.0.
All three of Connect’s internal topics are created as compacted topics, and new distributed worker configuration properties control the replication factor for all three topics and the number of partitions for the offsets and status topics; the config topic requires a single partition and does not allow it to be set via configuration. All of these new configuration properties have sensible defaults, meaning users can upgrade without having to change any of the existing configurations. In most situations, existing Connect deployments will have already created the storage topics prior to upgrading.
The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes than 3 will receive an error unless they explicitly set the replication factor for the three internal topics. This is actually desired behavior, since it signals the users that they should be aware they are not using sufficient replication for production use.
The integration tests use a cluster with a single broker, so they were changed to explicitly specify a replication factor of 1 and a single partition.
The `KafkaAdminClientTest` was refactored to extract a utility for setting up a `KafkaAdminClient` with a `MockClient` for unit tests.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2984 from rhauch/kafka-4667
Perform down conversion after throttling to avoid retaining
messages in memory during throttling since this could result
in OOM. Also update bytesOut metrics after throttling.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3068 from rajinisivaram/KAFKA-5250
Use IBM Kerberos module for SASL tests if running on IBM JDK
Developed with edoardocomar
Based on https://github.com/apache/kafka/pull/738 by rajinisivaram
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Edoardo Comar <ecomar@uk.ibm.com>
Closes#2878 from mimaison/KAFKA-3070
This is a rebase version of [PR#2973](https://github.com/apache/kafka/pull/2973).
guozhangwang , please review this updated PR.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3086 from umesh9794/mylocal
Today PartitionStateMachine and ReplicaStateMachine define and assert the
valid state transitions inline for each state. It's cleaner to move the
transition rules into ReplicaState/PartitionState and do the assertion at
the top of the handleStateChange.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3071 from onurkaraman/KAFKA-5258
Don't produce messages on a separate thread continuosly. Just produce one of each value and stop.
Close the producer once finished.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3080 from dguy/qs-test
mjsax dguy guozhangwang Could you please review the changes.
Author: Bharat Viswanadham <bharatv@us.ibm.com>
Reviewers: Matthias J. Sax, Damian Guy, Guozhang Wang
Closes#3073 from bharatviswa504/KAFKA-5210
This PR implements a new partition assignment strategy called "sticky", and it's purpose is to balance partitions across consumers in a way that minimizes moving partitions around, or, in other words, preserves existing partition assignments as much as possible.
This patch is co-authored with rajinisivaram and edoardocomar.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1020 from vahidhashemian/KAFKA-2273
Includes server-side code, protocol and AdminClient.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2941 from cmccabe/KAFKA-3266
…ffsetForLeaderEpoch request
Author: Jun Rao <junrao@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ben Stopford <benstopford@gmail.com>
Closes#3074 from junrao/kafka-5036
Author: Apurva Mehta <apurva@confluent.io>
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2994 from apurvam/KAFKA-5188-exactly-once-integration-tests
Author: Jorge Quilcate <quilcate.jorge@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2624 from jeqo/feature/rewind-consumer-group-offset
I remove `KeyValuePrinter` and `KStreamForeach` two class, then implements them by `KStreamPeek`.
So, now `KStreamPeek` can do `KeyValuePrinter` and `KStreamForeach` job.
Author: jameschien <jameschien@staff.ruten.com.tw>
Author: JamesChien <jedichien@users.noreply.github.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2955 from jedichien/trunk
Reduce rebalance and session timeouts for join requests to trigger throttling in the request quota test.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3057 from rajinisivaram/KAFKA-5182-quotatest
Add a public create API that takes a Properties instance.
Make the constructors for TopicDescription, TopicListing
and TopicPartitionInfo public to enable AdminClient
users to write better tests.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3070 from cmccabe/publicapi
Mirrormaker was updated to default to the new consumer in 3db752a565071c78e4b11eaafa739844fa785b04
Old consumer calls the param `auto.commit.enable`, new consumer calls it `enable.auto.commit`. So I updated the docstring to use the new consumer name for the param.
Author: Jeff Widman <jeff@jeffwidman.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2393 from jeffwidman/patch-1
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Apurva Metha, Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2945 from mjsax/kafka-4923-add-eos-to-streams
Addressed the below review comment from #PR #2998 from mjsax
I am wondering if it would be better, to "embed" the dry-run into the actual code and branch on each place. Otherwise, if things get changed, we could easily introduce bugs (ie, dry run show something different than what the actual reset code does.
We could introduce methods like mabyeSeekToBeginning() that either does the seek or only prints to stdout. This would ensure that the main logic is used to "feed" into dry-run and we don't have code duplication.
WDYT?
Author: Bharat Viswanadham <bharatv@us.ibm.com>
Reviewers: Matthias J. Sax, Damian Guy, Eno Thereska, Guozhang Wang
Closes#3005 from bharatviswa504/KAFKA-5166
Ensure checkpointable offsets for GlobalKTables are always written on close.
Author: Tommy Becker <tobecker@tivo.com>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3054 from twbecker/KAFKA-5241
Add ACL checks for Transactional APIs
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2979 from dguy/kafka-5129
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3013 from cmccabe/KAFKA-5215
The transient failure came from the controller processing the preferred replica leader election
before the restarted broker (the preferred replica leader) has joined isr, causing preferred
replica leader election to fail and for the final zookeeper state validation to fail.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3067 from onurkaraman/KAFKA-5175
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3063 from guozhangwang/KMinor-more-logging-in-fetcher
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3064 from hachikuji/KAFKA-5252
Also disable a couple of tests that were passing incorrectly until KAFKA-3096 is fixed.
The bug was for the following case:
`leader.isDefined && oldLeaderOpt.isEmpty && newLeaderOpt.isDefined && newLeaderOpt.get != leader.get`
We would consider it a successful election even though the new leader was not the expected leader.
I also changed the result type as we never return `None` (we throw an exception instead).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3031 from ijuma/fix-wait-until-leader-is-elected
This removes the need to force object initialisation via hacks to register
the relevant Yammer metrics during start-up.
It also works around issues caused by tests that delete JVM-wide singleton
metrics (like `MetricsDuringTopicCreationDeletionTest`). Without this
change, they would never be registered again. After this change, they will
be registered again during KafkaServer start-up.
It would be even better not to rely on JVM side singleton metrics (like we do
for Kafka Metrics), but that's a bigger change that should be considered
separately.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3059 from ijuma/kafka-5244-broker-static-stats-and-controller-stats-as-classes
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3058 from hachikuji/KAFKA-5248
CachingSessionStore wasn't properly using the default keySerde if no Serde was supplied. I saw the below error in the logs for one of my test cases.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2963 from KyleWinkelman/CachingSessionStore-fix-keySerde-use
Ewen Cheslack-Postava
Jiangjie Qin
Michael André Pearce
Apurva Mehta
Matthias J. Sax
Dana Powers
Ismael Juma
Xavier Léauté
Randall Hauch
Rajini Sivaram
Mickael Maison
umesh chaudhary
Jason Gustafson
Konstantine Karantasis
Onur Karaman
Damian Guy
Bharat Viswanadham
Vahid Hashemian
Colin P. Mccabe
Jun Rao
Guozhang Wang
Jorge Quilcate
James Chien
Jeff Widman
Tommy Becker
Kyle Winkelman