A couple of root causes of this flaky test is fixed:
1. The MockTime was incorrectly used across multiple test methods within the class, as a class rule. Instead we set it on each test case; also remove the scala MockTime dependency.
2. List topics may not contain the deleted topics while their ZK paths are yet to be deleted; so the delete-check-recreate pattern may fail to successfully recreate the topic at all. Change the checking to read from zk path directly instead.
Another minor fix is to remove the misleading wait condition error message as the accumData is always empty.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#4095 from guozhangwang/KMinor-reset-integration-test
The methods resetReconnectBackoff and updateReconnectBackoff in ClusterConnectionStates both take an instance of a private inner class as parameter and thus cannot be called from outside the class anyway.
Author: Soenke Liebau <soenke.liebau@opencore.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4114 from soenkeliebau/MINOR_private
The idempotent producer doesn't change that setting any more and the
accepted range has changed.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#4097 from ijuma/fix-javadoc-wrt-max-in-flight-for-idempotent
Currently, in branches _trunk_, _0.11.0_, and _1.0_ the property **max.in.flight.requests.per.connection** is incorrectly misspelled as _max.inflight.requests.per.connection_
harshach ijuma guozhangwang can you please review. Thank you.
Author: Hugo Louro <hmclouro@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4094 from hmcl/trunk_MINOR_Doc_InflightProp
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Tom Bentley <tbentley@redhat.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#3874 from lindong28/KAFKA-5163
Author: Manikumar Reddy <manikumar.reddy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3814 from omkreddy/KAFKA-4504
Author: Jacek Laskowski <jacek@japila.pl>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#4038 from jaceklaskowski/KAFKA-4818-isolationLevel
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4064 from mjsax/minor-add-state-serdes-test
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#4035 from hachikuji/KAFKA-5547-followup and squashes the following commits:
f6b04ce1a [Jason Gustafson] Add a couple missed common fields
d3473b14d [Jason Gustafson] Fix compilation errors and a few warnings
58a0ae695 [Jason Gustafson] MINOR: Avoid some unnecessary collection copies in KafkaApis
With these changes, we are ensuring that the partitions being reassigned are from non-zero offsets. We also ensure that every message in the log has producerId and sequence number.
This means that it successfully reproduces https://issues.apache.org/jira/browse/KAFKA-6003.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#4029 from apurvam/KAFKA-6016-add-idempotent-producer-to-reassign-partitions
Author: bartdevylder <bartdevylder@gmail.com>
Author: Bart De Vylder <bartdevylder@gmail.com>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#4044 from bartdevylder/KAFKA-6026
Author: Jason Gustafson <jason@confluent.io>
Reviewers: tedyu <yuzhihong@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#4047 from hachikuji/factor-out-some-common-fields
…up rebalances in progress
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3506 from cmccabe/KAFKA-5565
It is possible for batches with sequence numbers to be in the `deque` while at the same time the in flight batches in the `TransactionManager` are removed due to a producerId reset.
In this case, when the batches in the `deque` are drained, we will get a `NullPointerException` in the background thread due to this line:
```java
if (first.hasSequence() && first.baseSequence() != transactionManager.nextBatchBySequence(first.topicPartition).baseSequence())
```
Particularly, `transactionManager.nextBatchBySequence` will return null, because there no inflight batches being tracked.
In this patch, we simply allow the batches in the `deque` to be drained if there are no in flight batches being tracked in the TransactionManager. If they succeed, well and good. If the responses come back with an error, the batces will be ultimately failed in the producer with an `OutOfOrderSequenceException` when the response comes back.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#4022 from apurvam/KAFKA-6015-npe-in-record-accumulator
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3819 from guozhangwang/KMinor-rocksDB-573
For record conversion tests, check time >=0 since conversion times may be too small to be measured accurately. Since default value is -1, the test is still useful. Also increase message size in SslTransportLayerTest#testNetworkThreadTimeRecorded to avoid failures when processing time is too small.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4018 from rajinisivaram/KAFKA-6010-MemoryRecordsBuilderTest
- Improve tests and javadoc (including expected exceptions)
- Return correct authorization error if no describe topic
permission
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3937 from tombentley/KAFKA-5856-AdminClient.createPartitions-follow-up
- Simplify LogCleaner.cleanSegments and add comment regarding thread
unsafe usage of `LogSegment.append`. This was a result of investigating
KAFKA-4972.
- Fix compiler warnings (in some cases use the fully qualified name as a
workaround for deprecation warnings in import statements).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4016 from ijuma/simplify-log-cleaner-and-fix-warnings
Since we removed the unused `TRACE` option from `SecurityProtocol`, it now seems safer to expose it from `AuthenticationContext`. Additionally this patch exposes javadocs under security.auth and relocates the `Login` and `AuthCallbackHandler` to a non-public package.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3863 from hachikuji/use-security-protocol-in-auth-context
- Upgrade Gradle to 4.2.1, which handles Azul Zulu 9's version
correctly.
- Add tests to our Java version handling code
- Refactor the code to make it possible to add tests
- Rename `isIBMJdk` method to use consistent naming
convention.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4007 from ijuma/java-9-version-handling-improvements
This is less likely to break custom metric reporters and since the method
is deprecated, people will be warned about this potential issue.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Xavier Léauté <xavier@confluent.io>
Closes#3996 from ijuma/avoid-exception-in-measurable-value
It should be the number of records instead of the
number of batches.
A few additional clean-ups:
- Minor renames
- Removed unused variable
- Some test fixes
- Ignore a flaky test
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, tedyu <yuzhihong@gmail.com>
Closes#3989 from ijuma/kafka-5746-health-metrics-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3942 from hachikuji/KAFKA-5957
Adds new metrics to support health checks:
1. Error rates for each request type, per-error code
2. Request size and temporary memory size
3. Message conversion rate and time
4. Successful and failed authentication rates
5. ZooKeeper latency and status
6. Client version
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3705 from rajinisivaram/KAFKA-5746-new-metrics
Added Connect metrics specific to source tasks, and builds upon #3864 and #3911 that have already been merged into `trunk`.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3959 from rhauch/kafka-5901
The `metric.reporters` description in the documentation says to implement the `MetricReporter` class, but the actual class is `MetricsReporter`. [MetricsReporter.java](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/MetricsReporter.java)
The configurations documentation is also inconsistent as some references to classes do not have the full package name while others do.
ijuma
Author: Kevin Lu <kelu@paypal.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3875 from KevinLiLu/trunk
Instead of having the metrics registry and the
org.apache.kafka.common.metrics.Metrics object be separate things,
have the metrics registry hold a copy of the Metrics object.
That way, all the metricInstance stuff is hidden, and we don't
have to make sure that the metrics registry and the Metrics
object are configured identicailly (with the same tags).
I personally think this looks a little better.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3799 from wushujames/producer_sender_metrics_docs_different
Added metrics that are common to both sink and source tasks.
Marked as "**WIP**" since this PR is built upon #3864, and will need to be rebased once that has been merged into `trunk`. However, I would still appreciate initial reviews since this PR is largely additive.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3911 from rhauch/kafka-5900
Make documentation consistent across methods and throw
IllegalStateException instead of IllegalArgumentException in
some cases.
Also include a couple of minor fixes in upgrade.html.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3781 from lindong28/minor-admin-client-comment
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3944 from hachikuji/KAFKA-5960
1. Propagate `SSLException` as `SslAuthenticationException` to enable clients to report these and avoid retries
2. Updates to `SslTransportLayer` to process bytes received even if end-of-stream
3. Some tidy up of authentication handling
4. Report exceptions in SaslClientAuthenticator as AuthenticationExceptions
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3918 from rajinisivaram/KAFKA-5920-SSL-handshake-failure
As mentioned in MappedByteBuffers' class documentation, its
implementation was inspired by Lucene's MMapDirectory:
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.6.1/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L315
Without this change, unmapping fails with the following message:
> java.lang.IllegalAccessError: class kafka.log.AbstractIndex (in unnamed module 0x45103d6b) cannot access class jdk.internal.ref.Cleaner (in module java.base) because module java.base does not export jdk.internal.ref to unnamed module 0x45103d6b
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3879 from ijuma/kafka-5915-unmap-mapped-buffers-java-9
These methods help users check for cases in which this metadata was not returned by the broker (e.g. in the case of acks=0 or a duplicate error when idempotence is enabled).
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3878 from apurvam/KAFKA-5913-add-record-metadata-not-available-exception
Simplified the condition in Sender#failBatch()
Added log in TransactionManager#updateLastAckedOffset()
Author: tedyu <yuzhihong@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3935 from tedyu/trunk
Developed with edoardocomar
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Edoardo Comar <ecomar@uk.ibm.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3906 from mimaison/KAFKA-5735
- Remove DelayedCreatePartitions to reduce code duplication
- Avoid unnecessary ZK calls (call it once per request instead
of once per topic, if possible)
- Simplify code
- A few minor clean-ups
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Tom Bentley <tbentley@redhat.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3930 from ijuma/kafka-5856-admin-client-creation-partitions
1. Raise AuthenticationException for authentication failures in admin client
2. Handle AuthenticationException as a fatal error for transactional producer
3. Add comments to authentication exceptions
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3928 from rajinisivaram/KAFKA-5947-auth-failure
Jeff Widman
Guozhang Wang
tedyu
Soenke Liebau
Rajini Sivaram
Ismael Juma
Hugo Louro
Dong Lin
Manikumar Reddy
Jason Gustafson
Jacek Laskowski
Matthias J. Sax
Ewen Cheslack-Postava
Apurva Mehta
bartdevylder
Colin P. Mccabe
Jakub Scholz
Tom Bentley
Randall Hauch
Kevin Lu
James Cheng
Mickael Maison