Stores where logging is disabled where never consumed as the partitions were paused, but never resumed.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4002 from dguy/restore
Without this patch, if the replica's log was somehow truncated before
the leader's, it is possible for the replica fetcher thread to
continuously throw an OutOfOrderSequenceException because the
incoming sequence would be non-zero and there is no local state.
This patch changes the behavior so that the replica state is updated to
the leader's state if there was no local state for the producer at the
time of the append.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#4004 from apurvam/KAFKA-6003-handle-unknown-producer-on-replica
- fixes examples with regard to new API
- fixes `Topology#addGlobalStore` parameters
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#4003 from mjsax/minor-deprecated
For record conversion tests, check time >=0 since conversion times may be too small to be measured accurately. Since default value is -1, the test is still useful. Also increase message size in SslTransportLayerTest#testNetworkThreadTimeRecorded to avoid failures when processing time is too small.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4018 from rajinisivaram/KAFKA-6010-MemoryRecordsBuilderTest
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#4006 from mjsax/minor-javadoc-improvments-for-stores
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3984 from dguy/quickstart-update
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3956 from rajinisivaram/KAFKA-5970-delayedproduce-deadlock
Author: Joel Hamill <joel@Joel-Hamill-Confluent.local>
Author: Joel Hamill <11722533+joel-hamill@users.noreply.github.com>
Reviewers: Derrick Or <derrickor@gmail.com>, Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3862 from joel-hamill/joel-hamill/streams-dev-guide
A new mechanism was added recently to the Metrics framework to make it easier to generate the documentation. It uses a registry with a MetricsNameTemplate for each metric, and then those templates are used when creating the actual metrics. The metrics framework provides utilities that can generate the HTML documentation from the registry of templates.
This change moves the recently-added Connect metrics over to use these templates and to then generate the metric documentation for Connect.
This PR is based upon #3975 and can be rebased once that has been merged.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3987 from rhauch/kafka-5990
- Improve tests and javadoc (including expected exceptions)
- Return correct authorization error if no describe topic
permission
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3937 from tombentley/KAFKA-5856-AdminClient.createPartitions-follow-up
- Simplify LogCleaner.cleanSegments and add comment regarding thread
unsafe usage of `LogSegment.append`. This was a result of investigating
KAFKA-4972.
- Fix compiler warnings (in some cases use the fully qualified name as a
workaround for deprecation warnings in import statements).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4016 from ijuma/simplify-log-cleaner-and-fix-warnings
package name: org.apache.kafka.streams.state.internals
Minor change to approximateNumEntries() method in CompositeReadOnlyKeyValueStore class.
long total = 0;
for (ReadOnlyKeyValueStore<K, V> store : stores) {
total += store.approximateNumEntries();
}
return total < 0 ? Long.MAX_VALUE : total;
The check for negative value seems to account for wrapping. However, wrapping can happen within the for loop. So the check should be performed inside the loop.
Author: siva santhalingam <ssanthalingam@netskope.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3988 from shivsantham/trunk
Since we removed the unused `TRACE` option from `SecurityProtocol`, it now seems safer to expose it from `AuthenticationContext`. Additionally this patch exposes javadocs under security.auth and relocates the `Login` and `AuthCallbackHandler` to a non-public package.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3863 from hachikuji/use-security-protocol-in-auth-context
This simplifies running a console consumer as part of a custom group. Note that group id can be provided via only one of the three possible means: directly using `--group ` option (as part of this PR), as property via `--consumer-property` option, or inside a config file via `--consumer.config` option.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2150 from vahidhashemian/KAFKA-4416
- Upgrade Gradle to 4.2.1, which handles Azul Zulu 9's version
correctly.
- Add tests to our Java version handling code
- Refactor the code to make it possible to add tests
- Rename `isIBMJdk` method to use consistent naming
convention.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4007 from ijuma/java-9-version-handling-improvements
Added Connect metrics specific to source tasks, and builds upon #3864 and #3911 that have already been merged into `trunk`, and #3959 that has yet to be merged.
I'll rebase this PR when the latter is merged.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3975 from rhauch/kafka-5902
This PR improves the output format of DumpLogSegments when the `--offset-decoder` option is used for consuming `__consumer_offsets`, especially when it comes to group metadata.
An example of the partial output with existing formatting:
```
key: metadata::console-consumer-40190 payload: consumer:range:1:{consumer-1-20240b92-fbf4-44d5-bf8c-66b6d70c9948=[foo-0]}
```
An example of the same output with suggested formatting:
```
key: {"metadata":"console-consumer-40190"} payload: {"protocolType":"consumer","protocol":"range","generationId":1,"assignment":"{consumer-1-20240b92-fbf4-44d5-bf8c-66b6d70c9948=[foo-0]}"}
```
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1937 from vahidhashemian/KAFKA-4108
This just adds `<p>`/`</p>` to paragraphs in Kafak Connect docs.
ewencp are you a good person to review this?
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4008 from tombentley/MINOR-connect-docs-paragraph
- add some missing annotations for deprecated methods
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4005 from mjsax/minor-fix-javadoc-warnings
We log a warning instead, which is what we also do if the partition
hasn't been created yet.
A few other improvements:
- Return updated high watermark if fetch is returned immediately.
This seems to be more intuitive and is consistent with the case
where the fetch request is served from the purgatory.
- Centralise offline partition handling
- Remove unnecessary `tryCompleteDelayedProduce` that would
already have been done by the called method
- A few other minor clean-ups
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3954 from ijuma/kafka-5758-dont-fail-fetch-request-if-replica-is-not-follower
Author: Matthias J. Sax <matthias@confluent.io>
Author: Bharat Viswanadham <bharatv@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3970 from mjsax/kafka-5225-streams-resetter-properties
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3971 from dguy/kip-182-docs
And remove the deprecated `ConsumerOffsetChecker` example.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3405 from vahidhashemian/KAFKA-5495
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3994 from bbejeck/KAFKA-5985_document_need_to_close_iterators
This is less likely to break custom metric reporters and since the method
is deprecated, people will be warned about this potential issue.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Xavier Léauté <xavier@confluent.io>
Closes#3996 from ijuma/avoid-exception-in-measurable-value
Replaces the static `RequestMetrics` object with a class so that metrics
are created and removed during broker startup and shutdown to avoid metrics
tests being affected by metrics left behind by previous tests.
Also reinstates `kafka.api.MetricsTest` which was failing frequently earlier
due to tests removing the static request metrics.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3991 from rajinisivaram/KAFKA-5968
When logging is disabled and there are state stores the task never transitions from restoring to running. This is because we only ever check if the task has state stores and return false on initialization if it does. The check should be if we have changelog partitions, i.e., we need to restore.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, tedyu <yuzhihong@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3983 from dguy/restore-test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3978 from bbejeck/KAFKA-5932_no_fetch_previous_when_no_old_values_returned
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Closes#3943 from dguy/count-materialized
- compare KAFKA-5958
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3986 from mjsax/kafka-5949-exceptions-user-callbacks-KIP-161-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3948 from mjsax/kafka-4593-illegal-state-exception-in-restore
It should be the number of records instead of the
number of batches.
A few additional clean-ups:
- Minor renames
- Removed unused variable
- Some test fixes
- Ignore a flaky test
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, tedyu <yuzhihong@gmail.com>
Closes#3989 from ijuma/kafka-5746-health-metrics-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3942 from hachikuji/KAFKA-5957
Adds new metrics to support health checks:
1. Error rates for each request type, per-error code
2. Request size and temporary memory size
3. Message conversion rate and time
4. Successful and failed authentication rates
5. ZooKeeper latency and status
6. Client version
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3705 from rajinisivaram/KAFKA-5746-new-metrics
To check ordering, we augment the existing transactions test to read and write from topics with one partition. Since we are writing monotonically increasing numbers, the topics should always be sorted, making it very easy to check for out of order messages.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3969 from apurvam/KAFKA-5888-system-test-which-check-ordering
Added Connect metrics specific to source tasks, and builds upon #3864 and #3911 that have already been merged into `trunk`.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3959 from rhauch/kafka-5901
Rajini Sivaram
Damian Guy
Apurva Mehta
Dong Lin
Guozhang Wang
Matthias J. Sax
Jakub Scholz
Joel Hamill
Randall Hauch
Tom Bentley
Ismael Juma
siva santhalingam
Jason Gustafson
Vahid Hashemian
Manikumar Reddy
Bill Bejeck