- add some missing annotations for deprecated methods
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#4005 from mjsax/minor-fix-javadoc-warnings
We log a warning instead, which is what we also do if the partition
hasn't been created yet.
A few other improvements:
- Return updated high watermark if fetch is returned immediately.
This seems to be more intuitive and is consistent with the case
where the fetch request is served from the purgatory.
- Centralise offline partition handling
- Remove unnecessary `tryCompleteDelayedProduce` that would
already have been done by the called method
- A few other minor clean-ups
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3954 from ijuma/kafka-5758-dont-fail-fetch-request-if-replica-is-not-follower
Author: Matthias J. Sax <matthias@confluent.io>
Author: Bharat Viswanadham <bharatv@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3970 from mjsax/kafka-5225-streams-resetter-properties
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3971 from dguy/kip-182-docs
And remove the deprecated `ConsumerOffsetChecker` example.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3405 from vahidhashemian/KAFKA-5495
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3994 from bbejeck/KAFKA-5985_document_need_to_close_iterators
This is less likely to break custom metric reporters and since the method
is deprecated, people will be warned about this potential issue.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Xavier Léauté <xavier@confluent.io>
Closes#3996 from ijuma/avoid-exception-in-measurable-value
Replaces the static `RequestMetrics` object with a class so that metrics
are created and removed during broker startup and shutdown to avoid metrics
tests being affected by metrics left behind by previous tests.
Also reinstates `kafka.api.MetricsTest` which was failing frequently earlier
due to tests removing the static request metrics.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3991 from rajinisivaram/KAFKA-5968
When logging is disabled and there are state stores the task never transitions from restoring to running. This is because we only ever check if the task has state stores and return false on initialization if it does. The check should be if we have changelog partitions, i.e., we need to restore.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, tedyu <yuzhihong@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3983 from dguy/restore-test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3978 from bbejeck/KAFKA-5932_no_fetch_previous_when_no_old_values_returned
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Closes#3943 from dguy/count-materialized
- compare KAFKA-5958
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3986 from mjsax/kafka-5949-exceptions-user-callbacks-KIP-161-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3948 from mjsax/kafka-4593-illegal-state-exception-in-restore
It should be the number of records instead of the
number of batches.
A few additional clean-ups:
- Minor renames
- Removed unused variable
- Some test fixes
- Ignore a flaky test
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, tedyu <yuzhihong@gmail.com>
Closes#3989 from ijuma/kafka-5746-health-metrics-follow-up
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3942 from hachikuji/KAFKA-5957
Adds new metrics to support health checks:
1. Error rates for each request type, per-error code
2. Request size and temporary memory size
3. Message conversion rate and time
4. Successful and failed authentication rates
5. ZooKeeper latency and status
6. Client version
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3705 from rajinisivaram/KAFKA-5746-new-metrics
To check ordering, we augment the existing transactions test to read and write from topics with one partition. Since we are writing monotonically increasing numbers, the topics should always be sorted, making it very easy to check for out of order messages.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3969 from apurvam/KAFKA-5888-system-test-which-check-ordering
Added Connect metrics specific to source tasks, and builds upon #3864 and #3911 that have already been merged into `trunk`.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: tedyu <yuzhihong@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3959 from rhauch/kafka-5901
The `metric.reporters` description in the documentation says to implement the `MetricReporter` class, but the actual class is `MetricsReporter`. [MetricsReporter.java](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/MetricsReporter.java)
The configurations documentation is also inconsistent as some references to classes do not have the full package name while others do.
ijuma
Author: Kevin Lu <kelu@paypal.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3875 from KevinLiLu/trunk
It now records the correct size for `Send` and does more appropriate
logging if the connection is being closed or if no response is being
sent.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Denis Bolshakov, Ismael Juma <ismael@juma.me.uk>
Closes#3961 from huxihx/KAFKA-5976
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3979 from mjsax/kafka-5979-kip-120-regression
- catch user exception in user callback (TimestampExtractor, DeserializationHandler, StateRestoreListener) and wrap with StreamsException
Additional cleanup:
- rename globalRestoreListener to userRestoreListener
- remove unnecessary interface -> collapse SourceNodeRecordDeserializer and RecordDeserializer
- removed unused parameter loggingEnabled from ProcessorContext#register
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3939 from mjsax/kafka-5949-exceptions-user-callbacks
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3973 from bbejeck/KAFKA-5958_global_stores_access_state_restore_listener
Instead of having the metrics registry and the
org.apache.kafka.common.metrics.Metrics object be separate things,
have the metrics registry hold a copy of the Metrics object.
That way, all the metricInstance stuff is hidden, and we don't
have to make sure that the metrics registry and the Metrics
object are configured identicailly (with the same tags).
I personally think this looks a little better.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3799 from wushujames/producer_sender_metrics_docs_different
Added metrics that are common to both sink and source tasks.
Marked as "**WIP**" since this PR is built upon #3864, and will need to be rebased once that has been merged into `trunk`. However, I would still appreciate initial reviews since this PR is largely additive.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3911 from rhauch/kafka-5900
This depends on sun.misc.Signal and sun.misc.SignalHandler, which may be
removed in future releases. But along with sun.misc.Unsafe, these classes
are available in Java 9 (see JEP 260), so they are safe to use for now.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3668 from rajinisivaram/KAFKA-5679
This is the polished version.
1. The old merge() method in StreamsBuilder has been removed,
2. The merge() method in KStreamBuilder was changed so that it would use the single variable argument
rather than several variable arguments in the KStreamImpl implementation
3. The merge() method in KStream has been declared as final and tests have been added to test correctness.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3916 from ConcurrencyPractitioner/trunk
Make documentation consistent across methods and throw
IllegalStateException instead of IllegalArgumentException in
some cases.
Also include a couple of minor fixes in upgrade.html.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3781 from lindong28/minor-admin-client-comment
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3944 from hachikuji/KAFKA-5960
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3949 from mjsax/minor-update-streams-quickstart
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3036 from mimaison/KAFKA-3356
Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3950 from manjuapu/customer-logo-stream
Offset and partition are not converted from String to
long and int correctly.
Running the command line with --from-file option causes
the following exception:
java.lang.ClassCastException: java.lang.String cannot be
cast to java.lang.Integer
Reason: asInstanceOf used for the conversion.
Also, unit test is using --to-earliest and --from-file
together when executing the test. This is executing
--to-earliest option only and ignoring --from-file
option. Since the preparation part is also using
--to-earliest to create the file, this unit test
passes without testing --from-file option. Fixed
the unit test too.
Author: Erkan Unal <eunal@cisco.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3938 from eu657/eu657-patch-1
This PR is the first of several subtasks for [KAFKA-2376](https://issues.apache.org/jira/browse/KAFKA-2376) to add metrics to Connect worker processes. See that issue and [KIP-196 for details](https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework).
This PR adds metrics for each connector using Kafka’s existing `Metrics` framework. This is the first of several changes to add several groups of metrics, this change starts by adding a very simple `ConnectMetrics` object that is owned by each worker and that makes it easy to define multiple groups of metrics, called `ConnectMetricGroup` objects. Each metric group maps to a JMX MBean, and each metric within the group maps to an MBean attribute.
Future PRs will build upon this simple pattern to add metrics for source and sink tasks, workers, and worker rebalances.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewencp@confluent.io>
Closes#3864 from rhauch/kafka-5899
1. Propagate `SSLException` as `SslAuthenticationException` to enable clients to report these and avoid retries
2. Updates to `SslTransportLayer` to process bytes received even if end-of-stream
3. Some tidy up of authentication handling
4. Report exceptions in SaslClientAuthenticator as AuthenticationExceptions
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3918 from rajinisivaram/KAFKA-5920-SSL-handshake-failure
As mentioned in MappedByteBuffers' class documentation, its
implementation was inspired by Lucene's MMapDirectory:
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.6.1/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L315
Without this change, unmapping fails with the following message:
> java.lang.IllegalAccessError: class kafka.log.AbstractIndex (in unnamed module 0x45103d6b) cannot access class jdk.internal.ref.Cleaner (in module java.base) because module java.base does not export jdk.internal.ref to unnamed module 0x45103d6b
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3879 from ijuma/kafka-5915-unmap-mapped-buffers-java-9
The new overloads `StreamBuilder.table(String, Materialized)` and `StreamsBuilder.globalTable(String, Materialized)` need to set the serdes from `Materialized` on the internal `Consumed` instance that is created, otherwise the defaults will be used and may result in serialization errors
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3936 from dguy/table-materialized
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ted Yu <yuzhihong@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3913 from mjsax/kafka-5937-exceptions-processor-state-manager
Manikumar Reddy
Matthias J. Sax
Dong Lin
Ismael Juma
Damian Guy
Vahid Hashemian
Bill Bejeck
Rajini Sivaram
Jason Gustafson
Apurva Mehta
Randall Hauch
Kevin Lu
huxihx
Konstantine Karantasis
James Cheng
Manjula K
Richard Yu
Mickael Maison
Erkan Unal
Thibaud Chardonnens