Add overloads of `count`, `reduce`, and `aggregate` that are `Materialized` to `KGroupedStream`.
Refactor common parts between `KGroupedStream` and `WindowedKStream`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3827 from dguy/kafka-5654
Fixes two issues with the JMH benchmark example:
* Trivial: The output should be in `ops/ms` for readability
reasons (it's in the millions of operations per second)
* Important: The benchmark is not actually measuring the
LRU Cache performance as most of the time in each run is
wasted on concatenating `key + counter` as well as
`value + counter`. Fixed by pre-generating 10k K-V pairs
(100x the cache capacity) and iterating over them. This
brings the performance up by a factor of more than 5 on
a standard 4 core i7 (`~6k/ms` before goes to `~35k/ms`).
Author: Armin Braun <me@obrown.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2903 from original-brownbear/fix-jmh-example
Also:
1. Fix WorkerTest to use the correct `Mock` annotations. `org.easymock.Mock`
is not supported by PowerMock 2.x.
2. Rename `powermock` to `powermockJunit4` in `dependencies.gradle` for
clarity.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3881 from ijuma/kafka-5884-powermock-java
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: James Cheng <jylcheng@yahoo.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3882 from rajinisivaram/MINOR-KAFKA-5738-metricstemplates
This PR utilizes `org.apache.kafka.common.utils.LogContext` for logging in `KafkaStreams`. hachikuji, ijuma please review this and let me know your thoughts.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3727 from umesh9794/KAFKA-5754
This test is super flaky in the PR builder. https://issues.apache.org/jira/browse/KAFKA-5792 tracks the fix.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3877 from apurvam/MINOR-disable-adminclient-timeout-test
The `NextIteratorFunction` in `CompositeReadOnlyWindowStore` was incorrectly using the `timeFrom` as the `timeTo`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3868 from dguy/window-store-range-scan
Here we introduce client and broker changes to support multiple inflight requests while still guaranteeing idempotence. Two major problems to be solved:
1. Sequence number management on the client when there are request failures. When a batch fails, future inflight batches will also fail with `OutOfOrderSequenceException`. This must be handled on the client with intelligent sequence reassignment. We must also deal with the fatal failure of some batch: the future batches must get different sequence numbers when the come back.
2. On the broker, when we have multiple inflights, we can get duplicates of multiple old batches. With this patch, we retain the record metadata for 5 older batches.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3743 from apurvam/KAFKA-5494-increase-max-in-flight-for-idempotent-producer
It adds complexity for no benefit since we don't use
it anywhere.
Also removed a few unused imports, variables and
default parameters.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>
Closes#3856 from ijuma/remove-security-protocol-trace
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3852 from hachikuji/minor-use-log-context-txn-manager
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3854 from vahidhashemian/minor/fix_log_context_message_format
- EasyMock 3.5 supports Java 9.
- Fixed issues in `testFailedSendRetryLogic` and
`testCreateConnectorAlreadyExists` exposed by new EasyMock
version. The former was passing `anyObject` to
`andReturn`, which doesn't make sense. This was leaving
behind a global `any` matcher, which caused a few issues in
the new version. Fixing this meant that the correlation ids had
to be updated to actually match. The latter was missing a
couple of expectations that the previous version of EasyMock
didn't catch.
- Removed unnecessary PowerMock dependency from 3 tests.
- Disabled remaining PowerMock tests when running with Java 9
until https://github.com/powermock/powermock/issues/783 is
in a release.
- Once we merge this PR, we can enable tests in the Java 9 builds
in Jenkins.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3845 from ijuma/kafka-4501-easymock-powermock-java-9
The contribution is my original work and I license the work to the project under the project's open source licence.
Author: lperry <lperry@simplemachines.com.au>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3843 from leigh-perry/trunk
I have decided to use the following approach to fixing this bug:
1) Since the Window Size in WindowedDeserializer was originally unknown, I have initialized
a field _windowSize_ and created a constructor to allow it to be instantiated
2) The default size for __windowSize__ is _Long.MAX_VALUE_. If that is the case, then the
deserialize method will return an Unlimited Window, or else will return Timed one.
3) Temperature Demo was modified to demonstrate how to use this new constructor, given
that the window size is known.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3745 from ConcurrencyPractitioner/trunk
Add overloads of `count`, `aggregate`, `reduce` using `Materialized` to `KGroupedTable`
deprecate other overloads
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3829 from dguy/kafka-5655
Add `join`, `leftJoin`, `outerJoin` overloads that use `Materialized` to `KTable`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3826 from dguy/kafka-5653
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3828 from bbejeck/MINOR_update_processor_topology_test_driver
Add `SerializedInternal` class and remove getters from `Serialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3825 from dguy/kafka-5817-follow-up
Create `ProducedInternal` and remove getters from `Produced`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3810 from dguy/kafka-5816-follow-up
1. Now instead of just generic `Exception` methods declare more concrete
exceptions throwing or don't declare any throwing at all, if not needed.
2. `SimpleBenchmark.run()` throws `RuntimeException`
3. `SimpleBenchmark.produce()` throws `IllegalArgumentException`
4. Expect `ProcessorStateException` in
`StandbyTaskTest.testUpdateNonPersistentStore()`
/cc enothereska
Author: Evgeny Veretennikov <evg.veretennikov@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3485 from evis/5531-throw-concrete-exceptions
`SelectorTest.testCloseConnectionInClosingState` creates a channel with some staging receives and moves time forward to expire the channel. To ensure that the channel will be expired on the next poll, the channel must be muted to avoid expiry time being updated if more data is available for read.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3823 from rajinisivaram/MINOR-SelectorTest-closingChannel
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3816 from dguy/consumed-ctor
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3817 from dguy/printed-ctor-protected
Add overloads of `filter`, `filterNot`, `mapValues` that take `Materialized` as a param to `KTable`. Deprecate overloads using `storeName` and `storeSupplier`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3807 from dguy/ktable-filter-map
Changes:
1. When an exception is encountered in any of the methods in `Processor` while processing a channel, log the exception and close the connection. Continue to process other channels.
2. Fixes KAFKA-5790: SocketServer.processNewResponses should not skip a response if exception is thrown.
3. For `IllegalStateException` and `IOException` in `poll()`, don't close the `Selector`. Log the exception and continue.
4. Close channel on any failed send in `Selector`.
5. When closing channel fails or is closed, leave channel state as-is, indicating the state in which the channel was moved to closing.
6. Add tests for various failure scenarios.
7. Fix timing issue in `SocketServerTest.testConnectionIdReuse` by waiting for new connections to be processed by the server.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3548 from rajinisivaram/KAFKA-5607
Part of KIP-182
- Add `Printed` class and `KStream#print(Printed)`
- deprecate all other `print` and `writeAsText` methods
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3768 from dguy/kafka-5652-printed
Add the `WindowedKStream` interface and implementation of methods that don't require `Materialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3809 from dguy/kgrouped-stream-windowed-by
There have been a few bug fix releases since
the previous update.
Author: Andras Beni <andrasbeni@cloudera.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3791 from andrasbeni/dependency-upgrade
Author: Erik Kringen <erik.kringen@icloud.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3582 from ErikKringen/trunk
As described in the JIRA ticket, when `listeners=PLAINTEXT://0.0.0.0:9092`
(note the 0.0.0.0 "bind all interfaces" IP address) and
`advertised.listeners` is not specified it defaults to `listeners`,
but it makes no sense to advertise 0.0.0.0 as it's not a routable IP
address.
This patch checks for a 0.0.0.0 host in `advertised.listeners`
(whether via default or not) and fails with a meaningful error if it's
found.
This contribution is my original work and I license the work to the
project under the project's open source license.
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3382 from tombentley/advertised.listeners
Added some tips for running a single test file, test class and/or test method on the documentation landing page about tests
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3577 from ppatierno/minor-tests-doc
StickyAssignor javadoc has a bunch of formatting issues which make it pretty hard to read:
http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/StickyAssignor.html
cc vahidhashemian
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3670 from mimaison/sticky_assignor_javadoc
It's implemented such that there is no overhead if request logging is
disabled.
Also:
- Reduce metrics computation duplication in `updateRequestMetrics`
- Change a couple of log calls to use string interpolation instead of `format`
- Fix a few compiler warnings related to unused imports and unused default
arguments.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Roger Hoover <roger.hoover@gmail.com>, Jason Gustafson <jason@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3801 from ijuma/log-response-in-request-log
Damian Guy
Armin Braun
Ismael Juma
Rajini Sivaram
umesh chaudhary
Apurva Mehta
Jason Gustafson
Vahid Hashemian
Jeff Widman
Guozhang Wang
lperry
Richard Yu
Bill Bejeck
Bryan Baugher
Andrey Dyachkov
Evgeny Veretennikov
Colin P. Mccabe
Xavier Léauté
dejan2609
Andras Beni
Erik Kringen
Tom Bentley
Paolo Patierno
Mickael Maison