This patch ensures that the consumer groupId and clientId are available in all log messages which makes debugging much easier when a single application has multiple consumer instances. To make this easier, I've added a new `LogContext` object which builds a log prefix similar to the broker-side `kafka.utils.Logging` mixin. Additionally this patch changes the log level for a couple minor cases:
- Consumer wakeup events are now logged at DEBUG instead of TRACE
- Heartbeat enabling/disabling is now logged at DEBUG instead of TRACE
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3676 from hachikuji/log-consumer-wakeups
Fix range queries in `CompositeReadOnlyWindowStore` and `CompositeReadOnlySessionStore` to fetch across all stores (was previously just looking in the first store)
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3685 from dguy/kafka-5668
This is to complete Bill's PR #3664 on KAFKA-5733, incorporating the suggestion in https://github.com/facebook/rocksdb/issues/2734.
Some minor changes: move `open = true` in `openDB`.
Author: Bill Bejeck <bill@confluent.io>
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3681 from guozhangwang/K5733-rocksdb-bulk-load
Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with.
This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3662 from rhauch/kafka-5731
The commit brings improved test coverage for StreamsKafkaClientTest.java
Author: Andrey Dyachkov <andrey.dyachkov@zalando.de>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3663 from adyach/kafka-4643
When there are broker delays that cause a response to take longer
than `connections.max.idle.ms`, connections may be closed by the
broker (as well as by the client) before the response is processed.
If the port is reused, broker may send the outstanding response to
a new connection with the reused port. The new connection will end
up with correlation id mismatch, requiring process restart. This
is also a security exposure since clients receive response
intended for the wrong connection.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3530 from rajinisivaram/KAFKA-5595
This implements two improvements for request handling prior to SASL authentication:
1. Only parse request types that are allowed prior to authentication.
2. Limit the maximum request size (the default is 100Mb).
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3558 from hachikuji/minor-restrict-presasl-request-parsing
refactor StateStoreSuppliers such that a `MeteredKeyValueStore` is the outermost store.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3592 from dguy/key-value-store-refactor
Also make "created" message more consistent across clients.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3658 from ijuma/admin-client-should-register-with-app-info-parser
0. Minor fixes on the existing examples to merge all on a single input topic; also do not use `common.utils.Exit` as it is for internal usage only.
1. Add the archetype project for the quickstart. Steps to try it out:
a. `mvn install` on the quickstart directory.
b. `mvn archetype:generate \
-DarchetypeGroupId=org.apache.kafka \
-DarchetypeArtifactId=streams-quickstart-java \
-DarchetypeVersion=1.0.0-SNAPSHOT \
-DgroupId=streams-quickstart \
-DartifactId=streams-quickstart \
-Dversion=0.1 \
-Dpackage=StreamsQuickstart \
-DinteractiveMode=false` at any directory to create the project.
c. build the streams jar with version `1.0.0-SNAPSHOT` to local maven repository with `./gradlew installAll`; `cd streams-quickstart; mvn clean package`
d. create the input / output topics, start the console producer and consumer.
e. start the program: `mvn exec:java -Dexec.mainClass=StreamsQuickstart.Pipe/LineSplit/WordCount`.
f. type data on console producer and observe data on console consumer.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>, Eno Thereska <eno.thereska@gmail.com>
Closes#3630 from guozhangwang/KMinor-streams-quickstart-tutorial
Extracted `TaskManager` to handle all task related activities.
Make `StandbyTaskCreator`, `TaskCreator`, and `RebalanceListener` static classes so they must define their dependencies and can be testing independently of `StreamThread`
Added interfaces between `StreamPartitionAssignor` & `StreamThread` to reduce coupling.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Eno Thereska <eno.thereska@gmail.com>
Closes#3624 from dguy/stream-thread-refactor
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3629 from bbejeck/docs-updates-for-kip-167
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3644 from bbejeck/KAFKA-5711_bulk_restore_should_handle_deletes
These handlers were previously used on the broker to handle uncaught exceptions, but now the broker users the new Java request objects exclusively.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3646 from hachikuji/remove-old-request-error-handlers
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.
It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
when "auto.offset.reset" property is specified on the command line but overridden by the code during startup. Currently the ConsoleConsumer silently overrides that setting, which can create confusing behavior.
Author: Soenke Liebau <soenke.liebau@opencore.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3566 from soenkeliebau/KAFKA-5629
Fixed a bug in the InMemoryKeyValueStore restoration where a key with a `null` value is written in to the map rather than being deleted.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3650 from dguy/kafka-5717
This ensures all logs have the connector/task ID, whether tasks are source or sink, and formats them consistently.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3639 from ewencp/standardize-connector-task-logging
When a Connect distributed worker starts up talking with broker versions 0.10.1.0 and later, it will use the AdminClient to look for the internal topics and attempt to create them if they are missing. Although the AdminClient was added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in 0.10.1.0 and later. This feature works as expected when Connect uses a broker version 0.10.1.0 or later.
However, when a Connect distributed worker starts up using a broker older than 0.10.1.0, the AdminClient is not able to find the required APIs and thus will throw an UnsupportedVersionException. Unfortunately, this exception is not caught and instead causes the Connect worker to fail even when the topics already exist.
This change handles the UnsupportedVersionException by logging a debug message and doing nothing. The existing producer logic will get information about the topics, which will cause the broker to create them if they don’t exist and broker auto-creation of topics is enabled. This is the same behavior that existed prior to 0.11.0.0, and so this change restores that behavior for brokers older than 0.10.1.0.
This change also adds a system test that verifies Connect works with a variety of brokers and is able to run source and sink connectors. The test verifies that Connect can read from the internal topics when the connectors are restarted.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3641 from rhauch/kafka-5704
ewencp would be great to cherry-pick this back into 0.11.x if possible
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3645 from xvrl/system-test-cluster-id
Notable updates:
1. Gradle 4.1 includes a number of performance and
CLI improvements as well as initial Java 9 support.
2. Scala 2.12.3 has substantial compilation time
improvements.
3. lz4-java 1.4 allows us to remove a workaround in
KafkaLZ4BlockInputStream (not done in this PR).
4. snappy-java 1.1.4 improved performance of compression (5%)
and decompression (20%). There was a slight increase in the
compressed size in one of our tests.
Not updated:
1. PowerMock due to a couple of regressions. I investigated one of them
and filed https://github.com/powermock/powermock/issues/828.
2. Jackson, which will be done via #3631.
3. Rocksdb, which will be done via #3519.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3619 from ijuma/update-deps-for-1.0.0
Separate `batch.size`, `message-size` and `compression-code` from PerfConfig to a newly-created ProducerPerfConfig in order to hide them in ConsumerPerf tool.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3613 from huxihx/KAFKA-2360
This field doesn't seem to be used and the value for
`AwaitingSync` seems to be wrong (it seems like it
should have been `2` instead of `5`).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3572 from ijuma/remove-unused-group-state-field
The AdminClient does not properly clear calls from the callsInFlight structure.
Later, in an effort to clear the lingering call objects, it closes the connection
they are associated with. This disrupts new incoming calls, which then get
BrokerNotAvailableException.
This patch fixes this bug by properly removing completed calls from the
callsInFlight structure. It also adds the Call#aborted flag, which
ensures that we throw the right exception (TimeoutException instead of
DisconnectException) and only abort a connection once -- even if there
is a similar bug in the future which causes old Call objects to linger.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3584 from cmccabe/KAFKA-5658
1. Remove separate thread from test failing periodically due to race condition.
2. Remove anonymous `AbstractNotifyingBatchingRestoreCallback` declare as concrete inner class `RocksDBBatchingRestoreCallback` and set as package private variable. Class is static so it has to initialize it's dependency on `RocksDBStore`
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3640 from bbejeck/KAFKA-5701_fix_flaky_unit_test
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3625 from bbejeck/HOTFIX_need_to_correct_stanby_task_restoration_to_use_new_restore_api
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Eno Thereska <eno.thereska@gmail.com>
Closes#3604 from dguy/memory-management-docs
Producer should not drop header information when splitting batches. This PR also corrects a minor typo in Sender.java, where `spitting and retrying` should be `splitting and retrying`.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
Closes#3620 from huxihx/KAFKA-5700
Added handling of _DUCKTAPE_OPTIONS (mainly for enabling debugging)
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3578 from ppatierno/kafka-5643
Support a --custom-ducktape flag which allows developers to install
their own versions of ducktape into Docker images. This is helpful for
ducktape development.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>
Closes#3539 from cmccabe/KAFKA-5602
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3627 from hachikuji/minor-fix-controlled-shutdown-error-response
With current tests, the deserialization inside the KStreamPrint node processor which happens when key and/or values are byte[] isn't tested. This PR fixes that.
Author: Paolo Patierno <ppatierno@live.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>
Closes#3611 from ppatierno/minor-kstream-print-test
Jason Gustafson
Ismael Juma
Eno Thereska
Damian Guy
huxihx
Colin P. Mccabe
Xavier Léauté
Bill Bejeck
Konstantine Karantasis
Randall Hauch
Andrey Dyachkov
Rajini Sivaram
Guozhang Wang
Evgeny Veretennikov
Soenke Liebau
Ewen Cheslack-Postava
ppatierno
Balint Molnar