Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3505 from enothereska/minor-less-logs
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3496 from mjsax/KAFKA-5464-follow-up
Also fix quickstart.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3501 from ijuma/introduce-full-version-variable-in-docs
Ensure only one thread dies, not both.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3500 from enothereska/hotfix-queryable-state
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3497 from mjsax/disable-flaky-system-tests
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3493 from dguy/hotfix-test-failure
Operations documentation should guide user to employ `kafka-configs.sh` to add/remove configs for a topic.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3429 from huxihx/KAFKA-5508_Doc_for_altering_topics
Several fixes to state transition logic:
- Kafka streams will now be in ERROR when all threads are DEAD or when global thread stops unexpectedly
- Fixed transition logic in corner cases when thread is already dead or Kafka Streams is already closed
- Fixed incorrect transition diagram in StreamThread
- Unit tests to verify transitions
Also:
- re-enabled throwing an exception when an unexpected state change happens
- fixed a whole bunch of EoS tests that did not start a thread
- added more comments.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3432 from enothereska/KAFKA-5372-state-transitions
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3439 from mjsax/kafka-5464-streamskafkaclient-poll
Added logPrefix for avoiding stream thread name formatting replicated more times
Author: ppatierno <ppatierno@live.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3488 from ppatierno/kafka-5557
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3449 from mjsax/kafka-5167-streams-task-gets-stuck-after-re-balance-due-to-LockException
Fixed console output to be consistent with/without dry-run option
Author: ppatierno <ppatierno@live.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3443 from ppatierno/kafka-5525
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3484 from dguy/iq-doc
Fixed debug logging for the created state changelog topics
Added toString() for InternalTopicMetadata and InternalTopicConfig for above debug logging
Author: ppatierno <ppatierno@live.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3368 from ppatierno/kafka-5469
Update system tests to make use of the newly released 0.11 version.
Add on to https://github.com/apache/kafka/pull/3454
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3479 from enothereska/minor-compatibility-tests
Also fixed a bunch of broken links (details can be found in 34f8ecea0d)
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3473 from guozhangwang/KMinor-streams-doc-breakdown
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3428 from guozhangwang/KHotfix-add-copy-constructor-abstract-stream
More input validation for SchemaBuilder methods.
Author: Jeremy Custenborder <jcustenborder@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3474 from jcustenborder/KAFKA-5548
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3456 from hachikuji/KAFKA-5522
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>
Closes#3454 from ijuma/test-upgrades-from-0.11.0.x
- In clearAll method, get operation is removed.
- variable name `requestTimeout` changed to `requestTimeoutMs` for clarity
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3467 from Kamal15/frequest
remove unused eosEnabled field from ProcessorStateManager
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3469 from dguy/minor-remove-unused-boolean
Author: Michael G. Noll <michael@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#2651 from miguno/trunk-reduce-javadocs
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3390 from mjsax/kafka-5485-dont-suspend-tasks-twice
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3457 from hachikuji/rename-base-timestamp
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3446 from ijuma/improve-transaction-index-exception-sanity-check
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3303 from rajinisivaram/KAFKA-5402
This is used in the generated config table. Also fix a couple
of typos in the process.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3451 from ijuma/fix-doc-typos
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#3406 from hachikuji/KAFKA-5490
I'll let this have multiple runs on the branch builder to see if it fails, and investigate if so.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3441 from apurvam/MINOR-enable-transactions-bounce-test
Implementation for KIP-138: Change punctuate semantics
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3055 from mihbor/KIP-138
Use the file name whenever possible and replace / with [/\\]
when it's not.
Also remove unnecessary suppresions.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#3431 from ijuma/fix-checkstyle-suppressions-on-windows
We intended to make `KafkaConsumer.close()` idempotent,
but due to the fact that the `closed` variable is
checked without a lock prior to beginning close logic,
it is possible for two or more threads to see
`closed=false` and attempt to close.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3426 from hachikuji/minor-fix-consumer-idempotent-close
Before 0.11, we used to have limits for maximum message size on the producer, broker, and consumer side.
From 0.11 onward, these limits apply to record batches as a whole. This patch updates the documentation of the configs to make this explicit.
A separate patch will have more extensive upgrade notes to tie all the changes together in one narrative.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3374 from apurvam/KAFKA-5032-message-size-docs
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3310 from mjsax/kafka-5362-add-eos-system-tests-for-streams-api
NetworkClient's logging improvements:
- Include correlation id in a number of log statements
- Avoid eager toString call in parameter passed to log.debug
- Use node.toString instead of passing a subset of fields to the
logger
- Use requestBuilder instead of clientRequest in one of the log
statements
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3420 from ijuma/kafka-5506-offset-fetch-request-to-string-npe
-Tests for rolling upgrades for a streams app (keeping broker config fixed)
-Tests for rolling upgrades of brokers (keeping streams app config fixed)
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3411 from enothereska/KAFKA-5487-upgrade-test-streams
These are the following improvements I made:
1. On stream thread level, INFO will be demonstrating `Completed xx tasks in yy ms` or `Completed rebalance with xx state in yy ms`,
2. On Stream thread cache level, INFO on `Flushed xx records`.
3. On Stream thread level, DEBUG on internal batched operations like `created xx tasks`, and TRACE on individual operation like `created x task`.
4. Also using `isTraceEnabled` on the critical path to reduce overhead of creating `Object[]`.
5. Minor cleanups in the code.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Steven Schlansker, Nicolas Fouché, Kamal C, Ismael Juma, Bill Bejeck, Eno Thereska, Matthias J. Sax, Damian Guy
Closes#3354 from guozhangwang/K4829-tasks-log4j
After we call `release`, we should null out the reference so
that we neither use it or release it a second time.
This should fix the following exception that has been reported:
```text
[2017-06-23 03:24:02,485] ERROR stream-thread [...] Failed to close consumer: (org.apache.kafka.streams.processor.internals.StreamThread:1054)
org.apache.kafka.common.KafkaException: Failed to close kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1623)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1573)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1549)
at org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:1052)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:538)
Caused by: java.lang.IllegalStateException: release called on LoginManager with refCount == 0
at org.apache.kafka.common.security.authenticator.LoginManager.release(LoginManager.java:106)
at org.apache.kafka.common.network.SaslChannelBuilder.close(SaslChannelBuilder.java:125)
at org.apache.kafka.common.network.Selector.close(Selector.java:257)
at org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:505)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.close(ConsumerNetworkClient.java:439)
at org.apache.kafka.clients.ClientUtils.closeQuietly(ClientUtils.java:71)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1613)
```
It's worth noting that it's not clear how `SaslChannelBuilder.close()` is called more than
once and it would be good to understand that as well.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3422 from ijuma/sasl-channel-builder-idempotent
During a broker rolling upgrade, it's likely we don't have enough brokers ready yet. If streams does not need to create a topic it shouldn't check how many brokers are up.
The system test for this is in a separate PR: https://github.com/apache/kafka/pull/3411
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3418 from enothereska/hotfix-replication
In the stream library there are few cases where we don't leverage on logging methods features (i.e. using {} placeholder instead of string concatenation or passing the exception variable)
Author: ppatierno <ppatierno@live.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3419 from ppatierno/streams-consistent-logging
Eno Thereska
Matthias J. Sax
Damian Guy
Ismael Juma
huxihx
Paolo Patierno
Rajini Sivaram
Guozhang Wang
Jeremy Custenborder
Jason Gustafson
Ben Stopford
Kamal C
Apurva Mehta
Michael G. Noll
Dong Lin
Colin P. Mccabe
Michal Borowiecki
Ewen Cheslack-Postava