Implementation for KIP-138: Change punctuate semantics
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3055 from mihbor/KIP-138
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3310 from mjsax/kafka-5362-add-eos-system-tests-for-streams-api
These are the following improvements I made:
1. On stream thread level, INFO will be demonstrating `Completed xx tasks in yy ms` or `Completed rebalance with xx state in yy ms`,
2. On Stream thread cache level, INFO on `Flushed xx records`.
3. On Stream thread level, DEBUG on internal batched operations like `created xx tasks`, and TRACE on individual operation like `created x task`.
4. Also using `isTraceEnabled` on the critical path to reduce overhead of creating `Object[]`.
5. Minor cleanups in the code.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Steven Schlansker, Nicolas Fouché, Kamal C, Ismael Juma, Bill Bejeck, Eno Thereska, Matthias J. Sax, Damian Guy
Closes#3354 from guozhangwang/K4829-tasks-log4j
During a broker rolling upgrade, it's likely we don't have enough brokers ready yet. If streams does not need to create a topic it shouldn't check how many brokers are up.
The system test for this is in a separate PR: https://github.com/apache/kafka/pull/3411
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3418 from enothereska/hotfix-replication
In the stream library there are few cases where we don't leverage on logging methods features (i.e. using {} placeholder instead of string concatenation or passing the exception variable)
Author: ppatierno <ppatierno@live.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3419 from ppatierno/streams-consistent-logging
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3294 from jeyhunkarimov/KAFKA-4653
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3292 from jeyhunkarimov/KAFKA-4656
Throw IllegalArgumentException when attempting to create a `WindowStore` via `Stores` or directly with `RocksDBWindowStoreSupplier` when it has less than 2 segments.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3410 from dguy/kafka-4913
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3414 from mjsax/hotfix-commit-logging
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3293 from jeyhunkarimov/KAFKA-4658
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3290 from jeyhunkarimov/KAFKA-4655
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3106 from jeyhunkarimov/KAFKA-4785
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
Closes#3403 from guozhangwang/KMinor-turn-off-caching-in-demo
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3291 from jeyhunkarimov/KAFKA-4659
<strike> - actual fix for `StandbyTask#commit()` </strike>
Additionally (for debugging):
- EOS test, does not report "expected" value correctly
- add `IntegerDecoder` (to be use with `kafka.tools.DumpLogSegments`)
- add test for `StreamTask` to not checkpoint on commit if EOS enabled
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3375 from mjsax/kafka-5474-eos-standby-task
log info typos. should be standby task, not active task.
Author: zqhxuyuan <qihuang.zheng@fraudmetrix.cn>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#3362 from zqhxuyuan/trunk
- Use ResourceType.toJava instead of ResourceType.fromString. The latter
doesn't work for TransactionalId (or any type with two camel-case
words).
- Replace Throwable with ApiError in response classes.
- Return InvalidRequest instead of Unknown error if ANY or UNKNOWN
are provided during ACL creation.
- Rename `unknown()` to `isUnknown()` in a few places that
were missed previously.
- Add tests.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3364 from ijuma/acls-fromString-fixes
In the event of a crash, we always restore the data from the backing changelog. So it seems that we don't need to
persist all data to disk by calling flush when committing. Frequent flushing leads to a large number of small files for compaction increasing
compaction pressure. This PR will perform benchmarks to see if there is any performance gain in not calling `flush()` each time we commit.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3211 from bbejeck/MINOR_no_flush_on_commit
- multi-subtopology tests
- fencing test
- producer fenced bug fix: Streams did not recover correctly from ProducerFencedException
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3276 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3288 from jeyhunkarimov/KAFKA-4661
Originally we assume the task will be created exactly three times (twice upon starting up, once for each thread, and then one more time when rebalancing upon the thread failure). However there is a likelihood that upon starting up more than one rebalance will be triggered, and hence the tasks will be initialized more than 3 times, i.e. there will be more than three hashcodes of the `Transformer` object, causing the `errorInjected` to never be taken and exception never thrown.
The current fix is to use an atomic boolean instead and let threads compete on compare-and-set to make sure exactly one thread will throw exception, and will only throw once.
Without this patch I can reproduce the issue on my local machine with a single core ever 3-5 times; with this patch I have been running successfully for 10+ runs.
Ping mjsax ijuma for reviews.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3275 from guozhangwang/KHotfix-eos-integration-test
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3201 from mjsax/kafka-5362-add-eos-system-tests-for-streams-api
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3273 from mjsax/hotfix-flaky-stream-eos-test
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3135 from enothereska/exceptions-stores-KAFKA-5314
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3264 from mjsax/minor-javadocs-timestamp-extractor
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Apurva Mehta <apurva@confluent.io>
Closes#3251 from guozhangwang/KMinor-remove-ignored-streams-integration-test
Expose the streams commit interval config for different testing scenarios.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3244 from bbejeck/MINOR_add_commit_config_benchmark
This resolved the issue with Kafka Streams skipped records sensor reporting wrong values.
Jira ticket: https://issues.apache.org/jira/browse/KAFKA-5368
The contribution is my original work and I license the work to the project under the project's open source license.
Author: Hamidreza Afzali <hrafzali@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3206 from hrafzali/KAFKA-5368_skipped-records-sensor-bug
Author: Dale Peakall <dale@peakall.net>
Reviewers: Michael André Pearce <michael.andre.pearce@me.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3199 from subnova/streams-extendeddeserializer
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3195 from rajinisivaram/KAFKA-5345
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
subscribed stream may not be detected by all followers until
onJoinComplete returns.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3157 from bbejeck/KAFKA-5226_null_pointer_source_node_deserialize
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3172 from guozhangwang/K5350-compatibility-annotations
- Added a boolean `allow_auto_topic_creation` to MetadataRequest and
bumped the protocol version to V4.
- When connecting to brokers older than 0.11.0.0, the `allow_auto_topic_creation`
field won't be considered, so we send a metadata request for all topics
to keep the behavior consistent.
- Set `allow_auto_topic_creation` to false in the new AdminClient and
StreamsKafkaClient (which exists for the purpose of creating topics
manually); set it to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.
- Add `allowAutoTopicCreation` field to `Metadata`, which is used by
`DefaultMetadataUpdater`. This is not strictly needed for the new
`AdminClient`, but it avoids surprises if it ever adds a topic to `Metadata`
via `setTopics` or `addTopic`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3098 from ijuma/kafka-5291-admin-client-no-auto-topic-creation
Adjust "importance level" and add explanation to the docs.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#2855 from mjsax/minor-improve-streams-config-parameters
Handle` rocksdb.config.setter` being set as a class name or class
instance.
Author: Tommy Becker <tobecker@tivo.com>
Author: Tommy Becker <twbecker@gmail.com>
Reviewers: Matthias J. Sax, Damian Guy, Guozhang Wang
Closes#3155 from twbecker/KAFKA-5334
KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error
Change all the OptionParser constructor in Kafka into "new OptionParser(false)"
Author: xinlihua <xin.lihua1@zte.com.cn>
Author: unknown <00067310@A23338408.zte.intra>
Author: auroraxlh <xin.lihua1@zte.com.cn>
Author: xin <xin.lihua1@zte.com.cn>
Reviewers: Damian Guy, Guozhang Wang
Closes#2349 from auroraxlh/fix_OptionParser_bug
There is a Misspell in Annotations of ResetIntegrationTest.
Author: hejiefang <he.jiefang@zte.com.cn>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#3159 from hejiefang/KAFKA-5338
dguy , mjsax Please review the PR and let me know your comments.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Bill Bejeck, Matthias J. Sax, Guozhang Wang
Closes#3099 from umesh9794/mylocal
- introduces a new thread state DEAD
- ignores DEAD threads when querying
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3140 from mjsax/kafka-5309-stores-not-queryable
A follow up RP to fix [issue](2cd0b87bc8 (commitcomment-22200864))
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang
Closes#3109 from jeyhunkarimov/KAFKA-4144-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang
Closes#3104 from mjsax/minor-improve-embedded-kafka-cluster
Michal Borowiecki
Matthias J. Sax
Guozhang Wang
Eno Thereska
ppatierno
Jeyhun Karimov
Damian Guy
Qihuang Zheng
Ismael Juma
Bill Bejeck
Hamidreza Afzali
Dale Peakall
Rajini Sivaram
Colin P. Mccabe
Tommy Becker
xinlihua
hejiefang
Umesh Chaudhary