log info typos. should be standby task, not active task.
Author: zqhxuyuan <qihuang.zheng@fraudmetrix.cn>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#3362 from zqhxuyuan/trunk
- Use ResourceType.toJava instead of ResourceType.fromString. The latter
doesn't work for TransactionalId (or any type with two camel-case
words).
- Replace Throwable with ApiError in response classes.
- Return InvalidRequest instead of Unknown error if ANY or UNKNOWN
are provided during ACL creation.
- Rename `unknown()` to `isUnknown()` in a few places that
were missed previously.
- Add tests.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3364 from ijuma/acls-fromString-fixes
In the event of a crash, we always restore the data from the backing changelog. So it seems that we don't need to
persist all data to disk by calling flush when committing. Frequent flushing leads to a large number of small files for compaction increasing
compaction pressure. This PR will perform benchmarks to see if there is any performance gain in not calling `flush()` each time we commit.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3211 from bbejeck/MINOR_no_flush_on_commit
- multi-subtopology tests
- fencing test
- producer fenced bug fix: Streams did not recover correctly from ProducerFencedException
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3276 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3288 from jeyhunkarimov/KAFKA-4661
Originally we assume the task will be created exactly three times (twice upon starting up, once for each thread, and then one more time when rebalancing upon the thread failure). However there is a likelihood that upon starting up more than one rebalance will be triggered, and hence the tasks will be initialized more than 3 times, i.e. there will be more than three hashcodes of the `Transformer` object, causing the `errorInjected` to never be taken and exception never thrown.
The current fix is to use an atomic boolean instead and let threads compete on compare-and-set to make sure exactly one thread will throw exception, and will only throw once.
Without this patch I can reproduce the issue on my local machine with a single core ever 3-5 times; with this patch I have been running successfully for 10+ runs.
Ping mjsax ijuma for reviews.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3275 from guozhangwang/KHotfix-eos-integration-test
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3201 from mjsax/kafka-5362-add-eos-system-tests-for-streams-api
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3273 from mjsax/hotfix-flaky-stream-eos-test
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3135 from enothereska/exceptions-stores-KAFKA-5314
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3264 from mjsax/minor-javadocs-timestamp-extractor
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Apurva Mehta <apurva@confluent.io>
Closes#3251 from guozhangwang/KMinor-remove-ignored-streams-integration-test
Expose the streams commit interval config for different testing scenarios.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3244 from bbejeck/MINOR_add_commit_config_benchmark
This resolved the issue with Kafka Streams skipped records sensor reporting wrong values.
Jira ticket: https://issues.apache.org/jira/browse/KAFKA-5368
The contribution is my original work and I license the work to the project under the project's open source license.
Author: Hamidreza Afzali <hrafzali@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3206 from hrafzali/KAFKA-5368_skipped-records-sensor-bug
Author: Dale Peakall <dale@peakall.net>
Reviewers: Michael André Pearce <michael.andre.pearce@me.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3199 from subnova/streams-extendeddeserializer
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3195 from rajinisivaram/KAFKA-5345
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
subscribed stream may not be detected by all followers until
onJoinComplete returns.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3157 from bbejeck/KAFKA-5226_null_pointer_source_node_deserialize
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3172 from guozhangwang/K5350-compatibility-annotations
- Added a boolean `allow_auto_topic_creation` to MetadataRequest and
bumped the protocol version to V4.
- When connecting to brokers older than 0.11.0.0, the `allow_auto_topic_creation`
field won't be considered, so we send a metadata request for all topics
to keep the behavior consistent.
- Set `allow_auto_topic_creation` to false in the new AdminClient and
StreamsKafkaClient (which exists for the purpose of creating topics
manually); set it to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.
- Add `allowAutoTopicCreation` field to `Metadata`, which is used by
`DefaultMetadataUpdater`. This is not strictly needed for the new
`AdminClient`, but it avoids surprises if it ever adds a topic to `Metadata`
via `setTopics` or `addTopic`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3098 from ijuma/kafka-5291-admin-client-no-auto-topic-creation
Adjust "importance level" and add explanation to the docs.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#2855 from mjsax/minor-improve-streams-config-parameters
Handle` rocksdb.config.setter` being set as a class name or class
instance.
Author: Tommy Becker <tobecker@tivo.com>
Author: Tommy Becker <twbecker@gmail.com>
Reviewers: Matthias J. Sax, Damian Guy, Guozhang Wang
Closes#3155 from twbecker/KAFKA-5334
KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error
Change all the OptionParser constructor in Kafka into "new OptionParser(false)"
Author: xinlihua <xin.lihua1@zte.com.cn>
Author: unknown <00067310@A23338408.zte.intra>
Author: auroraxlh <xin.lihua1@zte.com.cn>
Author: xin <xin.lihua1@zte.com.cn>
Reviewers: Damian Guy, Guozhang Wang
Closes#2349 from auroraxlh/fix_OptionParser_bug
There is a Misspell in Annotations of ResetIntegrationTest.
Author: hejiefang <he.jiefang@zte.com.cn>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#3159 from hejiefang/KAFKA-5338
dguy , mjsax Please review the PR and let me know your comments.
Author: umesh chaudhary <umesh9794@gmail.com>
Reviewers: Bill Bejeck, Matthias J. Sax, Guozhang Wang
Closes#3099 from umesh9794/mylocal
- introduces a new thread state DEAD
- ignores DEAD threads when querying
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3140 from mjsax/kafka-5309-stores-not-queryable
A follow up RP to fix [issue](2cd0b87bc8 (commitcomment-22200864))
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang
Closes#3109 from jeyhunkarimov/KAFKA-4144-follow-up
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang
Closes#3104 from mjsax/minor-improve-embedded-kafka-cluster
- add broker compatibility system tests
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
Summary:
- add `reconnect.backoff.max.ms` common client configuration parameter
- if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an exponential backoff policy
- apply +/- 20% random jitter to smooth cluster reconnects
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1523 from dpkp/exp_backoff
Implements range scan for keys in windowed and session stores
Modifies caching session and windowed stores to use segmented cache keys.
Cache keys are internally prefixed with their segment id to ensure key ordering in the cache matches the ordering in the underlying store for keys spread across multiple segments.
This should also result in fewer cache keys getting scanned for queries spanning only some segments.
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#3027 from xvrl/windowstore-range-scan
Don't produce messages on a separate thread continuosly. Just produce one of each value and stop.
Close the producer once finished.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3080 from dguy/qs-test
I remove `KeyValuePrinter` and `KStreamForeach` two class, then implements them by `KStreamPeek`.
So, now `KStreamPeek` can do `KeyValuePrinter` and `KStreamForeach` job.
Author: jameschien <jameschien@staff.ruten.com.tw>
Author: JamesChien <jedichien@users.noreply.github.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2955 from jedichien/trunk
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Apurva Metha, Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2945 from mjsax/kafka-4923-add-eos-to-streams
Ensure checkpointable offsets for GlobalKTables are always written on close.
Author: Tommy Becker <tobecker@tivo.com>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#3054 from twbecker/KAFKA-5241
CachingSessionStore wasn't properly using the default keySerde if no Serde was supplied. I saw the below error in the logs for one of my test cases.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2963 from KyleWinkelman/CachingSessionStore-fix-keySerde-use
RocksDBSessionStore wasn't properly using the default aggSerde if no Serde was supplied.
Author: Kyle Winkelman <kyle.winkelman@optum.com>
Reviewers: Damian Guy, Guozhang Wang
Closes#2971 from KyleWinkelman/RocksDBSessionStore-fix-aggSerde-use
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Damian Guy, Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2466 from jeyhunkarimov/KAFKA-4144
…it is accessed from multiple threads
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3000 from cmccabe/KAFKA-5198
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2931 from mjsax/kafka-5140-flaky-reset-integration-test
a KStream.to() sink is also a topic
... so the KStreamTestDriver to fetch it when required
Author: Wim Van Leuven <wim.vanleuven@bigboards.io>
Author: Wim Van Leuven <wim.vanleuven@highestpoint.biz>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2716 from wimvanleuven/KAFKA-4927
Guozhang Wang
Eno Thereska
Damian Guy
Qihuang Zheng
Ismael Juma
Paolo Patierno
Bill Bejeck
Matthias J. Sax
Jeyhun Karimov
Hamidreza Afzali
Dale Peakall
Rajini Sivaram
Colin P. Mccabe
Tommy Becker
xinlihua
hejiefang
Umesh Chaudhary
Dana Powers
Xavier Léauté
James Chien
Kyle Winkelman
Wim Van Leuven