based on conversations with vahidhashemian rajinisivaram apurvam
The docs didn't make clear that what gets committed and what gets not may depend on the producer acks.
Author: Edoardo Comar <ecomar@uk.ibm.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3035 from edoardocomar/DOC-clarification-on-committed
See https://github.com/mitchellh/vagrant/issues/7531. The core of the issue is that vagrant rsync uses a fixed set of 1000 possible temp file entries for SSH ControlMaster files to cache SSH connections for rsyncing. A few notes:
* We can't break down the steps further and maintain performance due to various limitations in vagrant/vagrant-aws (rsync is only executed on `vagrant up`/`vagrant reload`/`vagrant rsync`, you can't enable/disable and rsync shared folder only during some of those stages, and provisioning only runs in parallel with vagrant-aws during `vagrant up`).
* We need to isolate each of the serialized rsync calls. (If we assumed `parallel` was available, we actually could get the parallelism back.) This is required because even across calls they could randomly choose the same temporary file.
* If there's a chance multiple instances were running on the same server at the same or nearly the same time, they can conflict since the same temp file entries are used globally. This means anything running on shared CI servers might end up syncing data between different CI jobs (!!), which could lead to some very strange results. Especially weird if they aren't even for the same type of job.
* Provisioning error check needs to be removed because it is catching rsync errors, but those can still happen in the initial `vagrant up` rsync step before the `vagrant up` provisioning step. It seems likely this bug was the cause of missing files anyway so this check might not be as valuable anymore.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3380 from ewencp/deparallelize-rsync
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3278 from apurvam/MINOR-add-logging-to-transaction-coordinator-in-all-failure-cases
These are internal methods with no tests and since we now have an `AdminClient`,
we should remove them.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3299 from ijuma/remove-unused-admin-utils-methods
This is contributed by Hao Chen.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3366 from guozhangwang/KMinor-fix-streams-link-in-uses
log info typos. should be standby task, not active task.
Author: zqhxuyuan <qihuang.zheng@fraudmetrix.cn>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#3362 from zqhxuyuan/trunk
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3353 from apurvam/KAFKA-5455-proper-javadocs-eos-clients
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jun Rao <junrao@gmail.com>
Closes#3361 from hachikuji/KAFKA-5435-ALT
- Use ResourceType.toJava instead of ResourceType.fromString. The latter
doesn't work for TransactionalId (or any type with two camel-case
words).
- Replace Throwable with ApiError in response classes.
- Return InvalidRequest instead of Unknown error if ANY or UNKNOWN
are provided during ACL creation.
- Rename `unknown()` to `isUnknown()` in a few places that
were missed previously.
- Add tests.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3364 from ijuma/acls-fromString-fixes
Before:
```
pri=TRACE t=Controller-1-to-broker-0-send-thread at=logger Controller 1 epoch 1 received response {error_code=0} for a request sent to broker <ip>:<port> (id: 0 rack: null)
```
After:
```
pri=TRACE t=Controller-1-to-broker-0-send-thread at=logger Controller 1 epoch 1 received response {error_code=0} for a request sent to broker <ip>:<port> (id: 0 rack: us-east-1d)
```
Author: Jeff Chao <jeffchao@me.com>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3358 from jeffchao/fix-controller-rack-aware-logging
https://issues.apache.org/jira/browse/KAFKA-5031
Implements additional check for `DefaultRecordBatch` that compares number of records declared in the header with actual number of records. Similarly for headers.
Author: gosubpl <github@gosub.pl>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3156 from gosubpl/KAFKA-5031
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3171 from hachikuji/minor-txn-channel-cleanups
“ as opposed to " don't render consistently across browsers. On current Kafka website they render correctly in Firefox but not Chrome (“runsâ€) - charset issue?
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Author: mihbor <mbor81@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3227 from mihbor/patch-7
More specifically, fix the case where a compressed V0 or V1 message is
larger than the producer batch size.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3356 from hachikuji/KAFKA-5456
Console Consumer help doesn't say that ``--partition`` option needs ``--offset`` otherwise will consume from the end of the partition. This minor fix makes that happen.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3153 from cotedm/console_consumer_help_fix
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3355 from apurvam/KAFKA-5457-memoryrecordsbuilder-has-room-for-should-account-for-headers
Removed the first two in favour of the latter.
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3350 from Kamal15/utilcleanup
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3348 from hachikuji/fix-has-unflushed-synchronization
In the event of a crash, we always restore the data from the backing changelog. So it seems that we don't need to
persist all data to disk by calling flush when committing. Frequent flushing leads to a large number of small files for compaction increasing
compaction pressure. This PR will perform benchmarks to see if there is any performance gain in not calling `flush()` each time we commit.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3211 from bbejeck/MINOR_no_flush_on_commit
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3339 from ijuma/kafka-5275-admin-client-api-consistency
Increased the timeout from 30sec to 60sec. When running the system tests with packaged Kafka, Connect workers can take about 30seconds to start.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3344 from rhauch/KAFKA-5450
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3340 from hachikuji/add-random-aborts-to-system-test
Also remove broker-id tags as we generally use them to provide
additional context.
Finally, do a few clean-ups (could not resist).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3341 from ijuma/tweak-transaction-metrics
- multi-subtopology tests
- fencing test
- producer fenced bug fix: Streams did not recover correctly from ProducerFencedException
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3276 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
Publish Javadoc for common.annotation package, which contains
InterfaceStability.
Finally, mark AdminClient classes with `Evolving` instead of `Unstable`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Colin Mccabe, Gwen Shapira
Closes#3316 from ijuma/kafka-5274-admin-client-javadoc
Headers are only preserved if the new consumer is
used since the old consumer does not support
them.
Add test case to verify the fix and to avoid regression.
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3322 from michaelandrepearce/KAFKA-5354
And add warning about usage.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3330 from ijuma/extended-serializer-javadoc
NetworkClient#disconnect should not erase the connection information. This will allow exponential
backoff to occur.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3309 from cmccabe/disc
Related to https://github.com/apache/kafka/pull/3321
Author: Konstantine Karantasis <konstantine@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3326 from kkonstantine/MINOR-Add-tests-for-PluginDesc
Before this patch, the `partitionErrors` was an immutable map. As a result if a single producer had a marker for multiple partitions, and if there were multiple response callbacks for a single append, we would get an `UnsupportedOperationException` in the `writeTxnMarker` handler.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3313 from apurvam/KAFKA-5438-fix-unsupportedoperationexception-in-writetxnmarker
The default client-id bandwidth quota config properties have been marked deprecated in the doc, but a warning may be useful before the property is removed in a future release.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3315 from rajinisivaram/MINOR-deprecate-staticquota
Skip topics that don't have any partitions in zkUtils.getAllPartitions()
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3295 from mimaison/KAFKA-5418
Edoardo Comar
Jason Gustafson
Ewen Cheslack-Postava
Apurva Mehta
Ismael Juma
Hao Chen
Qihuang Zheng
ppatierno
Jeff Chao
Kamal C
gosubpl
Michal Borowiecki
Michal Borowiecki
Dustin Cote
Tom Bentley
Bill Bejeck
Matthias J. Sax
Randall Hauch
Michael Andre Pearce
Colin P. Mccabe
Konstantine Karantasis
Rajini Sivaram
Mickael Maison