It describes the old deprecated clients and it's better to just
remove it.
The contribution is my original work and I license the work to the
project under the project's open source license.
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3385 from tombentley/KAFKA-4059
The current config sends org.apache.kafka and any unspecified logger to
stdout. They should go to `server.log` instead.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3402 from ijuma/kafka-5486-org.apache.kafka-logging-server.log
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3106 from jeyhunkarimov/KAFKA-4785
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
Closes#3403 from guozhangwang/KMinor-turn-off-caching-in-demo
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3291 from jeyhunkarimov/KAFKA-4659
<strike> - actual fix for `StandbyTask#commit()` </strike>
Additionally (for debugging):
- EOS test, does not report "expected" value correctly
- add `IntegerDecoder` (to be use with `kafka.tools.DumpLogSegments`)
- add test for `StreamTask` to not checkpoint on commit if EOS enabled
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3375 from mjsax/kafka-5474-eos-standby-task
With this patch, the `ProducePerfomance` tool can create transactions of differing durations.
This patch was used to to collect the initial set of benchmarks for transaction performance, documented here: https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit#gid=282787170
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3400 from apurvam/MINOR-add-transaction-size-to-producre-perf
This patch lowers the retry backoff when receiving a CONCURRENT_TRANSACTIONS error from an AddPartitions request. The default of 100ms would mean that back to back transactions would be 100ms long at minimum, making things to slow.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3377 from apurvam/HOTFIX-lower-retry-for-add-partitions
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3378 from mjsax/minor-producer-naming
the contribution is my original work and I license the work to the project under the project's open source license.
junrao , I had already made the code change before your last comment. I've done pretty much what you said, except that I've not used the current segment because I wasn't sure if it will always be available.
I'm happy to change it if you prefer.
I've run all the unit and integration tests which all passed.
Author: Kelvin Rutt <ruttkelvin@gmail.com>
Author: Kelvin Rutt <kelvin.rutt@sky.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3357 from kelvinrutt/kafka_5413_bugfix
Kafka Connect was adding duplicate group names in the response from the REST API's validation of connector configurations. This fixes the duplicates and maintains the order of the `ConfigDef` objects so that the `ConfigValue` results are in the same order.
This is a blocker and should be merged to 0.11.0.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3379 from rhauch/KAFKA-5472
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#3388 from hachikuji/KAFKA-5021
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Apurva Mehta <apurva@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3345 from guozhangwang/KMinor-streams-eos-docs
based on conversations with vahidhashemian rajinisivaram apurvam
The docs didn't make clear that what gets committed and what gets not may depend on the producer acks.
Author: Edoardo Comar <ecomar@uk.ibm.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3035 from edoardocomar/DOC-clarification-on-committed
See https://github.com/mitchellh/vagrant/issues/7531. The core of the issue is that vagrant rsync uses a fixed set of 1000 possible temp file entries for SSH ControlMaster files to cache SSH connections for rsyncing. A few notes:
* We can't break down the steps further and maintain performance due to various limitations in vagrant/vagrant-aws (rsync is only executed on `vagrant up`/`vagrant reload`/`vagrant rsync`, you can't enable/disable and rsync shared folder only during some of those stages, and provisioning only runs in parallel with vagrant-aws during `vagrant up`).
* We need to isolate each of the serialized rsync calls. (If we assumed `parallel` was available, we actually could get the parallelism back.) This is required because even across calls they could randomly choose the same temporary file.
* If there's a chance multiple instances were running on the same server at the same or nearly the same time, they can conflict since the same temp file entries are used globally. This means anything running on shared CI servers might end up syncing data between different CI jobs (!!), which could lead to some very strange results. Especially weird if they aren't even for the same type of job.
* Provisioning error check needs to be removed because it is catching rsync errors, but those can still happen in the initial `vagrant up` rsync step before the `vagrant up` provisioning step. It seems likely this bug was the cause of missing files anyway so this check might not be as valuable anymore.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3380 from ewencp/deparallelize-rsync
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3278 from apurvam/MINOR-add-logging-to-transaction-coordinator-in-all-failure-cases
These are internal methods with no tests and since we now have an `AdminClient`,
we should remove them.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3299 from ijuma/remove-unused-admin-utils-methods
This is contributed by Hao Chen.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3366 from guozhangwang/KMinor-fix-streams-link-in-uses
log info typos. should be standby task, not active task.
Author: zqhxuyuan <qihuang.zheng@fraudmetrix.cn>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
Closes#3362 from zqhxuyuan/trunk
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3353 from apurvam/KAFKA-5455-proper-javadocs-eos-clients
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jun Rao <junrao@gmail.com>
Closes#3361 from hachikuji/KAFKA-5435-ALT
- Use ResourceType.toJava instead of ResourceType.fromString. The latter
doesn't work for TransactionalId (or any type with two camel-case
words).
- Replace Throwable with ApiError in response classes.
- Return InvalidRequest instead of Unknown error if ANY or UNKNOWN
are provided during ACL creation.
- Rename `unknown()` to `isUnknown()` in a few places that
were missed previously.
- Add tests.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3364 from ijuma/acls-fromString-fixes
Before:
```
pri=TRACE t=Controller-1-to-broker-0-send-thread at=logger Controller 1 epoch 1 received response {error_code=0} for a request sent to broker <ip>:<port> (id: 0 rack: null)
```
After:
```
pri=TRACE t=Controller-1-to-broker-0-send-thread at=logger Controller 1 epoch 1 received response {error_code=0} for a request sent to broker <ip>:<port> (id: 0 rack: us-east-1d)
```
Author: Jeff Chao <jeffchao@me.com>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3358 from jeffchao/fix-controller-rack-aware-logging
https://issues.apache.org/jira/browse/KAFKA-5031
Implements additional check for `DefaultRecordBatch` that compares number of records declared in the header with actual number of records. Similarly for headers.
Author: gosubpl <github@gosub.pl>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3156 from gosubpl/KAFKA-5031
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3171 from hachikuji/minor-txn-channel-cleanups
“ as opposed to " don't render consistently across browsers. On current Kafka website they render correctly in Firefox but not Chrome (“runsâ€) - charset issue?
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Author: mihbor <mbor81@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3227 from mihbor/patch-7
More specifically, fix the case where a compressed V0 or V1 message is
larger than the producer batch size.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3356 from hachikuji/KAFKA-5456
Console Consumer help doesn't say that ``--partition`` option needs ``--offset`` otherwise will consume from the end of the partition. This minor fix makes that happen.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3153 from cotedm/console_consumer_help_fix
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3355 from apurvam/KAFKA-5457-memoryrecordsbuilder-has-room-for-should-account-for-headers
Removed the first two in favour of the latter.
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3350 from Kamal15/utilcleanup
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3348 from hachikuji/fix-has-unflushed-synchronization
In the event of a crash, we always restore the data from the backing changelog. So it seems that we don't need to
persist all data to disk by calling flush when committing. Frequent flushing leads to a large number of small files for compaction increasing
compaction pressure. This PR will perform benchmarks to see if there is any performance gain in not calling `flush()` each time we commit.
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3211 from bbejeck/MINOR_no_flush_on_commit
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3339 from ijuma/kafka-5275-admin-client-api-consistency
Increased the timeout from 30sec to 60sec. When running the system tests with packaged Kafka, Connect workers can take about 30seconds to start.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3344 from rhauch/KAFKA-5450
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3340 from hachikuji/add-random-aborts-to-system-test
Tom Bentley
Ismael Juma
Jeyhun Karimov
Guozhang Wang
Matthias J. Sax
Apurva Mehta
Ewen Cheslack-Postava
Jason Gustafson
Eno Thereska
Kelvin Rutt
Randall Hauch
Damian Guy
Edoardo Comar
Hao Chen
Qihuang Zheng
ppatierno
Jeff Chao
Kamal C
gosubpl
Michal Borowiecki
Michal Borowiecki
Dustin Cote
Bill Bejeck