* Turned 3 inner classes that weren't static but could be into `static` ones.
* Turned one `public` inner class that wasn't used publicly into a `private`.
Trivial but imo worthwhile to explicitly keep visibility and "staticness" correct in syntax (if only to be nice to the GC) :)
Author: Armin Braun <me@obrown.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2574 from original-brownbear/cleanup-inner-nonstatic
Fixed ClassCastException resulting from missing type hint in request logging.
Author: Armin Braun <me@obrown.io>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2571 from original-brownbear/fix-logging-err-response
…lass should be static
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2558 from cmccabe/KAFKA-4774
The throttling system test sometimes fail because it takes longer than the current 10 second time out for partitions to get assigned to the consumer.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2567 from apurvam/increase-timeout-for-partitions-assigned
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Ewen Cheslack-Postava <me@ewencp.org>
Closes#2566 from hachikuji/hotfix-request-logging
Currently the checkpoint file is deleted at state store initialization and it is only ever written again during a clean shutdown. This can result in significant delays during restarts as the entire store needs to be loaded from the changelog.
We can mitigate against this by frequently checkpointing the offsets. The checkpointing happens only during the commit phase, i.e, after we have manually flushed the store and the producer. So we guarantee that the checkpointed offsets are never greater than what has been flushed.
In the event of hard failure we can recover by reading the checkpoints and consuming from the stored offsets.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2471 from dguy/kafka-4317
More details:
* Replaced `struct` field in Request/Response with a `toStruct` method. This
makes the performance model (including memory usage) easier to understand.
Note that requests have `toStruct()` while responses have `toStruct(version)`.
* Replaced mutable `version` field in `Request.Builder` with an immutable
field `desiredVersion` and a `version` parameter passed to the `build` method.
* Optimised `handleFetchRequest` to avoid unnecessary creation of `Struct`
instances (from 4 to 2 in the worst case and 2 to 1 in the best case).
* Various clean-ups in request/response classes and their test. In particular,
it is now clear what we are testing. Previously, it looked like we were testing
more than we really were.
With this in place, we could remove `AbstractRequest.Builder` in the future by
doing the following:
* Change `AbstractRequest.toStruct` to accept a version (like responses).
* Change `AbstractRequest.version` to be `desiredVersion` (like `Builder`).
* Change `ClientRequest` to take `AbstractRequest`.
* Move validation from the `build` methods to the request constructors or
static factory methods.
* Anything else required for the code to compile again.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva.1618@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#2513 from ijuma/separate-struct
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2544 from becketqin/KAFKA-4340_follow_up
Lowered the default RocksDB settings for the block cache and write buffers
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes#2525 from dguy/kafka-4484
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2536 from hachikuji/minor-move-compression-io-construction
We are now using artificially broken hosts that are not resolved as potential
collisions (127.0.53.53s) by some DNS servers.
This change is the only way to build while using Google's `8.8.8.8` (at least in my network).
Author: Armin Braun <me@obrown.io>
Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2552 from original-brownbear/KAFKA-4765
https://issues.apache.org/jira/browse/KAFKA-4720
Peek is a handy method to have to insert diagnostics that do not affect the stream itself, but some external state such as logging or metrics collection.
Author: Steven Schlansker <sschlansker@opentable.com>
Reviewers: Damian Guy, Matthias J. Sax, Eno Thereska, Guozhang Wang
Closes#2493 from stevenschlansker/kafka-4720-peek
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva.1618@gmail.com>, Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#2545 from hachikuji/KAFKA-4761
…porter.configure
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2540 from cmccabe/KAFKA-4756
`ConsumerGroup` should be `Group`.
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2534 from cotedm/acl-znode-correction
Reverting some of the recent changes to quickstart doc. Further explanation is provided inline.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2539 from vahidhashemian/doc/fix_quickstart_issues
Update streams.html with GlobalKTable docs
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Matthias J. Sax, Guozhang Wang
Closes#2516 from dguy/global-tables-doc
Add a note to `KafkaStreams.metadataForKey(String, K, Serializer<K>)` to point out that in the case of a Window Store the Serializer should still be the record key serializer and not a window serializer
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Xavier Léauté, Guozhang Wang
Closes#2532 from dguy/minor-docs
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2475 from vahidhashemian/minor/use_explicit_Errors_type_when_possible
1. Added an architecture section.
2. Added a configuration / execution sub-section to developer guide.
Minor tweaks and a bunch of missing fixes from `kafka-site` repo.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Derrick Or <derrickor@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2488 from guozhangwang/KMinor-streams-docs-second-pass
Author: Grant Henke <ghenke@cloudera.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2246 from granthenke/truststore-password
Generate core project with correct source folders. In addition
set output folders same as command line build. Don't generate
unnecessary projects.
Author: Dhwani Katagade <dhwani_katagade@persistent.com>
Reviewers: Edoardo Comar <ecomar@uk.ibm.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2382 from dhwanikatagade/gradle_eclipse_plugin_path_fix
This PR fixes a blocker issue, where the streams client code cannot talk to the controller. It also enables a system test that was previously failing.
This PR is for trunk only. A separate PR with just the fix (but not the tests) will be created for 0.10.2.
Author: Eno Thereska <eno@confluent.io>
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy, Ismael Juma, Matthias J. Sax, Guozhang Wang
Closes#2522 from enothereska/KAFKA-4716-metadata
This caused the bounce and smoke tests to fail on trunk.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2524 from enothereska/hotfix-tests
GroupCoordinatorMetrics currently sets up join-time-max and sync-time-max incorrectly as a "new Avg()" MeasurableStat instead of "new Max()"
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2520 from onurkaraman/KAFKA-4749
Author: Eno Thereska <eno.thereska@gmail.com>
Author: Eno Thereska <eno@confluent.io>
Author: Ubuntu <ubuntu@ip-172-31-22-146.us-west-2.compute.internal>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2478 from enothereska/minor-benchmark-args
Provide test coverage for exception paths in: `schedule()`, `closeTopology()`, and `punctuate()`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2451 from dguy/kafka-4640
Author: Satish Duggana <sduggana@hortonworks.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2509 from satishd/buffer-cleanup
Added general explanation of the tool and what it does. Also added few details to the arguments.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Matthias J. Sax, Michael G. Noll, Guozhang Wang
Closes#2503 from gwenshap/KAFKA-4733
This change is in response to [KAFKA-4725](https://issues.apache.org/jira/browse/KAFKA-4725).
When a produce request is received, if the user/client is exceeding their produce quota, the response will be delayed until the quota is refilled appropriately.
Unfortunately, the request body is still referenced in the callback which in turn leaks the messages contained within the request.
This change allows the `KafkaApis` method to take ownership of the request body from the `RequestChannel.Request` object.
I am not sure whether this breaks other invariants which are assumed within other parts of Kafka.
Author: Tim Carey-Smith <tim@spork.in>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#2496 from halorgium/fix-throttled-response-leak
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2501 from becketqin/KAFKA-4734
1. Update value for queued.max.requests to 500
2. Removed invalid config 'controller.message.queue.size'
3. Removed flush configs including 'log.flush.interval.messages', 'log.flush.interval.ms' and 'log.flush.scheduler.interval.ms'
Author: huxi <huxi@zhenrongbao.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2490 from amethystic/kafka4727_server_config_doc_update
Jason Gustafson
Armin Braun
Allen Xiang
Colin P. Mccabe
Apurva Mehta
Prabhat Kashyap
Damian Guy
Ismael Juma
Jiangjie Qin
Aegeaner
Steven Schlansker
Matthias J. Sax
Eno Thereska
rnpridgeon
Dustin Cote
Vahid Hashemian
Guozhang Wang
Grant Henke
Dhwani Katagade
Bill Bejeck
Eno Thereska
Onur Karaman
Alex Loddengaard
Jonathan Monette
Satish Duggana
Gwen Shapira
Tim Carey-Smith
huxi