Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1852 from becketqin/KAFKA-4148
Include a few clean-ups (also in producer section), mention deprecation plans and reorder so that the new consumer documentation is before the old consumers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1880 from ijuma/remove-beta-from-new-consumer-documentation
Details in KIP-55.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1847 from rajinisivaram/KAFKA-4079
Now uses LogSegment.largestTimestamp to determine age of segment's messages.
Author: Eric Wasserman <eric.wasserman@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1794 from ewasserman/feat-1981
Update documentation for Kafka Connect distributed’s config.storage.topic, offset.storage.topic, and status.storage.topic configuration values to indicate that all three should refer to compacted topics.
Author: Mathieu Fenniak <mathieu.fenniak@replicon.com>
Reviewers: Jason Gustafson
Closes#1832 from mfenniak/kafka-connect-topic-docs
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1809 from becketqin/KAFKA-4099
Rephrase 'alpha quality' wording in Streams section of api.html.
Couple of other minor fixes in streams.html
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang, Ismael Juma, Michael G. Noll
Closes#1811 from dguy/kstreams-312
1. I think the instructions in step 2 of the security section which describe adding the CA to server/client truststores are swapped. That is, the instruction that says to add the CA to the server truststore adds it to the client truststore (and vice versa).
2. "clients keys" should be possessive ("clients' keys").
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1651 from ssaamm/trunk
Add an optional configuration for the SecureRandom PRNG implementation, with the default behavior being the same (use the default implementation in the JDK/JRE).
Author: Todd Palino <Todd Palino>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#1747 from toddpalino/trunk
added new consumer metrics section
refactored common metrics into new section
updated TOC
Author: Kaufman Ng <kaufman@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1361 from coughman/KAFKA-3479-consumer-metrics-doc
ijuma said that it would make sense to split out this work from KAFKA-3234, since KAFKA-3234 had both a mechanical change (generating docs) as well as a change requiring discussion (deprecating/renaming config options).
jjkoshy, I hope you don't mind that I took over this work. It's been 3 months since the last activity on KAFKA-3234, so I thought it would be okay to take over.
This work is essentially is the first 5-6 commits from Joel's https://github.com/apache/kafka/pull/907. However, since I'm not very experienced with git, I didn't do a direct merge/rebase, but instead largely hand-merged it. I did some minor cleanup. All credit goes to Joel, all blame goes to me. :)
For reference, I attached the auto-generated configuration.html file (as a PDF, because github won't let me attache html).
[configuration.pdf](https://github.com/apache/kafka/files/323901/configuration.pdf)
This is my first time writing Scala, so let me know if there are any changes needed.
I don't know who is the right person to review this. ijuma, can you help me redirect this to the appropriate person? Thanks.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1527 from wushujames/generate_topic_docs
By default Kafka is configured to allow ssl communication without hostname verification. This docs has been amended to include instructions on how to set that up in the event clients would like to take a more conservative approach.
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1384 from rnpridgeon/KAFKA-3667
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1630 from ssaamm/trunk
I've updated the ops documentation with information on using the XFS filesystem, based on LinkedIn's testing (and subsequent switch from EXT4).
I've also added some information to clarify the potential risk to the suggested EXT4 options (again, based on my experience with a multiple broker failure situation).
Author: Todd Palino <tpalino@linkedin.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Dana Powers <dana.powers@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1605 from toddpalino/trunk
Since 0.9.0.1 Configuration parameter log.cleaner.enable is now true by default.
Author: Nihed MBAREK <nihedmm@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1592 from nihed/patch-1
The reasons to remove it are:
1. It's currently broken. The purpose of the [JIRA](https://issues.apache.org/jira/browse/KAFKA-3761) was to report that the RunningAsController state gets overwritten back to "RunningAsBroker".
2. It's not a useful state.
a. If clients want to use this metric to know whether a broker is ready to receive requests or not, they do not care whether or not the broker is the controller
b. there is already a separate boolean property, KafkaController.isActive which contains this information.
Author: Roger Hoover <roger.hoover@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1437 from theduderog/KAFKA-3761-broker-state
We had mentioned this step in the performance impact section in the middle of a long paragraph, which made it easy to miss. I also tweaked the reason for setting `log.message.format.version` as it could be misinterpreted previously.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1514 from ijuma/tweak-upgrade-notes
To be consistent with `ConfigCommand` and `TopicCommand`.
No release includes this option yet, so we can simply change it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Mickael Maison, Grant Henke
Closes#1430 from ijuma/use-force-instead-of-yes-in-acl-command and squashes the following commits:
bdf3a57 [Ismael Juma] Update `AclCommandTest`
78b8467 [Ismael Juma] Change variable name to `forceOpt`
0bb27af [Ismael Juma] Use `--force` instead of `--yes` in `AclCommand`
Added a new argument to AclCommand: --yes. When set, automatically answer yes to prompts
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Gwen Shapira
Closes#1406 from mimaison/KAFKA-3732
Adding sizing recommendations for file descriptors to the ops guide.
Author: Dustin Cote <dustin@confluent.io>
Author: Dustin Cote <dustin@dustins-mbp.attlocal.net>
Reviewers: Gwen Shapira
Closes#1353 from cotedm/KAFKA-3683 and squashes the following commits:
8120318 [Dustin Cote] Adding file descriptor sizing recommendations
0908aa9 [Dustin Cote] Merge https://github.com/apache/kafka into trunk
32315e4 [Dustin Cote] Merge branch 'trunk' of https://github.com/cotedm/kafka into trunk
13309ed [Dustin Cote] Update links for new consumer API
4dcffc1 [Dustin Cote] Update links for new consumer API
Process grep command has been updated. Previous "ps | grep server-1.properties" command is showing nothing.
Author: Satendra Kumar <satendra@knoldus.com>
Reviewers: Gwen Shapira
Closes#1386 from satendrakumar06/patch-1
It was previously in the SASL section (probably by mistake).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>
Closes#1405 from ijuma/fix-security-upgrade-location-in-docs
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1060 from MayureshGharat/KAFKA-3393
This is not an exact revert as the code changed a bit since the
original commit. We also include a note in `upgrade.html`.
The original commit is 1182d61deb23b5cd86cbe462471f7df583a796e1.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Guozhang Wang
Closes#1391 from ijuma/kafka-3704-revert and squashes the following commits:
7891b67 [Ismael Juma] Tweak upgrade note based on Gwen's feedback
1673cd0 [Ismael Juma] Revert "KAFKA-3704: Remove hard-coded block size in KafkaProducer"
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1375 from becketqin/KAFKA-3698
If you're pushing close to the network capacity, 0.10's additional 8 bytes per message can lead to overload of your network. We (Heroku Kafka) ran into this issue whilst benchmarking 0.10 RC and the ijuma suggested it belonged in the update note.
Comments/suggestions welcome.
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1389 from tcrayford/upgrade_note_about_increased_network_bandwidth
…e throughput drop due to the addition of timestamp field.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1372 from becketqin/KAFKA-3565
This fixes test_producer_throughput with compression_type=snappy.
Also: added heap dump on out of memory error to `producer_performance.py` and corrected the upgrade note related to the change in buffer size for compression streams.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1385 from ijuma/kafka-3713-test_producer_throughput-snappy-fail and squashes the following commits:
54c7962 [Ismael Juma] Correct upgrade note about buffer size for compression stream
515040b [Ismael Juma] Call `compressor.close()` to fix memory leak
5311e5b [Ismael Juma] Dump heap on out of memory error when running `producer_performance.py`
This is a follow up of KAKFA-3421 to update the connect developer guide to include the configuration validation. Also includes a couple of minor fixes.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1366 from Ishiihara/connect-dev-doc
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1373 from ijuma/fix-producer-buffer-size-upgrade-note
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1358 from hachikuji/KAFKA-3583
Jiangjie Qin
Ismael Juma
Rajini Sivaram
Matthias J. Sax
Eric Wasserman
Mathieu Fenniak
Damian Guy
Andrew Otto
Alex Loddengaard
Ashish Singh
Samuel Taylor
Todd Palino
Ewen Cheslack-Postava
sahil kharb
Kaufman Ng
James Cheng
Ryan P
Gwen Shapira
Todd Palino
Nihed Bbarek
Alex Glikson
Thanasis Katsadas
Roger Hoover
Guozhang Wang
Filipe Azevedo
uchan-nos
Mickael Maison
Dustin Cote
Satendra Kumar
Sriharsha Chintalapani
MayureshGharat
Tom Crayford
Liquan Pei
Jason Gustafson