Kafka brokers have a config called "offsets.topic.replication.factor" that specify the replication factor for the "__consumer_offsets" topic. The problem is that this config isn't being enforced. If an attempt to create the internal topic is made when there are fewer brokers than "offsets.topic.replication.factor", the topic ends up getting created anyway with the current number of live brokers. The current behavior is pretty surprising when you have clients or tooling running as the cluster is getting setup. Even if your cluster ends up being huge, you'll find out much later that __consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement on the internal topic is another way of saying the cluster isn't fully setup yet.
The right behavior should be for "offsets.topic.replication.factor" to be enforced. Topic creation of the internal topic should fail with GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor" requirement is met. This closely resembles the behavior of regular topic creation when the requested replication factor exceeds the current size of the cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2177 from onurkaraman/KAFKA-3959
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Gwen Shapira <cshapi@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2387 from cmccabe/KAFKA-4664
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Michael G. Noll
Closes#2445 from ijuma/kafka-4578-upgrade-notes-0.10.2
This was missing from [an earlier PR](https://github.com/apache/kafka/pull/2247) that escaped these symbols in another section of the doc.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2453 from vahidhashemian/doc/escape_lt_gt_in_streams_code
Updated the docs with changes in KAFKA-4060.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Author: Hojjat Jafarpour <hojjat@HojjatJpoursMBP.attlocal.net>
Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang
Closes#2377 from hjafarpour/KAFKA-4060-docs-update
Renames `HoistToStruct` SMT to `HoistField`.
Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`
Adds HTML doc generation and updates to `connect.html`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2374 from shikhar/more-smt
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2366 from rajinisivaram/KAFKA-4626
This makes it consistent with MirrorMaker with the old consumer.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2072 from amethystic/kafka-4351_Regex_behavior_change_for_new_consumer
4 release cycles (0.9.0.0, 0.10.0.0, 0.10.1.0, 0.10.2.0) should be enough
to remove the beta label.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2286 from ijuma/kafka-3284-security-beta-label
Based on: https://github.com/apache/kafka-site/pull/27
I recommend also merging into 10.1.0.0 branch to avoid mishaps when re-publishing docs.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2077 from gwenshap/protocol_doc_fix
- Seperate Streams documentation out to a standalone page.
- Setup templates to use handlebars.js
- Create template variables to swap in frequently updated values like version number from a single file templateData.js
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2245 from derrickdoo/docTemplates
Performed minor cleanup and escaped `<` and `>` so code examples are shown correctly in the browser.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2247 from vahidhashemian/doc/fix_streams_doc
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll, Eno Thereska, Damian Guy, Guozhang Wang
Closes#2117 from mjsax/kafka-4393-improveInvalidTsHandling
Removed a duplicate line and also cleaned up some of the language around compaction guarantees.
Author: Apurva Mehta <apurva.1618@gmail.com>
Reviewers: Gwen Shapira
Closes#2089 from apurvam/fix-documentation-of-compaction and squashes the following commits:
03c5bdd [Apurva Mehta] Fix line length to be consistent with the rest of the file
0af1a86 [Apurva Mehta] MINOR: fix duplicate line in docs for compaction.
Removed a non-existing reference in table of contents and fixed some section numbers.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2004 from vahidhashemian/doc/fix_toc
And improve readability by adding proper punctuations.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2002 from vahidhashemian/doc/fix_typos
Make the latest version of our docs follow the latest site template structure.
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2003 from derrickdoo/docs-updates
Author: Jakub Dziworski <jakub.dziworski@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1996 from JakubDziworski/doc_fix_streams_link
Author: Jakub Dziworski <jakub.dziworski@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1994 from JakubDziworski/doc_fix
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1990 from vahidhashemian/doc/quickstart_update_windows
This is a minor change to fix the most glaring issues. We have another JIRA to revamp the upgrade docs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jeff Klukas <jeff@klukas.net>, Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1971 from ijuma/kafka-4257-upgrade-docs-inconsitencies
In doc it stays:
_"Our topic is divided into a set of totally ordered partitions, each of which is consumed by one consumer at any given time."_
And consumer is described as:
_"We'll call **processes** that subscribe to topics and process the feed of published messages **consumers**."_
Which might lead to a wrong conclusion - that each partition can be read by one process at any given time.
I think this statements misses information about **consumer groups**, so i propose:
_"Our topic is divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer (from each subscribed consumer groups) at any given time"_
This contribution is my original work and I license the work to the project under the project's open source license.
Author: pilo <jakub.pilimon@4finance.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1900 from pilloPl/minor/doc-fix
Reopening of https://github.com/apache/kafka/pull/1428
Author: Edoardo Comar <ecomar@uk.ibm.com>
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1908 from edoardocomar/KAFKA-3396
This is to imply that the Java consumer/producer are the recommended consumer/producer now.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1921 from vahidhashemian/KAFKA-3697
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1897 from becketqin/KAFKA-4194
Onur Karaman
Ewen Cheslack-Postava
Colin P. Mccabe
Ismael Juma
Vahid Hashemian
Hojjat Jafarpour
Shikhar Bhushan
Rajini Sivaram
dasl
huxi
Gwen Shapira
Scott Ferguson
Hikiko Murakami
Derrick Or
Matthias J. Sax
Stig Rohde Døssing
Alex Loddengaard
Apurva Mehta
Eno Thereska
Ben Stopford
Manikumar Reddy O
Jason Gustafson
Andrea Cosentino
Jakub Dziworski
pilo
Edoardo Comar
Jiangjie Qin
Jay Kreps