Implemented an in-memory window store allowing for range queries. A finite retention period defines how long records will be kept, ie the window of time for fetching, and the grace period defines the window within which late-arriving data may still be written to the store.
Unit tests were written to test the functionality of the window store, including its insert/update/delete and fetch operations. Single-record, all records, and range fetch were tested, for both time ranges and key ranges. The logging and metrics for late-arriving (dropped)records were tested as well as the ability to restore from a changelog.
Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
This PR adds a upgrade notes and changes examples to use the bootstrap-server.
Author: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
Reviewers: Srinivas <srinivas96alluri@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Closes#6118 from viktorsomogyi/topiccommand-adminclient-doc
My top 2 reasons for visiting the Kafka docs are to:
- View configurations
- View metrics
This PR aims to improve the user experience for viewing metrics:
- Add href links to the `Monitoring` section of the Table of Contents so users do not need to scroll or Ctrl/Cmd-F to find specific metric details (Monitoring section has grown large as more component & metrics are added)
Author: lu.kevin@berkeley.edu <kelu@paypal.com>
Reviewers: Viktor Somogyi <viktorsomogyi@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Closes#5511 from KevinLiLu/feature/minor-improve-docs
KIP-368 implementation to enable periodic re-authentication of SASL clients. Also adds a broker configuration option to terminate client connections that do not re-authenticate within the configured interval.
KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level. This patch adds documentation for this feature.
Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Some sections of the Monitoring metrics documentation list out the -total metrics, and some sections do not list them out. Make them consistent and list out the missing -total metrics.
1. Rename metrics scope of rocksDB window and session stores; also modify the store metrics accordingly with guidance on its correlations to metricsScope.
2. Add the missing total metrics for per-thread, per-task, per-node and per-store sensors.
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
* Set --source, --target and --release to 1.8.
* Build Scala 2.12 by default.
* Remove some conditionals in the build file now that Java 8
is the minimum version.
* Bump the version of Jetty, Jersey and Checkstyle (the newer
versions require Java 8).
* Fixed issues uncovered by the new version if Checkstyle.
* A couple of minor updates to handle an incompatible source
change in the new version of Jetty.
* Add dependency to jersey-hk2 to fix failing tests caused
by Jersey upgrade.
* Update release script to use Java 8 and to take into account
that Scala 2.12 is now built by default.
* While we're at it, bump the version of Gradle, Gradle plugins,
ScalaLogging, JMH and apache directory api.
* Minor documentation updates including the readme and upgrade
notes. A number of Streams Java 7 examples can be removed
subsequently.
Add docs for the ReplicationBytesInPerSec and ReplicationBytesOutPerSec metrics. These metrics were introduced in KIP-153 (KAFKA-5194) but the docs were not updated.
Built site-docs, and viewed them in a web browser to confirm.
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>
With KIP-175 there are a number of consumer group command options that can be used to describe a consumer group.
This PR updates the documentation on consumer group command to mention those options.
A new mechanism was added recently to the Metrics framework to make it easier to generate the documentation. It uses a registry with a MetricsNameTemplate for each metric, and then those templates are used when creating the actual metrics. The metrics framework provides utilities that can generate the HTML documentation from the registry of templates.
This change moves the recently-added Connect metrics over to use these templates and to then generate the metric documentation for Connect.
This PR is based upon #3975 and can be rebased once that has been merged.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#3987 from rhauch/kafka-5990
And remove the deprecated `ConsumerOffsetChecker` example.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3405 from vahidhashemian/KAFKA-5495
A couple of fixes to metric names to match the KIP
- Removed extra strings in the metric names that are already in the tags
- add a separate metric for "all"
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3491 from enothereska/hotfix-metric-names
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3249 from jklukas/docs-no-more-yammer-metrics-in-client
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3415 from vahidhashemian/doc/make_topic_config_a_section
Operations documentation should guide user to employ `kafka-configs.sh` to add/remove configs for a topic.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3429 from huxihx/KAFKA-5508_Doc_for_altering_topics
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Derrick Or <derrickor@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3214 from guozhangwang/KMinor-doc-java-brush
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#3169 from rajinisivaram/MINOR-producer-metrics
Autogenerate docs for the Consumer Fetcher's metrics. This is a smaller subset of the original PR https://github.com/apache/kafka/pull/1202.
CC ijuma benstopford hachikuji
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#2993 from wushujames/fetcher_metrics_docs
1. Update value for queued.max.requests to 500
2. Removed invalid config 'controller.message.queue.size'
3. Removed flush configs including 'log.flush.interval.messages', 'log.flush.interval.ms' and 'log.flush.scheduler.interval.ms'
Author: huxi <huxi@zhenrongbao.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2490 from amethystic/kafka4727_server_config_doc_update
- Seperate Streams documentation out to a standalone page.
- Setup templates to use handlebars.js
- Create template variables to swap in frequently updated values like version number from a single file templateData.js
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2245 from derrickdoo/docTemplates
Removed a non-existing reference in table of contents and fixed some section numbers.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2004 from vahidhashemian/doc/fix_toc
And improve readability by adding proper punctuations.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2002 from vahidhashemian/doc/fix_typos
This is to imply that the Java consumer/producer are the recommended consumer/producer now.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1921 from vahidhashemian/KAFKA-3697
Details in KIP-55.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1847 from rajinisivaram/KAFKA-4079
A. Sophie Blee-Goldman
lambdaliu
Viktor Somogyi-Vass
John Roesler
lu.kevin@berkeley.edu
Ron Dagostino
Alex Dunayevsky
Manikumar Reddy O
Mayank Tankhiwale
Kevin Lu
Guozhang Wang
Ismael Juma
wushujames
Vahid Hashemian
Rajini Sivaram
Randall Hauch
Eno Thereska
Dong Lin
Jeff Klukas
huxihx
Eno Thereska
Alex Loddengaard
huxi
Scott Ferguson
Derrick Or
Ben Stopford
Andrew Otto