The host and port entries under /brokers/ids/<bid> gets filled only for PLAINTEXT security protocol. For other protocols the host is null and the actual endpoint is under "endpoints". This causes NPE when running the consumer group and offset checker scripts in a kerberized env. By always reading the host and port values from the "endpoint", a more meaningful exception would be thrown rather than a NPE.
Author: Arun Mahadevan <aiyer@hortonworks.com>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1301 from arunmahadevan/cg_kerb_fix
Set OffsetsTopicReplicationFactorProp to 3 like MinInSyncReplicasProp Else a consumer was able to consume via assign but not via subscribe, so the testProduceAndConsume is now duplicated to check both paths
Author: Edoardo Comar <ecomar@uk.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1425 from edoardocomar/KAFKA-3728
- Fixed the logic calculating the windows that are affected by a new …event in the case of hopping windows and a small overlap.
- Added a unit test that tests for the issue
Author: Tom Rybak <trybak@gmail.com>
Reviewers: Michael G. Noll, Matthias J. Sax, Guozhang Wang
Closes#1462 from trybak/bugfix/KAFKA-3784-TimeWindows#windowsFor-false-positives
Currently javadoc doesn't specify charset.
This pull reqeust will set this to UTF-8.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1413 from sasakitoa/javadoc_garbled
I found both issues while investigating the issue described in PR #1425.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#1455 from ijuma/fix-integration-test-harness-and-zk-test-harness
Since the 'time' argument has a default value of -1, it makes sense to make it an optional argument.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1457 from vahidhashemian/KAFKA-3774
This PR simply adds a newline to a Javadoc sample in `KafkaConsumer` to flush the outputs.
Author: Johnny Lim <izeye@naver.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1390 from izeye/patch-1
Trivial patch
is trivial.
Author: Magnus Reftel <magnus.reftel@skatteetaten.no>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1458 from magnusr/feature/paren
`numAcks` is only used in the `trace` logging statement so it should be a `def` instead of a `val`. Also took the chance to improve the code and documentation a little.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1449 from ijuma/minor-avoid-trace-logging-computation-in-partition
This patch fix differentiates between when a consumer group is rebalancing or dead and reports the appropriate error message.
Author: Ishita Mandhan <imandha@us.ibm.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1429 from imandhan/KAFKA-3158
SkimpyOffsetMap.get() when full
Limited number of attempts to number of map slots after the internal
positionOf() goes into linear search mode.
Added unit test
Co-developed with mimaison
Author: edoardo <ecomar@uk.ibm.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1352 from edoardocomar/KAFKA-3682
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1440 from vahidhashemian/typo06/fix_typos_in_code_comments
requestObj() returns null for the o.a.k.c.requests objects so use header() for these.
Once all the requests will have been replaced by o.a.k.c.requests objects, we should be able to clean that up, but in the meantime it's useful to trace both.
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1435 from mimaison/kafkaapis_trace
ExecutorService needs to be shutdown on close, lest a zombie thread
prevent clean shutdown.
ewencp
Author: Peter Davis <peter.davis@expeditors.com>
Reviewers: Liquan Pei <liquanpei@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1383 from davispw/KAFKA-3710
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1401 from christian-posta/ceposta-connect-class-cast-error
To be consistent with `ConfigCommand` and `TopicCommand`.
No release includes this option yet, so we can simply change it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Mickael Maison, Grant Henke
Closes#1430 from ijuma/use-force-instead-of-yes-in-acl-command and squashes the following commits:
bdf3a57 [Ismael Juma] Update `AclCommandTest`
78b8467 [Ismael Juma] Change variable name to `forceOpt`
0bb27af [Ismael Juma] Use `--force` instead of `--yes` in `AclCommand`
junrao
Currently, the broker state is set to running before it registers itself in ZooKeeper. This is too early in the broker lifecycle. If clients use the broker state as an indicator that the broker is ready to accept requests, they will get errors. This change is to delay setting the broker state to running until it's registered in ZK.
Author: Roger Hoover <roger.hoover@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1426 from theduderog/broker-running-after-zk
Kafka has two configurable compression codecs: the one used by the client (source codec) and the one finally used when storing into the log (target codec). The target codec defaults to KafkaConfig.compressionType and can be dynamically configured through zookeeper.
The GroupCoordinator appends group membership information into the __consumer_offsets topic by:
1. making a message with group membership information
2. making a MessageSet with the single message compressed with the source codec
3. doing a log.append on the MessageSet
Without this patch, KafkaConfig.offsetsTopicCompressionCodec doesn't get propagated to OffsetConfig instantiation, so GroupMetadataManager uses a source codec of NoCompressionCodec when making the MessageSet. Let's say we have enough group information such that the message formed exceeds KafkaConfig.messageMaxBytes before compression but would fall below the threshold after compression using our source codec. Even if we had dynamically configured __consumer_offsets with our favorite compression codec, the log.append will throw RecordTooLargeException during analyzeAndValidateMessageSet since the message was unexpectedly uncompressed instead of having been compressed with the source codec defined by KafkaConfig.offsetsTopicCompressionCodec.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1394 from onurkaraman/KAFKA-3718
Added a new argument to AclCommand: --yes. When set, automatically answer yes to prompts
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Gwen Shapira
Closes#1406 from mimaison/KAFKA-3732
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1422 from enothereska/minor-integration-timeout2
Adding sizing recommendations for file descriptors to the ops guide.
Author: Dustin Cote <dustin@confluent.io>
Author: Dustin Cote <dustin@dustins-mbp.attlocal.net>
Reviewers: Gwen Shapira
Closes#1353 from cotedm/KAFKA-3683 and squashes the following commits:
8120318 [Dustin Cote] Adding file descriptor sizing recommendations
0908aa9 [Dustin Cote] Merge https://github.com/apache/kafka into trunk
32315e4 [Dustin Cote] Merge branch 'trunk' of https://github.com/cotedm/kafka into trunk
13309ed [Dustin Cote] Update links for new consumer API
4dcffc1 [Dustin Cote] Update links for new consumer API
Process grep command has been updated. Previous "ps | grep server-1.properties" command is showing nothing.
Author: Satendra Kumar <satendra@knoldus.com>
Reviewers: Gwen Shapira
Closes#1386 from satendrakumar06/patch-1
With this change, `test_producer_throughput` with message_size=10000, compression_type=snappy and a snappy buffer size of 32k can be executed in a heap of 192m in a local environment (768m is needed without this change).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1418 from ijuma/kafka-3747-close-record-batch-when-append-fails
It was previously in the SASL section (probably by mistake).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>
Closes#1405 from ijuma/fix-security-upgrade-location-in-docs
The task is called `aggregatedJavadoc` and the generated html will be under `<project.dir>/build/docs/javadoc/`.
I also disabled javadoc for `tools` and `log4j-appender` as they are not public API.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1398 from ijuma/kafka-3717-aggregate-javadoc
We want to phase out `ByteBufferMessageSet` eventually, so new code should favour `Record` where possible.
Also use a fixed timestamp in `testCorruptLz4ProduceRequest` to ensure that
the checksum is always the same.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#1357 from ijuma/produce-request-test-improvement
These dependencies are unnecessary and they are acquired
transitively via zkclient (jline, netty) and reflections (annotations).
Ewen did the hard work in figuring out why we have unexpected
additional dependencies since 0.9.x.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang, Gwen Shapira
Closes#1396 from ijuma/exclude-jline-netty-deps-in-streams and squashes the following commits:
3aa366f [Ismael Juma] Exclude findbugs annotations due to LGPL license
2d3d714 [Ismael Juma] Use local exclusion for `jline` and `netty`
482b6c0 [Ismael Juma] Exclude `jline` and `netty` dependencies in the `streams` project
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1060 from MayureshGharat/KAFKA-3393
This is not an exact revert as the code changed a bit since the
original commit. We also include a note in `upgrade.html`.
The original commit is 1182d61deb.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Guozhang Wang
Closes#1391 from ijuma/kafka-3704-revert and squashes the following commits:
7891b67 [Ismael Juma] Tweak upgrade note based on Gwen's feedback
1673cd0 [Ismael Juma] Revert "KAFKA-3704: Remove hard-coded block size in KafkaProducer"
Arun Mahadevan
Edoardo Comar
Tom Rybak
Sasaki Toru
Guozhang Wang
Greg Fodor
Ismael Juma
Matthias J. Sax
Vahid Hashemian
Johnny Lim
Flavio Junqueira
Magnus Reftel
Kaufman Ng
Liquan Pei
Joshi
Ishita Mandhan
Eno Thereska
Mickael Maison
manuzhang
leisore
Peter Davis
Christian Posta
Yukun Guo
Roger Hoover
Onur Karaman
Dustin Cote
Satendra Kumar
Sriharsha Chintalapani
shikhar
Geoff Anderson
Jiangjie Qin
MayureshGharat