Also removed the code for handling negative timeouts in `blockingReady` as `Selector.poll` has not supported that for a while.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1177 from ijuma/kafka-3495-blocking-send-and-receive-request-timeout
Before this patch arrays containing primitive types were not output:
```
Metadata Request (Version: 0) => [topics]
```
After this patch the type is listed:
```
Metadata Request (Version: 0) => [topics]
topics => STRING
```
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ashish Singh, Gwen Shapira
Closes#1174 from granthenke/protocol-arrays
This also fixes KAFKA-3453 and KAFKA-2866.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1155 from ijuma/kafka-3475-introduce-our-minikdc
Also:
* Rename remaining `CoreUtils.rm` to `delete` for consistency
* Use `try with resources` in `Utils` to simplify code
* Silence compiler warning due to exception catch clause in `TestUtils`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1153 from ijuma/remove-redundant-core-utils-rm
This was recently introduced in:
1fbe445dde
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1152 from ijuma/fix-typos-in-record-accumulator
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1056 from becketqin/KAFKA-3388
This should help when diagnosing issues with the console producer. This allows the logger to use `exception` rather than `exception.getMessage()`.
Author: Jeremy Custenborder <jcustenborder@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1079 from jcustenborder/KAFKA-3407
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1123 from hachikuji/KAFKA-3434
Replace `update` with `withPartitions`, which returns a copy instead of mutating the instance.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1118 from ijuma/kafka-3432-cluster-update-thread-safety
Also included a minor efficiency improvement in kafka.cluster.EndPoint.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1105 from ijuma/kafka-3431-replace-broker-end-point-with-node
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke, Gwen Shapira
Closes#1100 from ijuma/kafka-3426-invalid-protocol-type-errors-invalid-sizes
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1108 from hachikuji/KAFKA-3412
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1091 from granthenke/fetch-error
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Larkin Lowrey <llowrey@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1103 from ijuma/kafka-3378-follow-up
This is a different implementation to the one in #1085 by Larkin Lowrey (llowrey). The hard part here was actually finding the problem and all credit goes to llowrey.
This PR also fixes our handling of `finishConnect` (we now check the return value).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao
Closes#1094 from ijuma/KAFKA-3378-instantly-connecting-socket-channels
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>, Jun Rao <junrao@gmail.com>
Closes#1064 from hachikuji/KAFKA-3394
A new consumer config option 'exclude.internal.topics' was added to
allow excluding internal topics when wildcards are used to specify
consumers.
The new option takes a boolean value, with a default 'false' value (i.e.
no exclusion).
This patch is co-authored with rajinisivaram edoardocomar mimaison
Author: edoardo <ecomar@uk.ibm.com>
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma, Jun Rao, Gwen Shapira
Closes#1082 from edoardocomar/KAFKA-2832
This contribution is my original work, and I license it under the project's open source license.
CC jkreps
Author: Drausin Wulsin <daedalus2718@gmail.com>
Author: John Doe <daedalus2718@gmail.com>
Reviewers: Jason Gustafson
Closes#1055 from drausin/bugfix/consumer-records-iterator
This is a KIP-42 followup.
Currently, If sending the record fails before it gets to the server, ProducerInterceptor.onAcknowledgement() is called with metadata == null, and non-null exception. However, it is useful to pass topic and partition, if known, to ProducerInterceptor.onAcknowledgement() as well. This patch ensures that ProducerInterceptor.onAcknowledgement() gets record metadata with topic and maybe partition. If partition is not set in 'record' and KafkaProducer.send() fails before partition gets assigned, then ProducerInterceptor.onAcknowledgement() gets RecordMetadata with partition == -1. Only time when ProducerInterceptor.onAcknowledgement() gets null record metadata is when the client passes null record to KafkaProducer.send().
Author: Anna Povzner <anna@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ashish Singh <asingh@cloudera.com>, Jun Rao <junrao@gmail.com>
Closes#1015 from apovzner/kip42-3
Please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment for the overall design.
The update to TopicMetadataRequest/TopicMetadataResponse will be done in a different PR.
Author: Allen Wang <awang@netflix.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#132 from allenxwang/KAFKA-1215
Added topic-partition information to the exception message on batch expiry in RecordAccumulator
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Gwen Shapira, Lin Dong, Ismael Juma
Closes#695 from MayureshGharat/kafka-3013
* Fix and suppress number of unchecked warnings (except for Kafka Streams)
* Add `SafeVarargs` annotation to fix warnings
* Suppress unfixable deprecation warnings
* Replace deprecated by non-deprecated usage where possible
* Avoid reflective calls via structural types in Scala
* Tweak compiler settings for scalac and javac
Once we drop Java 7 and Scala 2.10, we can tweak the compiler settings further so that they warn us about more things.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke, Gwen Shapira, Guozhang Wang
Closes#1042 from ijuma/kafka-3375-suppress-depreccated-tweak-compiler
- Moves all generated docs under /docs/generated
- Generates docs for Protocol, Errors, and ApiKeys
- Adds new protocol.html page
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#970 from granthenke/protocol-doc-wip
By using `getHostString` (introduced in Java 7) instead of `getHostName`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Grant Henke
Closes#1030 from ijuma/kafka-3352-avoid-dns-reverse-look-ups
JAAS configuration may be set using other methods and hence the check for System property doesn't always match where the actual configuration used by Kafka is loaded from.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>, Flavio Junqueira <fpj@apache.org>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#967 from rajinisivaram/KAFKA-3279
Remove test cases testInvalidDefaultRange() and testInvalidDefaultString(). Defaults if not overridden will get checked on parse. Testing the defaults is unnecessary. This allows you to set that a parameter is required while setting a validator for that parameter. Added a test case testNullDefaultWithValidator that allows a null default with a validator for certain strings.
Author: Jeremy Custenborder <jcustenborder@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#936 from jcustenborder/KAFKA-3237
This patch reuse max.in.flight.request.per.connection. When it equals to one, we take it as user wants order protection. The current approach is make sure there is only one batch per partition on the fly.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#857 from becketqin/KAFKA-3197
* Include request id when parsing of request header fails
* Don't mute selector on a connection that was closed due to an error (otherwise a second exception is thrown)
* Throw appropriate exception from `ApiKeys.fromId` if invalid id is passed
* Fail fast in `AbstractRequest.getRequest` if we fail to handle an instance of `ApiKeys` (if this happens, it's a programmer error and the code in `getRequest` needs to be updated)
I ran into the top two issues while trying to figure out why a connection from a producer to a broker was failing (and it made things harder than necessary). While fixing them, I noticed the third and fourth issues.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1017 from ijuma/kafka-3341-improve-error-handling-invalid-requests
In version of 0.8.2.1, the old consumer will provide the metrics reporter per-topic consumer metrics under group 'ConsumerTopicMetrics'. For example:
*.ConsumerTopicMetrics.clientId.[topic name].BytesPerSec.count
*.ConsumerTopicMetrics.clientId.[topic name].MessagesPerSec.count
These consumer metrics are useful since it helps us monitor consumer rate for each topic. But the new consumer(0.9.0.0) doesn't expose per topic metrics anymore, even though I did find sensor objects in consumer metrics object collecting per-topic metrics.
After investigation, I found that these sensors are not registering any KafkaMetrics.
Author: Yifan Ying <yying@fitbit.com>
Reviewers: Grant Henke, Jason Gustafson, Guozhang Wang
Closes#939 from happymap/KAFKA-3233
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Yasuhiro Matsuda <yasuhiro.matsuda@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#990 from guozhangwang/K3311
and remove TOTAL_RECORDS_TO_PROCESS
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#985 from ymatsuda/config_params
This is my original work and I license the work to the project under the project's open source license.
Author: Richard Whaling <rwhaling@spantree.net>
Reviewers: Jason Gustafson, Gwen Shapira
Closes#968 from rwhaling/docs/kafkaconsumer-heartbeat-doc-improvement
Author: Tom Lee <github@tomlee.co>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Jiangjie Qin <jiangjie@linkedin.com>, Grant Henke <ghenke@cloudera.com>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#962 from hachikuji/KAFKA-2698
Ismael Juma
Jason Gustafson
Grant Henke
Jiangjie Qin
Jeremy Custenborder
Ewen Cheslack-Postava
Pierre-Yves Ritschard
Guozhang Wang
Liquan Pei
edoardo
Drausin Wulsin
Anna Povzner
Allen Wang
MayureshGharat
Rajini Sivaram
Matthias J. Sax
Ashish Singh
Yifan Ying
kenji yoshida
Yasuhiro Matsuda
Richard Whaling
Tom Lee