Added commitRecord(SourceRecord record) to SourceTask. This method is called during the callback from producer.send() when the message has been sent successfully. Added commitTaskRecord(SourceRecord record) to WorkerSourceTask to handle calling commitRecord on the SourceTask. Updated tests for calls to commitRecord.
Author: Jeremy Custenborder <jcustenborder@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#950 from jcustenborder/KAFKA-3260
Also clean-up `LogTest` a little.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang
Closes#1071 from ijuma/kafka-3047-explicit-offset-assignment-corrupt-log-test
Please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment for the overall design.
The update to TopicMetadataRequest/TopicMetadataResponse will be done in a different PR.
Author: Allen Wang <awang@netflix.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#132 from allenxwang/KAFKA-1215
Added topic-partition information to the exception message on batch expiry in RecordAccumulator
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Gwen Shapira, Lin Dong, Ismael Juma
Closes#695 from MayureshGharat/kafka-3013
* Fix and suppress number of unchecked warnings (except for Kafka Streams)
* Add `SafeVarargs` annotation to fix warnings
* Suppress unfixable deprecation warnings
* Replace deprecated by non-deprecated usage where possible
* Avoid reflective calls via structural types in Scala
* Tweak compiler settings for scalac and javac
Once we drop Java 7 and Scala 2.10, we can tweak the compiler settings further so that they warn us about more things.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke, Gwen Shapira, Guozhang Wang
Closes#1042 from ijuma/kafka-3375-suppress-depreccated-tweak-compiler
guozhangwang Very minor cleanup.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1063 from Ishiihara/minor-cleanup
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <mail@harsha.io>
Closes#1054 from omkreddy/KAFKA-2551
This fix was suggested by Maciek Makowski, who also reported the problem.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1029 from ijuma/KAFKA-3047-log-append-can-corrupt-the-log
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>, Jun Rao <junrao@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1018 from becketqin/KAFKA-2960
AWS instance size increased to m3.xlarge to allow all system tests to pass. ijuma ewencp have a look please.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Gwen Shapira
Closes#1046 from enothereska/minor-aws
Because protocol.html is going to be in its own page it needs the header and footer included.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#1043 from granthenke/protocol-docs-style
- Moves all generated docs under /docs/generated
- Generates docs for Protocol, Errors, and ApiKeys
- Adds new protocol.html page
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#970 from granthenke/protocol-doc-wip
By using `getHostString` (introduced in Java 7) instead of `getHostName`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Grant Henke
Closes#1030 from ijuma/kafka-3352-avoid-dns-reverse-look-ups
When invoking `gradle` on a recent version, it updates `gradlew.bat` to fix a typo. It's an annoyance at development time as it causes a diff on whatever branch one is working on.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1034 from ijuma/update-gradlew.bat
JAAS configuration may be set using other methods and hence the check for System property doesn't always match where the actual configuration used by Kafka is loaded from.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>, Flavio Junqueira <fpj@apache.org>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#967 from rajinisivaram/KAFKA-3279
Remove test cases testInvalidDefaultRange() and testInvalidDefaultString(). Defaults if not overridden will get checked on parse. Testing the defaults is unnecessary. This allows you to set that a parameter is required while setting a validator for that parameter. Added a test case testNullDefaultWithValidator that allows a null default with a validator for certain strings.
Author: Jeremy Custenborder <jcustenborder@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#936 from jcustenborder/KAFKA-3237
Per discussion with guozhangwang, `ignore` failing streams system tests until fix for KAFKA-3354 is checked in.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Guozhang Wang
Closes#1031 from granders/ignore-streams-systest
This patch reuse max.in.flight.request.per.connection. When it equals to one, we take it as user wants order protection. The current approach is make sure there is only one batch per partition on the fly.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#857 from becketqin/KAFKA-3197
Three main tests:
1. Setup: Producer (0.8) → Kafka Cluster → Consumer (0.8)
First rolling bounce: Set inter.broker.protocol.version = 0.8 and message.format.version = 0.8
Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version
2. Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9)
First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9
Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version
3. Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9)
First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9
Second rolling bonus: use inter.broker.protocol.version = 0.10 and message.format.version = 0.9
Plus couple of variations of these tests using old/new consumer and no compression / snappy compression.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#980 from apovzner/kafka-3201-02
…Response with o.a.k.c.requests equivalent
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma
Closes#927 from granthenke/offset-refactor
https://issues.apache.org/jira/browse/KAFKA-1476
Let me know if these kind of contributions should have their own requisite JIRA opened in advance.
Cheers..
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Gwen Shapira
Closes#945 from christian-posta/ceposta-tidy-up-consumer-groups-describe
* Include request id when parsing of request header fails
* Don't mute selector on a connection that was closed due to an error (otherwise a second exception is thrown)
* Throw appropriate exception from `ApiKeys.fromId` if invalid id is passed
* Fail fast in `AbstractRequest.getRequest` if we fail to handle an instance of `ApiKeys` (if this happens, it's a programmer error and the code in `getRequest` needs to be updated)
I ran into the top two issues while trying to figure out why a connection from a producer to a broker was failing (and it made things harder than necessary). While fixing them, I noticed the third and fourth issues.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1017 from ijuma/kafka-3341-improve-error-handling-invalid-requests
In version of 0.8.2.1, the old consumer will provide the metrics reporter per-topic consumer metrics under group 'ConsumerTopicMetrics'. For example:
*.ConsumerTopicMetrics.clientId.[topic name].BytesPerSec.count
*.ConsumerTopicMetrics.clientId.[topic name].MessagesPerSec.count
These consumer metrics are useful since it helps us monitor consumer rate for each topic. But the new consumer(0.9.0.0) doesn't expose per topic metrics anymore, even though I did find sensor objects in consumer metrics object collecting per-topic metrics.
After investigation, I found that these sensors are not registering any KafkaMetrics.
Author: Yifan Ying <yying@fitbit.com>
Reviewers: Grant Henke, Jason Gustafson, Guozhang Wang
Closes#939 from happymap/KAFKA-3233
Stop scritps such as kafka-server-stop.sh log messages of kill command's error when processes aren't running.
This PR changes this message to "No kafka server to stop".
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Gwen Shapira
Closes#971 from sasakitoa/stop_scripts_says_not_good_message
The fix basically ensures that the throttleTimeSensor is non-null before handing off to record the metric value. We also record the throttle time to 0 so that we don't recreate the sensor always.
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#989 from auradkar/KAFKA-3310
Adds a gradle task to generate a report of outdate release dependencies:
`gradle dependencyUpdates`
Updates a few minor versions.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma, Gwen Shapira
Closes#973 from granthenke/outdated-deps
Without this change `./gradlew releaseTarGz` (and its variants) will not include the RocksDB jar, which is required for Kafka Streams, in Kafka's `libs/` folder. The impact is that any Streams job will fail when it runs against a broker that was installed via a release tarball.
guozhangwang junrao : please review.
Author: Michael G. Noll <michael@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1007 from miguno/trunk-rocksdb-fixes
Adjust the listeners property rather than the port. Following the original instructions would result in all of the brokers being started with the same listeners setting, and so fail to work.
Author: Duncan Sands <baldrick@free.fr>
Reviewers: Ismael Juma, Gwen Shapira
Closes#1002 from CunningBaldrick/quickstart
Jeremy Custenborder
Ismael Juma
Yasuhiro Matsuda
Allen Wang
MayureshGharat
Jiangjie Qin
Liquan Pei
Manikumar reddy O
Ashish Singh
Jason Gustafson
Guozhang Wang
Eno Thereska
Gwen Shapira
Grant Henke
Rajini Sivaram
Matthias J. Sax
Geoff Anderson
Anna Povzner
Christian Posta
Yifan Ying
Michael G. Noll
Sasaki Toru
Aditya Auradkar
Duncan Sands