- Added validity checks for input parameters on subscribe, assign to avoid NPE, and provide an argument exception instead
- Updated behavior for subscription with null collection to be same as when subscription with emptyList.i.e., unsubscribes.
- Added tests on subscription, assign
Author: Rekha Joshi <rekhajoshm@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1601 from rekhajoshm/KAFKA-3905-1
Full credit for figuring out the cause of these failures goes to hachikuji.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Guozhang Wang, Ismael Juma, Jason Gustafson
Closes#1594 from vahidhashemian/KAFKA-3931
Fix timing window in producer by holding onto cluster object while processing send requests so that changes to cluster during metadata refresh don't cause NPE if a topic is deleted.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1478 from rajinisivaram/KAFKA-3562
This patch fixes two issues:
1. Subsequent regex subscriptions fail with the new consumer.
2. Subsequent regex subscriptions would not immediately refresh metadata to change the subscription of the new consumer and trigger a rebalance.
The final note on the JIRA stating that a later created topic that matches a consumer's subscription pattern would not be assigned to the consumer upon creation seems to be as designed. A repeat
`subscribe()` to the same pattern or some wait time until the next automatic metadata refresh would handle that case.
An integration test was also added to verify these issues are fixed with this PR.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1572 from vahidhashemian/KAFKA-3854
…stUtils, added method for pausing tests to TestUtils
Changes made:
1. Added utility method for creating consumer configs.
2. Added methods for creating producer, consumer configs with default values for de/serializers.
3. Pulled out method for waiting for test state to TestUtils (not using Thread.sleep).
4. Added utility class for creating streams configs and methods providing default de/serializers.
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1532 from bbejeck/KAFKA_3842_add_helper_functions_test_utils
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1500 from vahidhashemian/typo07/fix_javadoc_typos_consumerrebalancelistener
guozhangwang enothereska mjsax miguno
If you get a chance can you please take a look at this. I've done the repartitioning in the join, but it results in 2 internal topics for each join. This seems like overkill as sometimes we wouldn't need to repartition at all, others just 1 topic, and then sometimes both, but I'm not sure how we can know that.
I'd also need to implement something similar for leftJoin, but again, i'd like to see if i'm heading down the right path or if anyone has any other bright ideas.
For reference - https://github.com/apache/kafka/pull/1453 - the previous PR
Thanks for taking the time and looking forward to getting some welcome advice :-)
Author: Damian Guy <damian.guy@gmail.com>
Author: Damian Guy <damian@continuum.local>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1472 from dguy/KAFKA-3561
Follow up on KAFKA-724 (#1469) to allow OS socket buffer sizes auto tuning for both the broker and the clients.
Author: Sebastien Launay <sebastien@opendns.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1507 from slaunay/enhancement/os-socket-buffer-size-tuning-for-clients
This PR is the follow on to the closed PR #1410.
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1477 from bbejeck/KAFKA-3443_streams_support_for_regex_sources
Only log the client and server principals, which is what ZooKeeper does after ZOOKEEPER-2405.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#1498 from ijuma/kafka-3830-get-tgt-debug-confidential
See https://issues.apache.org/jira/browse/KAFKA-3711
I've tested locally that this change does indeed resolve the warning I mention in the ticket:
```
org.apache.kafka.clients.consumer.ConsumerConfig: The configuration metric.dropwizard.registry = kafka-metrics was supplied but isn't a known config.
```
where `metric.dropwizard.registry` is a configuration value defined in a custom `MetricReporter` (https://github.com/SimpleFinance/kafka-dropwizard-reporter).
With this change, the above warning no longer appears, as ewencp predicted.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1479 from jklukas/abstractconfig-originals
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1476 from ijuma/kafka-3781-exception-name-npe
If socket.receive.buffer.bytes/socket.send.buffer.bytes are set to -1, use the OS defaults.
Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1469 from rekhajoshm/KAFKA-724-rebased
If no messages are sent to a topic during the last refresh interval or if UNKNOWN_TOPIC_OR_PARTITION error is received, remove the topic from the metadata list. Topics are added to the list on the next attempt to send a message to the topic.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Author: rsivaram <rsivaram@uk.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#645 from rajinisivaram/KAFKA-2948
This PR simply adds a newline to a Javadoc sample in `KafkaConsumer` to flush the outputs.
Author: Johnny Lim <izeye@naver.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1390 from izeye/patch-1
Trivial patch
is trivial.
Author: Magnus Reftel <magnus.reftel@skatteetaten.no>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1458 from magnusr/feature/paren
With this change, `test_producer_throughput` with message_size=10000, compression_type=snappy and a snappy buffer size of 32k can be executed in a heap of 192m in a local environment (768m is needed without this change).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1418 from ijuma/kafka-3747-close-record-batch-when-append-fails
We want to phase out `ByteBufferMessageSet` eventually, so new code should favour `Record` where possible.
Also use a fixed timestamp in `testCorruptLz4ProduceRequest` to ensure that
the checksum is always the same.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#1357 from ijuma/produce-request-test-improvement
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1060 from MayureshGharat/KAFKA-3393
This is not an exact revert as the code changed a bit since the
original commit. We also include a note in `upgrade.html`.
The original commit is 1182d61deb23b5cd86cbe462471f7df583a796e1.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Guozhang Wang
Closes#1391 from ijuma/kafka-3704-revert and squashes the following commits:
7891b67 [Ismael Juma] Tweak upgrade note based on Gwen's feedback
1673cd0 [Ismael Juma] Revert "KAFKA-3704: Remove hard-coded block size in KafkaProducer"
This fixes test_producer_throughput with compression_type=snappy.
Also: added heap dump on out of memory error to `producer_performance.py` and corrected the upgrade note related to the change in buffer size for compression streams.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1385 from ijuma/kafka-3713-test_producer_throughput-snappy-fail and squashes the following commits:
54c7962 [Ismael Juma] Correct upgrade note about buffer size for compression stream
515040b [Ismael Juma] Call `compressor.close()` to fix memory leak
5311e5b [Ismael Juma] Dump heap on out of memory error when running `producer_performance.py`
This PR loosens access restrictions on `WrapperSerde` to `protected` so that users can define a `Serdes` subclass that provides additional custom serde members following the same pattern as the parent class.
This is my own work and is compatible with Kafka's license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Guozhang Wang
Closes#1382 from jklukas/expose-wrapperserde
Propagate IOException in SslTransportLayer channel.close to be consistent with PlaintextTransportLayer, close authenticator on channel close even if transport layer close fails.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1370 from rajinisivaram/minor-channelclose2
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1354 from hachikuji/minor-add-missing-assignor-tostrings
Add references to the new consumer property 'max.partition.fetch.bytes' along with the old consumer property 'fetch.message.max.bytes' in the corresponding warning messages of TopicCommand.
Also, create and leverage a static variable for the default value of the new consumer property.
Also, use 'DEFAULT_...' for default propoerty constant names in the code instead of '..._DEFAULT'.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Ashish Singh <asingh@cloudera.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1239 from vahidhashemian/KAFKA-3579
This contribution is my original work and I license the work under Apache 2.0.
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1212 from dpkp/KAFKA-3160
so that unresolvable DNS names are ignored and only throw an error if no other bootstrap servers are resolvable.
Author: Jonathan Bond <jbond@netflix.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#792 from bondj/KAFKA-3112
Whenever the BufferPool throws a "Failed to allocate memory within the configured max blocking time" exception, it should also remove the condition object from the waiters deque
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Chen Zhu <amandazhu19620701@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1314 from MayureshGharat/kafka-3651
AbstractPartitionAssignor.assign has an ambiguous line in its documentation:
> param partitionsPerTopic The number of partitions for each subscribed topic (may be empty for some topics)
Does empty mean the topic has an entry with value zero, or that the entry is excluded from the map altogether? The current implementation in AbstractPartitionAssignor excludes the entry from partitionsPerTopic if the topic isn't in the metadata.
RoundRobinAssignorTest.testOneConsumerNonexistentTopic interprets emptiness as providing the topic with a zero value.
RangeAssignor interprets emptiness as excluding the entry from the map.
RangeAssignorTest.testOneConsumerNonexistentTopic interprets emptiness as providing the topic with a zero value.
This implementation chooses to solve the NPE by deciding to exclude topics from partitionsPerTopic when the topic is not in the metadata.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1326 from onurkaraman/KAFKA-3661
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1322 from hachikuji/KAFKA-3659
Handle unsupported version of ApiVersionsRequest during SASL auth as well as normal operation by returning an error response.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1310 from rajinisivaram/KAFKA-3652
Rekha Joshi
Vahid Hashemian
Grant Henke
Rajini Sivaram
Sriharsha Chintalapani
Kenji Yoshida
Bill Bejeck
Damian Guy
Sebastien Launay
Ismael Juma
Guozhang Wang
Jeff Klukas
Liquan Pei
Greg Fodor
Johnny Lim
Magnus Reftel
manuzhang
Yukun Guo
MayureshGharat
Jason Gustafson
Dana Powers
Jonathan Bond
Chen Zhu
Onur Karaman