guozhangwang enothereska mjsax miguno
If you get a chance can you please take a look at this. I've done the repartitioning in the join, but it results in 2 internal topics for each join. This seems like overkill as sometimes we wouldn't need to repartition at all, others just 1 topic, and then sometimes both, but I'm not sure how we can know that.
I'd also need to implement something similar for leftJoin, but again, i'd like to see if i'm heading down the right path or if anyone has any other bright ideas.
For reference - https://github.com/apache/kafka/pull/1453 - the previous PR
Thanks for taking the time and looking forward to getting some welcome advice :-)
Author: Damian Guy <damian.guy@gmail.com>
Author: Damian Guy <damian@continuum.local>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1472 from dguy/KAFKA-3561
Follow up on KAFKA-724 (#1469) to allow OS socket buffer sizes auto tuning for both the broker and the clients.
Author: Sebastien Launay <sebastien@opendns.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1507 from slaunay/enhancement/os-socket-buffer-size-tuning-for-clients
This PR is the follow on to the closed PR #1410.
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1477 from bbejeck/KAFKA-3443_streams_support_for_regex_sources
Only log the client and server principals, which is what ZooKeeper does after ZOOKEEPER-2405.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#1498 from ijuma/kafka-3830-get-tgt-debug-confidential
See https://issues.apache.org/jira/browse/KAFKA-3711
I've tested locally that this change does indeed resolve the warning I mention in the ticket:
```
org.apache.kafka.clients.consumer.ConsumerConfig: The configuration metric.dropwizard.registry = kafka-metrics was supplied but isn't a known config.
```
where `metric.dropwizard.registry` is a configuration value defined in a custom `MetricReporter` (https://github.com/SimpleFinance/kafka-dropwizard-reporter).
With this change, the above warning no longer appears, as ewencp predicted.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1479 from jklukas/abstractconfig-originals
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1476 from ijuma/kafka-3781-exception-name-npe
If socket.receive.buffer.bytes/socket.send.buffer.bytes are set to -1, use the OS defaults.
Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1469 from rekhajoshm/KAFKA-724-rebased
If no messages are sent to a topic during the last refresh interval or if UNKNOWN_TOPIC_OR_PARTITION error is received, remove the topic from the metadata list. Topics are added to the list on the next attempt to send a message to the topic.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Author: rsivaram <rsivaram@uk.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#645 from rajinisivaram/KAFKA-2948
This PR simply adds a newline to a Javadoc sample in `KafkaConsumer` to flush the outputs.
Author: Johnny Lim <izeye@naver.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1390 from izeye/patch-1
Trivial patch
is trivial.
Author: Magnus Reftel <magnus.reftel@skatteetaten.no>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1458 from magnusr/feature/paren
With this change, `test_producer_throughput` with message_size=10000, compression_type=snappy and a snappy buffer size of 32k can be executed in a heap of 192m in a local environment (768m is needed without this change).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1418 from ijuma/kafka-3747-close-record-batch-when-append-fails
We want to phase out `ByteBufferMessageSet` eventually, so new code should favour `Record` where possible.
Also use a fixed timestamp in `testCorruptLz4ProduceRequest` to ensure that
the checksum is always the same.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#1357 from ijuma/produce-request-test-improvement
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1060 from MayureshGharat/KAFKA-3393
This is not an exact revert as the code changed a bit since the
original commit. We also include a note in `upgrade.html`.
The original commit is 1182d61deb.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Guozhang Wang
Closes#1391 from ijuma/kafka-3704-revert and squashes the following commits:
7891b67 [Ismael Juma] Tweak upgrade note based on Gwen's feedback
1673cd0 [Ismael Juma] Revert "KAFKA-3704: Remove hard-coded block size in KafkaProducer"
This fixes test_producer_throughput with compression_type=snappy.
Also: added heap dump on out of memory error to `producer_performance.py` and corrected the upgrade note related to the change in buffer size for compression streams.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1385 from ijuma/kafka-3713-test_producer_throughput-snappy-fail and squashes the following commits:
54c7962 [Ismael Juma] Correct upgrade note about buffer size for compression stream
515040b [Ismael Juma] Call `compressor.close()` to fix memory leak
5311e5b [Ismael Juma] Dump heap on out of memory error when running `producer_performance.py`
This PR loosens access restrictions on `WrapperSerde` to `protected` so that users can define a `Serdes` subclass that provides additional custom serde members following the same pattern as the parent class.
This is my own work and is compatible with Kafka's license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Guozhang Wang
Closes#1382 from jklukas/expose-wrapperserde
Propagate IOException in SslTransportLayer channel.close to be consistent with PlaintextTransportLayer, close authenticator on channel close even if transport layer close fails.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1370 from rajinisivaram/minor-channelclose2
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1354 from hachikuji/minor-add-missing-assignor-tostrings
Add references to the new consumer property 'max.partition.fetch.bytes' along with the old consumer property 'fetch.message.max.bytes' in the corresponding warning messages of TopicCommand.
Also, create and leverage a static variable for the default value of the new consumer property.
Also, use 'DEFAULT_...' for default propoerty constant names in the code instead of '..._DEFAULT'.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Ashish Singh <asingh@cloudera.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1239 from vahidhashemian/KAFKA-3579
This contribution is my original work and I license the work under Apache 2.0.
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1212 from dpkp/KAFKA-3160
so that unresolvable DNS names are ignored and only throw an error if no other bootstrap servers are resolvable.
Author: Jonathan Bond <jbond@netflix.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#792 from bondj/KAFKA-3112
Whenever the BufferPool throws a "Failed to allocate memory within the configured max blocking time" exception, it should also remove the condition object from the waiters deque
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Chen Zhu <amandazhu19620701@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1314 from MayureshGharat/kafka-3651
AbstractPartitionAssignor.assign has an ambiguous line in its documentation:
> param partitionsPerTopic The number of partitions for each subscribed topic (may be empty for some topics)
Does empty mean the topic has an entry with value zero, or that the entry is excluded from the map altogether? The current implementation in AbstractPartitionAssignor excludes the entry from partitionsPerTopic if the topic isn't in the metadata.
RoundRobinAssignorTest.testOneConsumerNonexistentTopic interprets emptiness as providing the topic with a zero value.
RangeAssignor interprets emptiness as excluding the entry from the map.
RangeAssignorTest.testOneConsumerNonexistentTopic interprets emptiness as providing the topic with a zero value.
This implementation chooses to solve the NPE by deciding to exclude topics from partitionsPerTopic when the topic is not in the metadata.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1326 from onurkaraman/KAFKA-3661
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1322 from hachikuji/KAFKA-3659
Handle unsupported version of ApiVersionsRequest during SASL auth as well as normal operation by returning an error response.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1310 from rajinisivaram/KAFKA-3652
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1311 from guozhangwang/K3639
`maxTimeToBlock` needs to be updated in each loop iteration. Also record waitTime before throwing `TimeoutException`
Author: Chen Zhu <amandazhu19620701@gmail.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1304 from zhuchen1018/KAFKA-3648
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1306 from Ishiihara/minor-consumer-cleanup
When an address is written textually, the zone index is appended to the address, separated by a percent sign (%). The actual syntax of zone indices depends on the operating system.
Author: Som Sahu <sosahu@microsoft.com>
Author: Soumyajit Sahu <soumyajit-sahu@users.noreply.github.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1305 from soumyajit-sahu/fixIPV6RegexPattern_trunk
Also:
* Remove redundant `time.milliseconds` call in `Sensor.record`
* Clean-up a number of tests and remove a manual test that is no longer required
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Liquan Pei <liquanpei@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1265 from ijuma/kafka-3128-zookeeper-metrics
this is consistent with the Max stat implementation.
Author: Zack Dever <zdever@pandora.com>
Reviewers: Aditya Auradkar, Gwen Shapira
Closes#1143 from zackdever/min-stat-fix
Server-side implementation and tests for handling ApiVersionsRequest before SaslHandshakeRequest.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Gwen Shapira, Ismael Juma
Closes#1286 from rajinisivaram/KAFKA-3618
Grant Henke
Damian Guy
Sebastien Launay
Rajini Sivaram
bbejeck
Ismael Juma
Guozhang Wang
Jeff Klukas
Liquan Pei
Joshi
Greg Fodor
Johnny Lim
Magnus Reftel
manuzhang
Yukun Guo
MayureshGharat
Jason Gustafson
Vahid Hashemian
Dana Powers
Jonathan Bond
Chen Zhu
Onur Karaman
Ewen Cheslack-Postava
Som Sahu
Yuto Kawamura
Zack Dever