In `wrapForOutput` and `wrapForInput` methods of `org.apache.kafka.common.record.Compressor`, `Class.forName("[compression codec]")` and `getConstructor` methods are invoked for each `wrapForOutput` / `wrapForInput` call. Reflection calls are expensive and impact performance at high volumes. This patch adds a cache for `Constructor` to reduce the reflection overhead.
In our production deployments, this has reduced producer CPU usage by about 20%
Author: Maksim Logvinenko <mlogvinenko@gmail.com>
Reviewers: Ismael Juma
Closes#580 from logarithm/compressor-getclass-cache
Sanity check array size in requests before allocation
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ashish Singh <asingh@cloudera.com>, Jun Rao <junrao@gmail.com>
Closes#577 from rajinisivaram/KAFKA-2878
Also mark `PrincipalBuilder` as `Unstable` and tweak docs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#542 from ijuma/kafka-2847-remove-principal-builder-class-from-client-configs
This avoids spurious log warning messages. Also tweak log message
if wrapResult.getStatus != CLOSED.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jun Rao <junrao@gmail.com>
Closes#511 from ijuma/kafka-2817-unconnected-ssl-transport-layer-close
Added PASSWORD_STRING in ConfigDef that returns "[hidden]" when method toString is invoked.
Author: Jakub Nowak <jakub.nowak94@interia.pl>
Reviewers: Ismael Juma, Gwen Shapira, Jun Rao
Closes#371 from Mszak/ssl-password-protection
Patched Selector.poll() to close the connection on any exception.
Author: Jun Rao <junrao@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Gwen Shapira <cshapi@gmail.com>
Closes#501 from junrao/KAFKA-2813
Removed the check for expiring only those batches whose metadata is unavailable. Now the batches will be expired irrespective of whether the leader is available or not, as soon as it reaches the requestimeout threshold.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#503 from MayureshGharat/kafka-2805
Process any remaining data in the network read buffer in `SslTransportLayer` when `read()` is invoked. On handshake completion, there could be application data ready to be processed that was read into `netReadBuffer` during handshake processing. `read()` is already invoked from `Selector` after handshake completion, but data already read into the `netReadBuffer` was not being processed. This PR adds a check for remaining data and continues with processing data if data is available.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#493 from rajinisivaram/KAFKA-2801
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ben Stopford, Geoff Anderson, Guozhang Wang
Closes#432 from ewencp/kafka-2752-copycat-clean-bounce-test
-- Refactored multi-consumer integration group assignment validation tests for round-robin assignment
-- Added multi-consumer integration tests for session timeout expiration:
1. When a consumer stops polling
2. When a consumer calls close()
-- Fixes to issues found with session timeout expiration tests woth help from Jason Gustafson: Try to avoid SendFailedException exception by cancelling the scheduled tasks and ensuring metadata update before sending group leave requests + send leave group request with retries.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#472 from apovzner/cpkafka-81
Close socket channel in finally block to avoid file descriptor leak when remote end closes the connection
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#460 from rajinisivaram/KAFKA-2779
Use IBM ConfigFile class with IBM JDK since JavaLoginConfig provided by SUN provider is not included with IBM JDK.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Flavio Junqueira <fpj@apache.org>, Jun Rao <junrao@gmail.com>
Closes#357 from rajinisivaram/KAFKA-2680
Also refactor `KerberosNameParser` and `KerberosName` to make the code
clearer and easier to use when `shortName` is not needed.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#390 from ijuma/kafka-2711
This has 2 fixes:
KAFKA-2664 - This patch changes the underlying map implementation of Metrics.java to a ConcurrentHashMap. Using a CopyOnWriteMap caused new metrics creation to get extremely slow when the existing corpus of metrics is large. Using a ConcurrentHashMap seems to speed up metric creation time significantly
KAFKA-2663 - Splitting out the throttleTime from the remote time. On throttled requests, the remote time went up artificially.
Some status on using a ConcurrentMap. Time to create :
- 100k sensors (1.5 seconds)
- 200k sensors (3 seconds)
- 400k sensors (9 seconds)
- 500k sensors (14 seconds)
Please refer this test (originally written by Joel) http://pastebin.com/LnKjbY9a
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>
Closes#369 from auradkar/K-2664
…Some bug fixes that I encountered while running the tests.
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#300 from Parth-Brahmbhatt/KAFKA-2598
I've added a couple of initial tests to verify the functionality. I've tested that the JAAS config file loads properly and SASL with DIGEST-MD5 works with ZooKeeper.
Author: Flavio Junqueira <fpj@apache.org>
Author: flavio junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#324 from fpj/KAFKA-2640
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#376 from ijuma/KAFKA-2675-sasl-kerberos-follow-up
Alex Sherwin
Tomasz Nurkiewicz
Maksim Logvinenko
Jesse Anderson
Rajini Sivaram
jholoman
Jason Gustafson
Ismael Juma
Jakub Nowak
Jun Rao
Mayuresh Gharat
Ewen Cheslack-Postava
Gwen Shapira
Guozhang Wang
Anna Povzner
Grant Henke
Matthew Bruce
Aditya Auradkar
Parth Brahmbhatt
Flavio Junqueira