This also adds some other needed infrastructure for distributed Copycat, most
importantly the DistributedHerder, and refactors some code for handling
Kafka-backed logs into KafkaBasedLog since this is shared betweeen offset and
config storage.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira, James Cheng
Closes#241 from ewencp/kafka-2372-copycat-distributed-config
Enables Cipher suite setting. Code was previously reviewed by ijuma, harshach. Moving to an independent PR.
Author: benstopford <benstopford@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#301 from benstopford/cipher-switch
Parametrize console consumer sanity test, replication tests and benchmarks tests to run with both PLAINTEXT and SSL.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Geoff Anderson, Ewen Cheslack-Postava, Guozhang Wang
Closes#271 from rajinisivaram/KAFKA-2581
This patch is different than the one attached to the JIRA - I'm applying the new javadoc rules to all subprojects while the one in the JIRA applies only to "clients". We need this since Copycat has the same issues.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#147 from gwenshap/KAFKA-2203
This is a followup ticket from KAFKA-2084 to improve the windowSize calculation in Quotas. I've made the following changes:
1. Added a windowSize function on Rate
2. Calling Rate.windowSize in ClientQuotaManager to return the exact window size to use when computing the delay time.
3. Changed the window size calculation subtly. The current calculation had a bug wherein, it used the number of elapsed seconds from the "lastWindowSeconds" of the most recent Sample object. However, the lastWindowSeconds is the time when the sample is created.. this causes an issue because it implies that the current window elapsed time is always "0" when the sample is created. This is incorrect as demonstrated in a testcase I added in MetricsTest. I've fixed the calculation to count the elapsed time from the "oldest" sample in the set since that gives us an accurate value of the exact amount of time elapsed
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#213 from auradkar/K-2443
This restores the behaviour before 1265d7cb7.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#294 from ijuma/fix-processor-thread-name
* Call `ConnectionQuotas.decr` when calling `Selector.close` and when disconnections happen.
* Expand `SocketServerTest` to test for this and to close sockets.
* Refactor and clean-up `SocketServer` and `Acceptor` to make the code easier to understand.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#288 from ijuma/kafka-2614-connection-count-not-updated
guozhangwang
* added back type safe stateful transform methods (kstream.transform() and kstream.transformValues())
* changed kstream.process() to void
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#292 from ymatsuda/transform_method
A few of Kafka Stream's interfaces and classes are not as well-aligned with Java 8's functional interfaces. By making these changes, when Kafka moves to Java 8 these classes can extend standard Java 8 functional interfaces while remaining backward compatible. This will make it easier for developers to use Kafka Streams, and may allow us to eventually remove these custom interfaces and just use the standard Java 8 interfaces.
The changes include:
1. The 'apply' method of KStream's `Predicate` functional interface was renamed to `test` to match the method name on `java.util.function.BiPredicate`. This will allow KStream's `Predicate` to extend `BiPredicate` when Kafka moves to Java 8, and for the `KStream.filter` and `filterOut` methods to accept `BiPredicate`.
2. Renamed the `ProcessorDef` and `WindowDef` interfaces to `ProcessorSupplier` and `WindowSupplier`, respectively. Also the `SlidingWindowDef` class was renamed to `SlidingWindowSupplier`, and the `MockProcessorDef` test class was renamed to `MockProcessorSupplier`. The `instance()` method in all were renamed to `get()`, so that all of these can extend/implement Java 8's `java.util.function.Supplier<T>` interface in the future with no other changes and while remaining backward compatible. Variable names that used some form of "def" were changed to use "supplier".
These two sets of changes were made in separate commits.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#270 from rhauch/kafka-2600
guozhangwang
This code change properly types ProcessorDef. This also makes KStream.process() typesafe.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Ismael Juma, Guozhang Wang
Closes#289 from ymatsuda/typing_ProcessorDef
Tried two fixes. I prefer the second approach because it saves an additional offset search.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#204 from becketqin/KAFKA-2477
Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket for Kafka-2120.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Dong Lin, Ismael Juma, Guozhang Wang
Closes#282 from MayureshGharat/Kafka-2428
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#277 from Parth-Brahmbhatt/KAFKA-2587
As discussed in KAFKA-2419 - I've added a time based sensor retention config to Sensor. Sensors that have not been "recorded" for 'n' seconds are eligible for expiration.
In addition to the time based retention, I've also altered several tests to close the Metrics and scheduler objects since they can cause leaks while running tests. This causes TestUtils.verifyNonDaemonThreadStatus to fail.
Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz>
Author: Aditya Auradkar <aauradka@aauradka-mn1.(none)>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#233 from auradkar/K-2419
Unit tests which mock buffer overflow and underflow in the SSL transport layer and fixes for the couple of issues in buffer overflow handling described in the JIRA.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#205 from rajinisivaram/KAFKA-2534
To support Decimal, this also adds support for schema parameters, which is an
extra set of String key value pairs which provide extra information about the
schema. For Decimal, this is used to encode the scale parameter, which is part
of the schema instead of being passed with every value.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Guozhang Wang
Closes#281 from ewencp/kafka-2476-copycat-logical-types
ymatsuda junrao Could you take a quick look? The current unit test is failing on this.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma, Jun Rao
Closes#276 from guozhangwang/HF-ProcessorStateManager
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava, Jason Gustafson, Guozhang Wang
Closes#272 from ijuma/kafka-2640-remove-complete-all-poll-timeout
This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to kafka/docs repository.
Some questions: This PR contains generated javadocs also. Do we need to copy javadocs here?
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Gwen Shapira, Ismael Juma
Closes#171 from omkreddy/KAFKA-2425-MOVE-DOCS-TO-KAFKA-REPO
Remove state storage upon unclean shutdown and fix streaming metrics used for local state.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Edward Ribeiro, Yasuhiro Matsuda, Jun Rao
Closes#265 from guozhangwang/K2591
The ` .metadata` and `.recommenders` keep IDE workspace state and should not be committed.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Guozhang Wang
Closes#259 from rhauch/kafka-2597
I tested and verified that `vagrant --version | egrep -o "\d+\.\d+\.\d+"` works on Mac but failed on RedHad 6.4, while `vagrant --version | egrep -o "[0-9]\.[0-9]\.[0-9]"` works on both OS.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Geoff Anderson, Guozhang Wang
Closes#246 from lindong28/Vagrant-setup-on-Fedora
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Ben Stopford <benstopford@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#273 from ijuma/kafka-2517-ssl-zero-copy-regression
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#230 from Parth-Brahmbhatt/KAFKA-2212
It compares upper bound with itself.
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Reviewers: Aditya Auradkar, Ismael Juma, Guozhang Wang
Closes#182 from eribeiro/equals-bug
They now both use 2 spaces for indents, which is what `checkstyle.xml` was
already doing. `import.xml` had a mixture of tabs and 4 spaces previously.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#253 from ijuma/fix-xml-indents
GC is set to G1 collector.
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#218 from omkreddy/KAFKA-2514
Ewen Cheslack-Postava
benstopford
Gwen Shapira
Rajini Sivaram
Aditya Auradkar
Grant Henke
Ismael Juma
Yasuhiro Matsuda
Randall Hauch
Jason Gustafson
Jiangjie Qin
Mayuresh Gharat
Parth Brahmbhatt
Aditya Auradkar
Ashish Singh
Edward Ribeiro
Guozhang Wang
GabrielNicolasAvellaneda
Manikumar reddy O
Dong Lin
Mayuresh Gharat