Removed default hardcoded keystore and truststore in /tmp so that default JVM keystore/truststore may be used when keystore/truststore is not specified in Kafka server or client properties
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#312 from rajinisivaram/KAFKA-2656
I've split the work of KAFKA-1695 because this refactoring touches a large number of files. Most of the changes are trivial, but I feel it will be easier to review this way.
This pull request includes the one Parth-Brahmbhatt started to address KAFKA-1695.
Author: flavio junqueira <fpj@apache.org>
Author: Flavio Junqueira <fpj@apache.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#303 from fpj/KAFKA-2639
Let's say every consumer in a group has session timeout s. Currently, if a consumer leaves the group, the worst case time to stabilize the group is 2s (s to detect the consumer failure + s for the rebalance window). If a consumer instead can declare they are leaving the group, the worst case time to stabilize the group would just be the s associated with the rebalance window.
This is a low priority optimization!
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#103 from onurkaraman/leave-group
See here for more discussion: https://issues.apache.org/jira/browse/KAFKA-2419
Basically, the fix involves adding a param to Metrics to indicate if it is capable of metric cleanup or not.
Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#323 from auradkar/KAFKA-2419-fix
Trivial fix to get rid of unused statements in kafkaProducer.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Edward Ribeiro, Guozhang Wang
Closes#320 from MayureshGharat/kafka-2120-followup
This also adds some other needed infrastructure for distributed Copycat, most
importantly the DistributedHerder, and refactors some code for handling
Kafka-backed logs into KafkaBasedLog since this is shared betweeen offset and
config storage.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira, James Cheng
Closes#241 from ewencp/kafka-2372-copycat-distributed-config
Enables Cipher suite setting. Code was previously reviewed by ijuma, harshach. Moving to an independent PR.
Author: benstopford <benstopford@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#301 from benstopford/cipher-switch
This is a followup ticket from KAFKA-2084 to improve the windowSize calculation in Quotas. I've made the following changes:
1. Added a windowSize function on Rate
2. Calling Rate.windowSize in ClientQuotaManager to return the exact window size to use when computing the delay time.
3. Changed the window size calculation subtly. The current calculation had a bug wherein, it used the number of elapsed seconds from the "lastWindowSeconds" of the most recent Sample object. However, the lastWindowSeconds is the time when the sample is created.. this causes an issue because it implies that the current window elapsed time is always "0" when the sample is created. This is incorrect as demonstrated in a testcase I added in MetricsTest. I've fixed the calculation to count the elapsed time from the "oldest" sample in the set since that gives us an accurate value of the exact amount of time elapsed
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#213 from auradkar/K-2443
* Call `ConnectionQuotas.decr` when calling `Selector.close` and when disconnections happen.
* Expand `SocketServerTest` to test for this and to close sockets.
* Refactor and clean-up `SocketServer` and `Acceptor` to make the code easier to understand.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#288 from ijuma/kafka-2614-connection-count-not-updated
Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket for Kafka-2120.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Dong Lin, Ismael Juma, Guozhang Wang
Closes#282 from MayureshGharat/Kafka-2428
As discussed in KAFKA-2419 - I've added a time based sensor retention config to Sensor. Sensors that have not been "recorded" for 'n' seconds are eligible for expiration.
In addition to the time based retention, I've also altered several tests to close the Metrics and scheduler objects since they can cause leaks while running tests. This causes TestUtils.verifyNonDaemonThreadStatus to fail.
Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz>
Author: Aditya Auradkar <aauradka@aauradka-mn1.(none)>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#233 from auradkar/K-2419
Unit tests which mock buffer overflow and underflow in the SSL transport layer and fixes for the couple of issues in buffer overflow handling described in the JIRA.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#205 from rajinisivaram/KAFKA-2534
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava, Jason Gustafson, Guozhang Wang
Closes#272 from ijuma/kafka-2640-remove-complete-all-poll-timeout
Remove state storage upon unclean shutdown and fix streaming metrics used for local state.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Edward Ribeiro, Yasuhiro Matsuda, Jun Rao
Closes#265 from guozhangwang/K2591
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Ben Stopford <benstopford@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#273 from ijuma/kafka-2517-ssl-zero-copy-regression
It compares upper bound with itself.
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Reviewers: Aditya Auradkar, Ismael Juma, Guozhang Wang
Closes#182 from eribeiro/equals-bug
This work has been contributed by Jesse Anderson, Randall Hauch, Yasuhiro Matsuda and Guozhang Wang. The detailed design can be found in https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client.
Author: Guozhang Wang <wangguoz@gmail.com>
Author: Yasuhiro Matsuda <yasuhiro.matsuda@gmail.com>
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Author: ymatsuda <yasuhiro.matsuda@gmail.com>
Author: Randall Hauch <rhauch@gmail.com>
Author: Jesse Anderson <jesse@smokinghand.com>
Author: Ismael Juma <ismael@juma.me.uk>
Author: Jesse Anderson <eljefe6a@gmail.com>
Reviewers: Ismael Juma, Randall Hauch, Edward Ribeiro, Gwen Shapira, Jun Rao, Jay Kreps, Yasuhiro Matsuda, Guozhang Wang
Closes#130 from guozhangwang/streaming
hachikuji ewencp I found this problem when adding new consumer to mirror maker which commits offset in the rebalance callback. It is not clear to me why we are triggering rebalance for commitSync() and fetchCommittedOffset(). Can you help review to see if I miss something?
Regarding commitSync, After each poll() the partitions will be either assigned to a consumer or it will be already revoked. As long as user is using internal offset map, the offset map will always be valid. i.e. the offset map will always only contain the assigned partitions when commitSync is called. Hence there is no need to trigger a rebalance in commitSync().
The same guarantee also apply to fetchCommittedOffset(), isn't the only requirement is to ensure we know the coordinator?
Another related issue is that today the IllegalGenerationIdException is a bit confusing. When we receive an IllegalGenerationIdException from heartbeat, we need to use that same generation Id to commit offset and the coordinator will take it. So the generation ID was not really illegal. I will file a ticket for this issue.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#221 from becketqin/KAFKA-2555
Author: Dong Lin <lindong28@gmail.com>
Author: Dong Lin <lindong@cis.upenn.edu>
Reviewers: Jason Gustafson, Ismael Juma, Guozhang Wang
Closes#118 from lindong28/KAFKA-2390
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#195 from Parth-Brahmbhatt/KAFKA-2211
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Jun Rao <junrao@gmail.com>
Closes#194 from ijuma/kafka-2440-use-network-client-in-fetcher
A shot to remove commit type from new consumer. The coordinator constructor takes a default offset commit callback mainly for testing purpose.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ewen Cheslack-Postava, Jason Gustafson, Guohang Wang
Closes#134 from becketqin/KAFKA-2389
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Jason Gustafson, Guozhang Wang, Edward Ribeiro, Ismael Juma
Closes#128 from SinghAsDev/KAFKA-1893
ewencp
The changes here are smaller than they look - mostly refactoring/cleanup.
- ConsumerPerformanceService: added new_consumer flag, and exposed more command-line settings
- benchmark.py: refactored to use `parametrize` and `matrix` - this reduced some amount of repeated code
- benchmark.py: added consumer performance tests with new consumer (using `parametrize`)
- benchmark.py: added more detailed test descriptions
- performance.py: broke into separate files
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava, Jason Gustafson, Gwen Shapira
Closes#179 from granders/KAFKA-2489-benchmark-new-consumer
Rajini Sivaram
flavio junqueira
Onur Karaman
Guozhang Wang
Aditya Auradkar
Mayuresh Gharat
Manikumar reddy O
Jason Gustafson
Ewen Cheslack-Postava
benstopford
Aditya Auradkar
Ismael Juma
Edward Ribeiro
Mayuresh Gharat
Jiangjie Qin
Dong Lin
Parth Brahmbhatt
Ashish Singh
Joel Koshy
jholoman
Geoff Anderson