Two tests:
1. One consumer subscribes to 2 topics, each with 2 partitions; includes adding and removing a topic.
2. Several consumers subscribe to 2 topics, several partition each; includes adding one more consumer after initial assignment is done and verified.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Guozhang Wang
Closes#413 from apovzner/cpkafka-76
The patch has two changes:
1. fixed a bug in controller that it sends UpdateMetadataRequest of all the partitions in the cluster.
2. Uses the following rules to propagate ISR change: 1) if there are ISR changes pending propagation and the last ISR change is more than five seconds ago, propagate the changes. 2) if there is ISR change at T in the recent five seconds, delay the propagation until T + 5s. 3) if the last propagation is more than 1 min ago, ignore rule No.2 and propagate ISR change if there are changes pending propagation.
This algorithm avoids a fixed configuration of ISR propagation interval as we discussed about in KIP-29.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#402 from becketqin/KAFKA-2722
Use IBM ConfigFile class with IBM JDK since JavaLoginConfig provided by SUN provider is not included with IBM JDK.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Flavio Junqueira <fpj@apache.org>, Jun Rao <junrao@gmail.com>
Closes#357 from rajinisivaram/KAFKA-2680
onurkaraman Could you have a look? This is the patch I discussed with you.
Author: Dong Lin <lindong28@gmail.com>
Author: Dong Lin <lindong@cis.upenn.edu>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Joel Koshy <jjkoshy@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#316 from lindong28/KAFKA-2660
Followed the approach specified here: https://issues.apache.org/jira/browse/KAFKA-2502
I also made a minor fix to ConfigCommand to expose the right options on add-config.
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Gwen Shapira
Closes#381 from auradkar/K-2502
This has 2 fixes:
KAFKA-2664 - This patch changes the underlying map implementation of Metrics.java to a ConcurrentHashMap. Using a CopyOnWriteMap caused new metrics creation to get extremely slow when the existing corpus of metrics is large. Using a ConcurrentHashMap seems to speed up metric creation time significantly
KAFKA-2663 - Splitting out the throttleTime from the remote time. On throttled requests, the remote time went up artificially.
Some status on using a ConcurrentMap. Time to create :
- 100k sensors (1.5 seconds)
- 200k sensors (3 seconds)
- 400k sensors (9 seconds)
- 500k sensors (14 seconds)
Please refer this test (originally written by Joel) http://pastebin.com/LnKjbY9a
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>
Closes#369 from auradkar/K-2664
…Some bug fixes that I encountered while running the tests.
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#300 from Parth-Brahmbhatt/KAFKA-2598
I've added a couple of initial tests to verify the functionality. I've tested that the JAAS config file loads properly and SASL with DIGEST-MD5 works with ZooKeeper.
Author: Flavio Junqueira <fpj@apache.org>
Author: flavio junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#324 from fpj/KAFKA-2640
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#376 from ijuma/KAFKA-2675-sasl-kerberos-follow-up
There are several gauges in core that are registered but cannot be accessed programmatically. For example, gauges "LeaderCount", "PartitionCount", "UnderReplicatedParittions" are all registered in ReplicaManager.scala but there is no way to access them programmatically if one has access to the kafka.server object. Other metrics, such as isrExpandRate (also in ReplicaManager.scala) can be accessed. The solution here is trivial, add a var <variable name> in front of newGauge, as shown below
val partitionCount newGauge(
"PartitionCount",
new Gauge[Int] {
def value = allPartitions.size
}
)
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#364 from enothereska/gauges
This pull request adds a configuration parameter and a migration tool. It is also based on pull request #303, which should go in first.
Author: flavio junqueira <fpj@apache.org>
Author: Flavio Junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#313 from fpj/KAFKA-2641
Probably happened while resolving conflicts, commit: 86eb74d923
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson
Closes#350 from ijuma/restore-ssl-consumer-test
Changes in this patch are:
1. ClientIdConfigHandler now passes through the config changes to the quota manager.
2. Removed static KafkaConfigs for quota overrides. These are no longer needed since we can override configs through ZooKeeper.
3. Added testcases to verify that the config changes are propogated from ZK (written using AdminTools) to the actual Metric objects.
Author: Aditya Auradkar <aauradka@aauradka-mn1.(none)>
Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz>
Reviewers: Dong Lin <lindong28@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#298 from auradkar/K-2209
This fix applies to three JIRAs, since they are all connected.
KAFKA-2459Connection backoff/blackout period should start when a connection is disconnected, not when the connection attempt was initiated
Backoff when connection is disconnected
KAFKA-2615Poll() method is broken wrt time
Added Time through the NetworkClient API. Minimal change.
KAFKA-1843Metadata fetch/refresh in new producer should handle all node connection states gracefully
I’ve partially addressed this for a specific failure case in the JIRA.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ewen Cheslack-Postava, Jason Gustafson, Ismael Juma, Guozhang Wang
Closes#290 from enothereska/trunk
- Both TopicCommand and ConfigCommand warn if message.max.bytes increases
- Log failures on the broker if replication gets stuck due to an oversized message
- Added blocking call to warning.
Author: benstopford <benstopford@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#322 from benstopford/CPKAFKA-61
This PR implements SASL/Kerberos which was originally submitted by harshach as https://github.com/apache/kafka/pull/191.
I've been submitting PRs to Harsha's branch with fixes and improvements and he has integrated all, but the most recent one. I'm creating this PR so that the Jenkins can run the tests on the branch (they pass locally).
Author: Ismael Juma <ismael@juma.me.uk>
Author: Sriharsha Chintalapani <harsha@hortonworks.com>
Author: Harsha <harshach@users.noreply.github.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Parth Brahmbhatt <brahmbhatt.parth@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#334 from ijuma/KAFKA-1686-V1
LogCleanerIntegrationTest calls LogCleaner.awaitCleaned() to wait until cleaner has processed up to given offset. However, existing awaitCleaned() implementation doesn't wait for this. This patch fix the problem.
Author: Dong Lin <lindong@cis.upenn.edu>
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#327 from lindong28/KAFKA-2669
Removed default hardcoded keystore and truststore in /tmp so that default JVM keystore/truststore may be used when keystore/truststore is not specified in Kafka server or client properties
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#312 from rajinisivaram/KAFKA-2656
Before we switched from `BlockingChannel` to `NetworkClient`, we were
always reporting a successful connection due to the fact that
`BlockingChannel.connect` catches and swallows all exceptions. We
are now reporting failures (which is better), but `error` seems too
noisy (as can be seen in our tests).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#280 from ijuma/reduce-connection-failure-logging-level
I've split the work of KAFKA-1695 because this refactoring touches a large number of files. Most of the changes are trivial, but I feel it will be easier to review this way.
This pull request includes the one Parth-Brahmbhatt started to address KAFKA-1695.
Author: flavio junqueira <fpj@apache.org>
Author: Flavio Junqueira <fpj@apache.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#303 from fpj/KAFKA-2639
Let's say every consumer in a group has session timeout s. Currently, if a consumer leaves the group, the worst case time to stabilize the group is 2s (s to detect the consumer failure + s for the rebalance window). If a consumer instead can declare they are leaving the group, the worst case time to stabilize the group would just be the s associated with the rebalance window.
This is a low priority optimization!
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#103 from onurkaraman/leave-group
See here for more discussion: https://issues.apache.org/jira/browse/KAFKA-2419
Basically, the fix involves adding a param to Metrics to indicate if it is capable of metric cleanup or not.
Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#323 from auradkar/KAFKA-2419-fix
This is a minimal revert of some backward incompatible changes made in KAFKA-2205, with the addition of the deprecation logging message.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#305 from granthenke/topic-configs
Added --timeout-ms argument to ConsoleConsumer that works with both old and new consumer. Also modified ducktape ConsoleConsumer service to use this arg instead of consumer.timeout.ms config that works only with the old consumer.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Aditya Auradkar, Ismael Juma, Guozhang Wang
Closes#274 from rajinisivaram/KAFKA-2603
Enables Cipher suite setting. Code was previously reviewed by ijuma, harshach. Moving to an independent PR.
Author: benstopford <benstopford@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#301 from benstopford/cipher-switch
Parametrize console consumer sanity test, replication tests and benchmarks tests to run with both PLAINTEXT and SSL.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Geoff Anderson, Ewen Cheslack-Postava, Guozhang Wang
Closes#271 from rajinisivaram/KAFKA-2581
This is a followup ticket from KAFKA-2084 to improve the windowSize calculation in Quotas. I've made the following changes:
1. Added a windowSize function on Rate
2. Calling Rate.windowSize in ClientQuotaManager to return the exact window size to use when computing the delay time.
3. Changed the window size calculation subtly. The current calculation had a bug wherein, it used the number of elapsed seconds from the "lastWindowSeconds" of the most recent Sample object. However, the lastWindowSeconds is the time when the sample is created.. this causes an issue because it implies that the current window elapsed time is always "0" when the sample is created. This is incorrect as demonstrated in a testcase I added in MetricsTest. I've fixed the calculation to count the elapsed time from the "oldest" sample in the set since that gives us an accurate value of the exact amount of time elapsed
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#213 from auradkar/K-2443
This restores the behaviour before 1265d7cb7.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#294 from ijuma/fix-processor-thread-name
* Call `ConnectionQuotas.decr` when calling `Selector.close` and when disconnections happen.
* Expand `SocketServerTest` to test for this and to close sockets.
* Refactor and clean-up `SocketServer` and `Acceptor` to make the code easier to understand.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#288 from ijuma/kafka-2614-connection-count-not-updated
Jason Gustafson
Guozhang Wang
Anna Povzner
Jiangjie Qin
Rajini Sivaram
Dong Lin
Aditya Auradkar
Joel Koshy
Flavio Junqueira
Parth Brahmbhatt
Ewen Cheslack-Postava
Ismael Juma
Eno Thereska
Ashish Singh
Dong Lin
Aditya Auradkar
benstopford
Sriharsha Chintalapani
Onur Karaman
Aditya Auradkar
Manikumar reddy O
Grant Henke