* Added StateStoreSupplier
* StateStore
* Added init(ProcessorContext context) method
* TopologyBuilder
* Added addStateStore(StateStoreSupplier supplier, String... processNames)
* Added connectProessorAndStateStores(String processorName, String... stateStoreNames)
* This is for the case processors are not created when a store is added to the topology. (used by KStream)
* KStream
* add stateStoreNames to process(), transform(), transformValues().
* Refactored existing state stores to implement StateStoreSupplier
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#387 from ymatsuda/state_store_supplier
Use IBM ConfigFile class with IBM JDK since JavaLoginConfig provided by SUN provider is not included with IBM JDK.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Flavio Junqueira <fpj@apache.org>, Jun Rao <junrao@gmail.com>
Closes#357 from rajinisivaram/KAFKA-2680
onurkaraman Could you have a look? This is the patch I discussed with you.
Author: Dong Lin <lindong28@gmail.com>
Author: Dong Lin <lindong@cis.upenn.edu>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Joel Koshy <jjkoshy@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#316 from lindong28/KAFKA-2660
Updated kafka-producer-perf-test.sh to use org.apache.kafka.clients.tools.ProducerPerformance.
Updated build.gradle to add kafka-tools-0.9.0.0-SNAPSHOT.jar to kafka/libs folder.
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Gwen Shapira, Ismael Juma
Closes#242 from omkreddy/KAFKA-2562
ewencp Nothing too complicated here
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava, Gwen Shapira
Closes#392 from granders/minor-remove-system-test
Also refactor `KerberosNameParser` and `KerberosName` to make the code
clearer and easier to use when `shortName` is not needed.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#390 from ijuma/kafka-2711
ewencp
Some *.properties files were missing from `kafkatest` package. This update makes `kafkatest` services once again useable by external dependencies
I've tested this change on aws with confluent system tests
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava
Closes#385 from granders/minor-update-test-manifest
Followed the approach specified here: https://issues.apache.org/jira/browse/KAFKA-2502
I also made a minor fix to ConfigCommand to expose the right options on add-config.
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Gwen Shapira
Closes#381 from auradkar/K-2502
ewencp
Updated the provisioning script to install ntp daemon.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Gwen Shapira
Closes#383 from granders/minor-systest-clock-skew
This has 2 fixes:
KAFKA-2664 - This patch changes the underlying map implementation of Metrics.java to a ConcurrentHashMap. Using a CopyOnWriteMap caused new metrics creation to get extremely slow when the existing corpus of metrics is large. Using a ConcurrentHashMap seems to speed up metric creation time significantly
KAFKA-2663 - Splitting out the throttleTime from the remote time. On throttled requests, the remote time went up artificially.
Some status on using a ConcurrentMap. Time to create :
- 100k sensors (1.5 seconds)
- 200k sensors (3 seconds)
- 400k sensors (9 seconds)
- 500k sensors (14 seconds)
Please refer this test (originally written by Joel) http://pastebin.com/LnKjbY9a
Author: Aditya Auradkar <aauradkar@linkedin.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>
Closes#369 from auradkar/K-2664
…Some bug fixes that I encountered while running the tests.
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#300 from Parth-Brahmbhatt/KAFKA-2598
KAFKA-2644 adds MiniKdc for system tests and hence needs a target to collect all MiniKdc jars. At the moment, system tests run `gradlew jar`. Replacing that with `gradlew systemTestLibs` will enable kafka jars and test dependency jars to be built and copied into appropriate locations. Submitting this as a separate PR so that the new target can be added to the build scripts that run system tests before KAFKA-2644 is committed. A separate target for system test artifacts will allow dependency changes to be made in future without breaking test runs.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#361 from rajinisivaram/kafka-systemTestLibs
I've added a couple of initial tests to verify the functionality. I've tested that the JAAS config file loads properly and SASL with DIGEST-MD5 works with ZooKeeper.
Author: Flavio Junqueira <fpj@apache.org>
Author: flavio junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#324 from fpj/KAFKA-2640
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#376 from ijuma/KAFKA-2675-sasl-kerberos-follow-up
There are several gauges in core that are registered but cannot be accessed programmatically. For example, gauges "LeaderCount", "PartitionCount", "UnderReplicatedParittions" are all registered in ReplicaManager.scala but there is no way to access them programmatically if one has access to the kafka.server object. Other metrics, such as isrExpandRate (also in ReplicaManager.scala) can be accessed. The solution here is trivial, add a var <variable name> in front of newGauge, as shown below
val partitionCount newGauge(
"PartitionCount",
new Gauge[Int] {
def value = allPartitions.size
}
)
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#364 from enothereska/gauges
ewencp gwenshap
This needs some refactoring to avoid the duplicated code between replication test and upgrade test, but in shape for initial feedback.
I'm interested in feedback on the added `KafkaConfig` class and `kafka_props` file. This addition makes it:
- easier to attach different configs to different nodes (e.g. during broker upgrade process)
- easier to reason about the configuration of a particular node
Notes:
- in the default values in the KafkaConfig class, I removed many properties which were in kafka.properties before. This is because most of those properties were set to what is already the default value.
- when running non-trunk VerifiableProducer, I append the trunk tools jar to the classpath, and run it with the non-trunk kafka-run-class.sh script
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Dong Lin, Ewen Cheslack-Postava
Closes#229 from granders/KAFKA-1888-upgrade-test
StreamThread should keep calling consumer.poll() even when no task is assigned. This is necessary to get a task.
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#373 from ymatsuda/no_task
The branch wasn't rebased after the capitalisation fix for SSL
classes.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#368 from ijuma/fix-kafka-log4j-appender-compiler-error
guozhangwang
* A task id is now a class, ```TaskId```, that has a topic group id and a partition id fields.
* ```TopologyBuilder``` assigns a topic group id to a topic group. Related methods are changed accordingly.
* A state store uses the partition id part of the task id as the change log partition id.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#365 from ymatsuda/task_id
guozhangwang
* added ```PartitionGrouper``` (abstract class)
* This class is responsible for grouping partitions. Each group forms a task.
* Users may implement this class for custom grouping.
* added ```DefaultPartitionGrouper```
* our default implementation of ```PartitionGrouper```
* added ```KafkaStreamingPartitionAssignor```
* We always use this as ```PartitionAssignor``` of stream consumers.
* Actual grouping is delegated to ```PartitionGrouper```.
* ```TopologyBuilder```
* added ```topicGroups()```
* This returns groups of related topics according to the topology
* added ```copartitionSources(sourceNodes...)```
* This is used by DSL layer. It asserts the specified source nodes must be copartitioned.
* added ```copartitionGroups()```
* This returns groups of copartitioned topics
* KStream layer
* keep track of source nodes to determine copartition sources when steams are joined
* source nodes are set to null when partitioning property is not preserved (ex. ```map()```, ```transform()```), and this indicates the stream is no longer joinable
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#353 from ymatsuda/grouping
This adds coordination between DistributedHerders using the generalized consumer
support, allowing automatic balancing of connectors and tasks across workers. A
few pieces that require interaction between workers (resolving config
inconsistencies, forwarding of configuration changes to the leader worker) are
incomplete because they require REST API support to implement properly.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson, Gwen Shapira
Closes#321 from ewencp/kafka-2371-distributed-herder
This pull request adds a configuration parameter and a migration tool. It is also based on pull request #303, which should go in first.
Author: flavio junqueira <fpj@apache.org>
Author: Flavio Junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#313 from fpj/KAFKA-2641
Yasuhiro Matsuda
Gwen Shapira
Rajini Sivaram
Dong Lin
Ashish Singh
Manikumar reddy O
Geoff Anderson
Ewen Cheslack-Postava
Anna Povzner
Ismael Juma
Aditya Auradkar
Joel Koshy
Flavio Junqueira
Parth Brahmbhatt
Jason Gustafson
Guozhang Wang
Eno Thereska
Jiangjie Qin
Grant Henke