Delay the cleanup of state directories that are not locked and not owned by the current thread such that we only remove the directory if its last modified is < now - cleanupDelayMs.
This also helps to avoid a race between threads on the same instance, where during rebalance, one thread releases the lock on the state directory, and before another thread can take the lock, the cleanup runs and removes the data.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2486 from dguy/KAFKA-4724
This is a refactoring follow-up of https://github.com/apache/kafka/pull/2166. Main refactoring changes:
1. Extract `InMemoryKeyValueStore` out of `InMemoryKeyValueStoreSupplier` and remove its duplicates in test package.
2. Add two abstract classes `AbstractKeyValueIterator` and `AbstractKeyValueStore` to collapse common functional logics.
3. Added specialized `BytesXXStore` to accommodate cases where key value types are Bytes / byte[] so that we can save calling the dummy serdes.
4. Make the key type in `ThreadCache` from byte[] to Bytes, as SessionStore / WindowStore's result serialized bytes are in the form of Bytes anyways, so that we can save unnecessary `Bytes.get()` and `Bytes.wrap(bytes)`.
Each of these should arguably be a separate PR and I apologize for the mess, this is because this branch was extracted from a rather large diff that has multiple refactoring mingled together and dguy and myself have already put lots of efforts to break it down to a few separate PRs, and this is the only left-over work. Such PR won't happen in the future.
Ping dguy enothereska mjsax for reviews
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Jun Rao
Closes#2333 from guozhangwang/K3452-followup-state-store-refactor
Exception paths in `register()`, `topic()`, `partition()`, `offset()`, and `timestamp()`, were not covered by any existing tests
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Matthias J. Sax
Closes#2447 from dguy/KAFKA-4646
Add coverage for exception paths in `initialize()`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
Closes#2452 from dguy/kafka-4649
…tReset
Author: bbejeck <bbejeck@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Guozhang Wang
Closes#2464 from bbejeck/KAFKA-4662_improve_topology_builder_test_coverage
Author: Maysam Yabandeh <myabandeh@dropbox.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#2474 from ijuma/kafka-4039-deadlock-during-shutdown
The issue of transiently having duplicates is due to the bad design of the left join itself: in order to ignore the partial joined results such as `A:null`, it lets the producer to potentially send twice to source stream one and rely on all the following conditions to be true in order to pass the test:
1. `receiveMessages` happen to have fetched all the produced results and have committed offsets.
2. streams app happen to have completed sending all result data.
3. consumer used in `receiveMessages` will complete getting all messages in a single poll().
If any of the above is not true, the test fails.
Fixed this test to add a filter right after left join to filter out partial joined results. Minor cleanup on integration test utils.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Ewen Cheslack-Postava
Closes#2485 from guozhangwang/K3896-duplicate-join-results
the toString method prints the topology, but had no tests making sure it works and/or doesn't cause exceptions
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Guozhang Wang
Closes#2444 from dguy/KAFKA-4645
Add a test to ensure a `StreamsException` is thrown when an exception other than `StreamsException` is caught
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2450 from dguy/KAFKA-4647
Most of the exception paths weren't covered. Now they are.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes#2442 from dguy/KAFKA-4642
Kafka brokers have a config called "offsets.topic.replication.factor" that specify the replication factor for the "__consumer_offsets" topic. The problem is that this config isn't being enforced. If an attempt to create the internal topic is made when there are fewer brokers than "offsets.topic.replication.factor", the topic ends up getting created anyway with the current number of live brokers. The current behavior is pretty surprising when you have clients or tooling running as the cluster is getting setup. Even if your cluster ends up being huge, you'll find out much later that __consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement on the internal topic is another way of saying the cluster isn't fully setup yet.
The right behavior should be for "offsets.topic.replication.factor" to be enforced. Topic creation of the internal topic should fail with GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor" requirement is met. This closely resembles the behavior of regular topic creation when the requested replication factor exceeds the current size of the cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2177 from onurkaraman/KAFKA-3959
Makes task assignment more sticky by preferring to assign tasks to clients that had previously had the task as active task. If there are no clients with the task previously active, then search for a standby. Finally falling back to the least loaded client.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2429 from dguy/kafka-4677
Author: Jeyhun Karimov <je.karimov@gmail.com>
Reviewers: Matthias J. Sax, Eno Thereska, Damian Guy, Guozhang Wang
Closes#2420 from jeyhunkarimov/KAFKA-4613
- also deprecate ZK config for Streams
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Guozhang Wang
Closes#2459 from mjsax/javaDocImprovements8
In some Places for the loop was used but it can be replaced by the for each.
In One file if else if else was used so I replaced the same with match.
Author: Akash Sethi <akash.sethi@knoldus.in>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2435 from akashsethi24/trunk
Some exception paths not previously covered. Extracted `ensureCopartitioning` into a static class.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2448 from dguy/KAFKA-4644
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2403 from mjsax/addStreamsClientCompatibilityTest
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#2406 from ijuma/kafka-4636-per-listener-security-settings
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Xavier Léauté <xavier@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2454 from guozhangwang/KMinor-trace-logging-add-metrics-twice
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska, Damian Guy, Guozhang Wang
Closes#2441 from ijuma/streams-kafka-client-drops-security-configs
`bin-kafka-console-producer.sh` should be `bin/kafka-console-producer.sh`.
Author: Will Marshall <wcm214@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2410 from wmarshall484/typo-fix
This is the HOTFIX PR for any issues detected with KIP-104 until code freeze. Note: do not merge until close to code freeze.
The name changes reflect feedback received while writing the documentation.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Dan Norwood, Guozhang Wang
Closes#2398 from enothereska/hotfix-streams-metrics
Found a few recently added unit tests did not close KStreamTestDriver after the test itself is closed; this can cause RocksDB virtual function called if the contained topology has some persistent store since they will be initialized but not closed in time.
MINOR fix: found that when closing KStreamTestDriver, we need to first flushing all stores before closing any of them; this is triggered from the `KTableKTableLeftJoin.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions`.
MINOR fix: in CachingXXXStore, the `name` field is actually used as the cache's namespace, not really the store name or its corresponding topic name. Fixed it by renaming it to `cacheName` and use `this.name()` elsewhere which will call the underlying store's name.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ewen Cheslack-Postava, Eno Thereska, Damian Guy
Closes#2432 from guozhangwang/K3502-kstream-builder-test
close the sessions store in `After` to release rocksdb resources.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2430 from dguy/minor-close-sesion-store
The root cause of this issue is that in InternalTopicManager we are creating topics one-at-a-time, and for this test, there are 31 topics to be created, as a result it is possible that the consumer could time out during the assignment in rebalance, and the next leader has to do the same again because of "makeReady" calls are one-at-a-time.
This patch batches the topics into a single create request and also use the StreamsKafkaClient directly to fetch metadata for validating the created topics. Also optimized a bunch of inefficient code in InternalTopicManager and StreamsKafkaClient.
Minor cleanup: make the exception message more informative in integration tests.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Jason Gustafson
Closes#2405 from guozhangwang/K3896-fix-kstream-repartition-join-test
- also some code refactoring and bug fixes
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2337 from mjsax/javaDocImprovements4
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2401 from mjsax/kafka-4671-window-retention-policy
1. In StreamThread, always use subscribe(Pattern, ..) function in order to avoid sending MetadataRequest with specific topic names and cause brokers to possibly auto-create subscribed topics; the pattern is generated as "topic-1|topic-2..|topic-n".
2. In ConsumerCoordinator, let the leader to refresh its metadata if the generated assignment contains some topics that is not contained in the subscribed topics; also in SubscriptionState, modified the verification for regex subscription to against the regex pattern instead of the matched topics since the returned assignment may contain some topics not yet created when joining the group but existed after the rebalance; also modified some unit tests in `KafkaConsumerTest` to accommodate the above changes.
3. Minor cleanup: changed String[] to List<String> to avoid overloaded functions.
4. Minor cleanup: enforced strong typing in SinkNodeFactory and removed unnecessary unchecked tags.
5. Minor cleanup: augmented unit test error message and fixed a potential transient failure in KafkaStreamTest.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2379 from guozhangwang/K4633-regex-pattern
Variance changes introduced in KIP-100 cause compilation failures with lambda expression in Java 8. To my knowledge this only affects the following method
`KStreams.transform(TransformerSupplier<...>, String...)`
prior to the changes it was possible to write:
`streams.transform(MyTransformer::new)`
where `MyTransformer` extends `Transformer`
After the changes the Java compiler is unable to infer correct return types for the lambda expressions. This change fixed this by reverting to invariant return types for transformer suppliers.
please cherry-pick into 0.10.2.x
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Ismael Juma, Damian Guy, Guozhang Wang
Closes#2402 from xvrl/lambdas-oh-my
ZK removed reveal a bug in `StreamPartitionAssigner` but did not fix it properly. This is a follow up bug fix.
Issue:
- If topic metadata is missing, `StreamPartitionAssigner` should not create any affected tasks that consume topics with missing metadata.
- Depending downstream tasks should not be create either.
- For tasks that are not created, no store changelog topics (if any) should get created
- For tasks that write output to not-yet existing internal repartitioning topics, those repartitioning topics should not get created
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#2404 from mjsax/kafka-4060-zk-test-follow-up
This log message tends to be extremely verbose when state stores are being restored
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2412 from xvrl/reduce-verbosity
Re-branched the trunk and applied the changes to the new branch to simplify commit log.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2389 from hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-followup-from-trunk
Address Ismael's comments upon merging
This class doesn't need to override this method as it is handled appropriately by the super class
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2397 from dguy/hotfix-npe-state-store
interface for `Processor` in comments incorrectly had `transform` rather than `process`.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Ismael Juma <ismael@juma.me.uk>
Closes#2396 from dguy/minor-javadoc
In RocksDBStore, options / wOptions / fOptions are constructed in the constructor, which needs to be dismissed in the close() call; however in some tests, the generated topology is not initialized at all, and hence the corresponding state stores are supposed to not be able to be closed as well since their `init` function is not called. This could cause the above option objects to be not released.
This is fixed in this patch to move the logic out of constructor and inside `init` functions, so that no RocksDB objects will be created in the constructor only. Also some minor cleanups:
1. In KStreamTestDriver.close(), we lost the logic to close the state stores but only call `flush`; it is now changed back to call both.
2. Moved the forwarding logic from KStreamTestDriver to MockProcessorContext to remove the mutual dependency: these functions should really be in ProcessorContext, not the test driver.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2381 from guozhangwang/K3502-pure-virtual-function-unit-tests
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2391 from mjsax/kafka-4060-zk-follow-up-system-tests
This is a follow up of https://github.com/apache/kafka/pull/2166 - refactoring the store hierarchies as requested
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2360 from dguy/state-store-refactor
After debugging this i can see the times that it fails there is a race between when the topic is actually created/ready on the broker and when the assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets called with a `Cluster` with no topics. Hence the test hangs as no tasks get assigned. To fix this I added a `waitForTopics` method to `EmbeddedKafkaCluster`. This will wait until the topics have been created.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2371 from dguy/integration-test-fix
Remove applicationId parameter as it is no longer used.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2385 from dguy/minor-remove-unused-param
Damian Guy
Guozhang Wang
Bill Bejeck
Maysam Yabandeh
Onur Karaman
Matthias J. Sax
Jeyhun Karimov
Akash Sethi
Ismael Juma
Will Marshall
Gwen Shapira
Eno Thereska
Xavier Léauté
Hojjat Jafarpour