The root cause of this issue is that in InternalTopicManager we are creating topics one-at-a-time, and for this test, there are 31 topics to be created, as a result it is possible that the consumer could time out during the assignment in rebalance, and the next leader has to do the same again because of "makeReady" calls are one-at-a-time.
This patch batches the topics into a single create request and also use the StreamsKafkaClient directly to fetch metadata for validating the created topics. Also optimized a bunch of inefficient code in InternalTopicManager and StreamsKafkaClient.
Minor cleanup: make the exception message more informative in integration tests.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Jason Gustafson
Closes#2405 from guozhangwang/K3896-fix-kstream-repartition-join-test
- also some code refactoring and bug fixes
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2337 from mjsax/javaDocImprovements4
This is a documentation-only patch discussed on the mailing list. The
intent is to have these changes propagated to the protocol wiki
(https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol) .
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#918 from dpkp/produce_response_errors
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2424 from cmccabe/KAFKA-4688
`FileChannel.read` may not fill the destination buffer even if there
are enough bytes in the channel to do so. Add a couple of utility
methods that ensure this and use them from all the relevant places.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2304 from amethystic/kafka4576_FileChannel_read
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2426 from hachikuji/improve-consumer-test-error-messages
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes#2401 from mjsax/kafka-4671-window-retention-policy
1. In StreamThread, always use subscribe(Pattern, ..) function in order to avoid sending MetadataRequest with specific topic names and cause brokers to possibly auto-create subscribed topics; the pattern is generated as "topic-1|topic-2..|topic-n".
2. In ConsumerCoordinator, let the leader to refresh its metadata if the generated assignment contains some topics that is not contained in the subscribed topics; also in SubscriptionState, modified the verification for regex subscription to against the regex pattern instead of the matched topics since the returned assignment may contain some topics not yet created when joining the group but existed after the rebalance; also modified some unit tests in `KafkaConsumerTest` to accommodate the above changes.
3. Minor cleanup: changed String[] to List<String> to avoid overloaded functions.
4. Minor cleanup: enforced strong typing in SinkNodeFactory and removed unnecessary unchecked tags.
5. Minor cleanup: augmented unit test error message and fixed a potential transient failure in KafkaStreamTest.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2379 from guozhangwang/K4633-regex-pattern
Updated the docs with changes in KAFKA-4060.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Author: Hojjat Jafarpour <hojjat@HojjatJpoursMBP.attlocal.net>
Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang
Closes#2377 from hjafarpour/KAFKA-4060-docs-update
Author: Magnus Reftel <magnus.reftel@skatteetaten.no>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2151 from reftel/feature/authorizer_name_reference
This is a KIP-104/105 follow-up. Thanks to ijuma for pointing out.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2350 from enothereska/minor-broker-level-config
Variance changes introduced in KIP-100 cause compilation failures with lambda expression in Java 8. To my knowledge this only affects the following method
`KStreams.transform(TransformerSupplier<...>, String...)`
prior to the changes it was possible to write:
`streams.transform(MyTransformer::new)`
where `MyTransformer` extends `Transformer`
After the changes the Java compiler is unable to infer correct return types for the lambda expressions. This change fixed this by reverting to invariant return types for transformer suppliers.
please cherry-pick into 0.10.2.x
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Ismael Juma, Damian Guy, Guozhang Wang
Closes#2402 from xvrl/lambdas-oh-my
ZK removed reveal a bug in `StreamPartitionAssigner` but did not fix it properly. This is a follow up bug fix.
Issue:
- If topic metadata is missing, `StreamPartitionAssigner` should not create any affected tasks that consume topics with missing metadata.
- Depending downstream tasks should not be create either.
- For tasks that are not created, no store changelog topics (if any) should get created
- For tasks that write output to not-yet existing internal repartitioning topics, those repartitioning topics should not get created
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy, Guozhang Wang
Closes#2404 from mjsax/kafka-4060-zk-test-follow-up
Renames `HoistToStruct` SMT to `HoistField`.
Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`
Adds HTML doc generation and updates to `connect.html`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2374 from shikhar/more-smt
This log message tends to be extremely verbose when state stores are being restored
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2412 from xvrl/reduce-verbosity
Current implementation of ProducerPerformance creates static payload. This is not very useful in testing compression or when you want to test with production/custom payloads. So, we decided to add support for providing payload file as an input to producer perf test script.
We made the following changes:
1. Added support to provide a payload file which can have the list of payloads that you actually want to send.
2. Moved payload generation inside the send loop for cases when payload file is provided.
Following are the changes to how the producer-performance is evoked:
1. You must provide "--record-size" or "--payload-file" but not both. This is because, record size cannot be guaranteed when you are using custom events.
e.g. ./kafka-producer-perf-test.sh --topic test_topic --num-records 100000 --producer-props bootstrap.servers=127.0.0.1:9092 acks=0 buffer.memory=33554432 compression.type=gzip batch.size=10240 linger.ms=10 --throughput -1 --payload-file ./test_payloads --payload-delimiter ,
2. Earlier "--record-size" was a required config, now you must provide exactly one of "--record-size" or "--payload-file". Providing both will result in an error.
3. Support for an additional parameter "--payload-delimiter" has been added which defaults to "\n"
Author: Sandesh K <sandesh.karkera@flipkart.com>
Reviewers: dan norwood <norwood@confluent.io>, Jun Rao <junrao@gmail.com>
Closes#2158 from SandeshKarkera/PerfProducerChanges
Issue: https://issues.apache.org/jira/browse/KAFKA-4614
Fixes the problem that the broker threads suffered by long GC pause.
When GC thread collects mmap objects which were created for index files, it unmaps memory mapping so kernel turns to delete a file physically. This work may transparently read file's metadata from physical disk if it's not available on cache.
This seems to happen typically when we're using G1GC, due to it's strategy to left a garbage for a long time if other objects in the same region are still alive.
See the link for the details.
Author: Yuto Kawamura <kawamuray.dadada@gmail.com>
Reviewers: Apurva Mehta <apurva.1618@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>,
Closes#2352 from kawamuray/KAFKA-4614-force-munmap-for-index
Re-branched the trunk and applied the changes to the new branch to simplify commit log.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes#2389 from hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-followup-from-trunk
Address Ismael's comments upon merging
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2184 from cmccabe/KAFKA-4457
This class doesn't need to override this method as it is handled appropriately by the super class
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2397 from dguy/hotfix-npe-state-store
interface for `Processor` in comments incorrectly had `transform` rather than `process`.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Ismael Juma <ismael@juma.me.uk>
Closes#2396 from dguy/minor-javadoc
Remove workaround for testing multiple SASL mechanisms using
sasl.jaas.config and the new support for multiple client
modules within a JVM.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2373 from rajinisivaram/KAFKA-4568
In RocksDBStore, options / wOptions / fOptions are constructed in the constructor, which needs to be dismissed in the close() call; however in some tests, the generated topology is not initialized at all, and hence the corresponding state stores are supposed to not be able to be closed as well since their `init` function is not called. This could cause the above option objects to be not released.
This is fixed in this patch to move the logic out of constructor and inside `init` functions, so that no RocksDB objects will be created in the constructor only. Also some minor cleanups:
1. In KStreamTestDriver.close(), we lost the logic to close the state stores but only call `flush`; it is now changed back to call both.
2. Moved the forwarding logic from KStreamTestDriver to MockProcessorContext to remove the mutual dependency: these functions should really be in ProcessorContext, not the test driver.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2381 from guozhangwang/K3502-pure-virtual-function-unit-tests
1. Added javadoc to public classes
2. Removed `s` from config name for consistency with interface name
3. The policy interface now implements Configurable and AutoCloseable as per the KIP
4. Use `null` instead of `-1` in `RequestMetadata`
5. Perform all broker validation before invoking the policy
6. Add tests
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2388 from ijuma/create-topic-policy-docs-and-config-name-change
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2391 from mjsax/kafka-4060-zk-follow-up-system-tests
This is a follow up of https://github.com/apache/kafka/pull/2166 - refactoring the store hierarchies as requested
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2360 from dguy/state-store-refactor
After debugging this i can see the times that it fails there is a race between when the topic is actually created/ready on the broker and when the assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets called with a `Cluster` with no topics. Hence the test hangs as no tasks get assigned. To fix this I added a `waitForTopics` method to `EmbeddedKafkaCluster`. This will wait until the topics have been created.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2371 from dguy/integration-test-fix
Switched console_consumer, verifiable_consumer and verifiable_producer to use new sasl.jaas_config property instead of static JAAS configuration file when used with SASL_PLAINTEXT.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2323 from rajinisivaram/KAFKA-4580
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2383 from hachikuji/minor-cleanup-kip-88
Runs sanity test and one replication test using SASL/SCRAM.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2355 from rajinisivaram/KAFKA-4590
Remove applicationId parameter as it is no longer used.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2385 from dguy/minor-remove-unused-param
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2384 from hachikuji/KAFKA-4622
Fixes KAFKA-3857
Changes proposed in this pull request:
An additional log cleaner metric has been added:
time-since-last-run-ms: Time since the last log cleaner run, in milliseconds. This metric would be reset to 0 every time log cleaner thread runs. If this metric keeps constantly increasing, it indicates that the log cleaner thread is not alive.
If you are creating alerts around log cleaner, you could monitor this metric. A high "time-since-last-run-ms" value (eg: 600000) indicates that the log cleaner hasn't been running since the last 10 minutes.
The code has been tested. JMX metric has been verified.
Note: This pull request is a continuation of the following pull request. PR#1593 was quite old and I had some trouble rebasing it. Decided to start a fresh PR.
927b28cf41 (diff-ca1c127eee4b3c748ae73028f6abeab8)
Author: Kiran Pillarisetty <pillarisetty@tivo.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#2378 from kiranptivo/log_cleaner_jmx_metric
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#2074 from vahidhashemian/KAFKA-3853
This PR is extracted from https://github.com/apache/kafka/pull/2333 as an incremental fix to ease the reviewing:
1. Removed `storeToProcessorNodeMap` from ProcessorTopology since it was previously used to set the context current record, and can now be replaced with the dirty entry in the named cache.
2. Replaced `sourceStoreToSourceTopic` from ProcessorTopology with `storeToChangelogTopic` map, which includes the corresponding changelog topic name for all stores that are changelog enabled.
3. Modified `ProcessorStateManager` to rely on `sourceStoreToSourceTopic` when retrieving the changelog topic; this makes the second parameter `loggingEnabled` in `register` not needed any more, and we can deprecate the old API with a new one.
4. Also fixed a minor issue in `KStreamBuilder`: if the storeName is not provided in the `table(..)` function, do not create the underlying materialized store. Modified the unit tests to cover this case.
5. Fixed a bunch of other unit tests failures that are exposed by this refactoring, in which we are not setting the applicationId correctly when constructing the mocking processor topology.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Ewen Cheslack-Postava
Closes#2338 from guozhangwang/KMinor-refactor-state-to-changelogtopic
Colin P. Mccabe
Guozhang Wang
Matthias J. Sax
Dana Powers
huxi
Jason Gustafson
Hojjat Jafarpour
Magnus Reftel
Eno Thereska
Vogeti
Xavier Léauté
Shikhar Bhushan
Sandesh K
Vahid Hashemian
Yuto Kawamura
Rajini Sivaram
Damian Guy
Ismael Juma
Ewen Cheslack-Postava
Kiran Pillarisetty
Jiangjie Qin