src-kafka

Commit Graph

Author	SHA1	Message	Date
Matthias J. Sax	ba07d828c5	KAFKA-5362: Add EOS system tests for Streams API Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3201 from mjsax/kafka-5362-add-eos-system-tests-for-streams-api	8 years ago
Matthias J. Sax	21194a63ed	HOTFIX: for flaky Streams EOS integration tests Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #3273 from mjsax/hotfix-flaky-stream-eos-test	8 years ago
Guozhang Wang	b4b65e9eaf	KAFKA-5357 follow-up: Yammer metrics, not Kafka Metrics Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3242 from guozhangwang/K5357-yammer-metrics	8 years ago
Apurva Mehta	eff79849a0	MINOR: Set log level for producer internals to trace for transactions test We need this to debug most issues with the transactions system test. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3261 from apurvam/MINOR-set-log-level-for-producer-to-trace-for-transactions-test	8 years ago
Balint Molnar	1e87b09ec7	KAFKA-5391; Replace zkClient.delete* method with an equivalent zkUtills ijuma can you please review. Author: Balint Molnar <balintmolnar91@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3245 from baluchicken/KAFKA-5391	8 years ago
Eno Thereska	f4e0deca43	KAFKA-5314; exception handling and cleanup for state stores Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3135 from enothereska/exceptions-stores-KAFKA-5314	8 years ago
Matthias J. Sax	3de2d296e9	MINOR: improve JavaDocs for TimestampExtractor interface Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3264 from mjsax/minor-javadocs-timestamp-extractor	8 years ago
Eno Thereska	d1a279dcc7	MINOR: Cache metrics were missing Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3262 from enothereska/minor-missing-metric	8 years ago
Ismael Juma	12612e8297	KAFKA-5329; Fix order of replica list in metadata cache Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Jun Rao <junrao@gmail.com> Closes #3257 from ijuma/kafka-5329-fix-order-of-replica-list-in-metadata-cache	8 years ago
Colin P. Mccabe	2fc91afbad	KAFKA-5292; Fix authorization checks in AdminClient * NetworkClient.java: when trace logging is enabled, show AbstractResponse Struct objects, rather than just a memory address of the AbstractResponse. * AclOperation.java: add documentation of what ACLs imply other ACLs. * Resource.java: add CLUSTER, CLUSTER_NAME constants. * Reconcile the Java and Scala classes for ResourceType, OperationType, etc. Add unit tests to ensure they can be converted to each other. * AclCommand.scala: we should be able to apply ACLs containing Alter and Describe operations to Cluster resources. * SimpleAclAuthorizer: update the authorizer to handle the ACL inheritance rules described in AclOperation.java. * KafkaApis.scala: update createAcls and deleteAcls to use ALTER on CLUSTER, as described in the KIP. describeAcls should use DESCRIBE on CLUSTER. Use fromJava methods instead of fromString methods to convert from Java objects to Scala ones. * SaslSslAdminClientIntegrationTest.scala: do not use AllowEveryoneIfNoAclIsFound. Add a configureSecurityBeforeServerStart hook which installs the ACLs necessary for the tests. Add a test of ACL authorization ALLOW and DENY functionality. Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3240 from cmccabe/KAFKA-5292	8 years ago
Apurva Mehta	79db393ffa	KAFKA-5385; ProducerBatch expiry should go through Sender.failBatch Before this patch, we would call `producerBatch.done` directly from the accumulator when expiring batches. This meant that we would not transition to the `ABORTABLE_ERROR` state in the transaction manager, allowing other transactional requests (including Commits!) to go through, even though the produce failed. This patch modifies the logic so that we call `Sender.failBatch` on every expired batch, thus ensuring that the transaction state is accurate. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #3252 from apurvam/KAFKA-5385-fail-transaction-if-batches-expire	8 years ago
Jason Gustafson	effda15f1f	MINOR: A few logging improvements in group coordinator Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3259 from hachikuji/group-coordinator-logging-improvements	8 years ago
Balint Molnar	6aeca10123	KAFKA-5389; Replace zkClient.exists method with zkUtils.pathExists ijuma can you please review Author: Balint Molnar <balintmolnar91@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3243 from baluchicken/KAFKA-5389	8 years ago
Colin P. Mccabe	38ae746448	KAFKA-5394; KafkaAdminClient#timeoutCallsInFlight does not work as ex… KAFKA-5394; Fix disconnections due to timeouts in AdminClient * Create KafkaClient#disconnect to tear down a connection and deliver disconnects to all the requests on it. * AdminClient.java: fix mismatched braces in JavaDoc. * Make the AdminClientConfig constructor visible for testing. * KafkaAdminClient: add TimeoutProcessorFactory to make the TimeoutProcessor swappable for testing. * Make TimeoutProcessor a static class rather than an inner class. Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3250 from cmccabe/KAFKA-5394	8 years ago
Eno Thereska	5c3d7ca711	MINOR: log4j template should accept log_level The log_level parameter is used in system tests in kafka.py. However the log4j template accepted that parameter in only one place. This led to a large number of DEBUG lines printed even when the intention was to capture only INFO lines. Led to huge log files. Thanks to ijuma for noticing this. Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3247 from enothereska/minor-log4j-template-fix	8 years ago
Jason Gustafson	dcbdce31ba	KAFKA-5378; Return LSO in FetchResponse plus some metrics Author: Jason Gustafson <jason@confluent.io> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3248 from hachikuji/KAFKA-5378	8 years ago
Guozhang Wang	57b0d0fe57	MINOR: syntax brush for java / bash / json / text Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Derrick Or <derrickor@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3214 from guozhangwang/KMinor-doc-java-brush	8 years ago
Damian Guy	f8d6dba345	MINOR: set group initial rebalance delay to 0 in server.properties override the setting of `group.initial.rebalance.delay` in server.properties Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3254 from dguy/minor-prop-change	8 years ago
Rajini Sivaram	bb914a0445	KAFKA-5380: Fix transient failure in KafkaConsumerTest, close consumers 1. Fix ordering of metadata update request for regex subscription to avoid timing issue when heartbeat thread updates metadata 2. Override metadata cluster in MockClient for `KafkaConsumer#testChangingRegexSubscription` to avoid timing issues during update 3. Close consumer in all KafkaConsumer tests since they leave behind heartbeat threads. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3238 from rajinisivaram/KAFKA-5380	8 years ago
Guozhang Wang	b01c83ea91	MINOR: remove ignore annotations in Streams EOS integration test Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Jason Gustafson <jason@confluent.io>, Apurva Mehta <apurva@confluent.io> Closes #3251 from guozhangwang/KMinor-remove-ignored-streams-integration-test	8 years ago
Apurva Mehta	202cb8ea89	KAFKA-5366; Add concurrent reads to transactions system test This currently fails in multiple ways. One of which is most likely KAFKA-5355, where the concurrent consumer reads duplicates. During broker bounces, the concurrent consumer misses messages completely. This is another bug. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3217 from apurvam/KAFKA-5366-add-concurrent-reads-to-transactions-system-test	8 years ago
Nick Pillitteri	d655d806ee	KAFKA-4942: Fix commitTimeoutMs being set before the commit actually started This fixes KAFKA-4942 This supersededs #2730 /cc simplesteph gwenshap ewencp Author: Nick Pillitteri <nickp@smartertravelmedia.com> Author: simplesteph <stephane.maarek@gmail.com> Reviewers: simplesteph <stephane.maarek@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #2912 from 56quarters/fix-connect-offset-commit	8 years ago
Bill Bejeck	2244c68ebc	MINOR: expose commit interval config for SimpleBenchmark Expose the streams commit interval config for different testing scenarios. Author: Bill Bejeck <bill@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3244 from bbejeck/MINOR_add_commit_config_benchmark	8 years ago
Damian Guy	ad8e7cf5e8	MINOR: update stream docs for kip-134 Add a section in the streams docs about the broker config introduced in KIP-134 Author: Damian Guy <damian.guy@gmail.com> Reviewers: Michael G. Noll, Bill Bejeck, Guozhang Wang Closes #3179 from dguy/kip134-doc	8 years ago
Rajini Sivaram	3de683993d	KAFKA-4948; Wait for offset commit in test to fix transient failure `DescribeConsumerGroupTest#testDescribeExistingGroupWithNoMembersWithNewConsumer` shuts down the consumer executor thread and then checks that the assignments returned by `describeGroup` contain the consume group with no members. But if the executor thread is shut down before any offsets are committed, the assignments returned by `describeGroup` doesn't contain the group at all. This PR waits for an offset commit by waiting for the group to appear in `describeGroup` assignments prior to shutting down the executor. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3246 from rajinisivaram/KAFKA-4948	8 years ago
Jason Gustafson	cb78ba1294	KAFKA-5364; ensurePartitionAdded does not handle pending partitions in abortable error state Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3231 from hachikuji/KAFKA-5364	8 years ago
Ismael Juma	313f8d7ddc	MINOR: Create offsets topic explicitly in DescribeConsumerGroupTest This should fix transient failures due to timeouts caused by slow auto creation of the offsets topic. The one exception is `testDescribeGroupWithNewConsumerWithShortInitializationTimeout` where we want initialisation to take longer than the timeout so we let it be auto created. That test has also been a bit flaky and I reduced the timeout to 1 ms. Also: - Simplified resource handling - Removed usage of EasyMock that didn't make sense. - `findCoordinator` should retry if `sendAnyNode` throws an exception Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3237 from ijuma/kafka-49480-describe-consumer-group-test-failure	8 years ago
Rajini Sivaram	17b2bde4be	KAFKA-5051; Avoid reverse DNS lookup to obtain hostname for TLS Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #2835 from rajinisivaram/KAFKA-5051	8 years ago
Sean McCauliff	4b37918ef4	KAFKA-5239; Allocate memory outside the lock in producer buffer pool Move byte buffer allocation out of lock. Add unit test for restoring count when OOM is thrown from byte buffer allocation. Author: Sean McCauliff <smccauliff@linkedin.com> Reviewers: Jiangjie Qin <becket.qin@gmail.com> Closes #3053 from smccauliff/kafka-5239	8 years ago
Rajini Sivaram	9934d28a34	KAFKA-5325; Avoid and handle exceptions for Kerberos re-login If producer creates a connection during Kerberos re-login (after logout, before login), there are no principals in the subject and `SaslClientAuthenticator.configure` may throw an exception while trying to determine the principal. A socket channel is created and its key registered with the selector, but the `RuntimeException` thrown leaves the key registered with the selector without adding the channel to the channel list. This results in an infinite loop of `NullPointerExceptions`. The PR applies two fixes: 1. Convert the `RuntimeException` to a meaningful `KafkaException` 2. Handle any exception in `buildChannel`, cleanup and throw `IOException`. Retries will take care of re-connections. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3208 from rajinisivaram/KAFKA-5325	8 years ago
Jason Gustafson	2bb48a4fa4	KAFKA-5376; Ensure aborted transactions are propagated in DelayedFetch Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3239 from hachikuji/KAFKA-5376	8 years ago
Ismael Juma	b119d04093	KAFKA-5112; Update compatibility system tests to include 0.10.2.1 Also update message format tests now that we have a third message format. Finally, set group.initial.rebalance.delay.ms=100. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Jason Gustafson <jason@confluent.io> Closes #2701 from ijuma/update-upgrade-tests-for-0.11	8 years ago
Ismael Juma	39eb31feae	MINOR: Optimise performance of `Topic.validate()` I included a JMH benchmark and the results follow. The implementation in this PR takes no more than 1/10th of the time when compared to trunk. I also included results for an alternative implementation that is a little slower than the one in the PR. Trunk: ```text TopicBenchmark.testValidate topic avgt 15 134.107 ± 3.956 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 316.241 ± 13.379 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 636.026 ± 30.272 ns/op ``` Implementation in the PR: ```text TopicBenchmark.testValidate topic avgt 15 13.153 ± 0.383 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 26.139 ± 0.896 ns/op TopicBenchmark.testValidate very-long-topic-name.with_more_text avgt 15 44.829 ± 1.390 ns/op ``` Alternative implementation where boolean validChar = Character.isLetterOrDigit(c) \|\| c == '.' \|\| c == '_' \|\| c == '-'; ```text TopicBenchmark.testValidate topic avgt 15 18.883 ± 1.044 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 36.696 ± 1.220 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 65.956 ± 0.669 ns/op ``` Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3234 from ijuma/optimise-topic-is-valid	8 years ago
Damian Guy	8e3ed7028e	KAFKA-5357: capture unexpected exception in txn marker sender thread and treat as fatal error Replace recursion in `TransactionMarkerChannelManager#appendToLogCallback` with retryQueue. Retry the enqueued log appends each time the InterBrokerSendThread runs Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com> Closes #3192 from dguy/kafka-5357	8 years ago
Bill Bejeck	1c70e7ac12	MINOR: Adding deprecated KTable methods to docs from KIP-114 Author: Bill Bejeck <bill@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3215 from bbejeck/kip114-doc	8 years ago
Ismael Juma	b62f156eb4	MINOR: Default GroupInitialRebalanceDelayMsProp to 0 and OffsetsTopicPartitionsProp to 5 in tests This should make tests faster. Tests that require specific values can override these values. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Damian Guy <damian.guy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3236 from ijuma/default-initial-rebalance-delay-and-offsets-topic-partitions	8 years ago
Matthias J. Sax	d4a070fcfa	MINOR: update docs with regard to KIP-123 Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3232 from mjsax/minor-update-docs-for-kip-123	8 years ago
Jason Gustafson	3557f097b8	KAFKA-5355; Test cases to ensure isolation level propagated in delayed fetch Include a few logging improvements. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3230 from hachikuji/KAFKA-5355-TESTS	8 years ago
Ismael Juma	c7bc8f7d8c	MINOR: Remove redundant volatile write in RecordHeaders The JMH benchmark included shows that the redundant volatile write causes the constructor of `ProducerRecord` to take more than 50% longer: ProducerRecordBenchmark.constructorBenchmark avgt 15 24.136 ± 1.458 ns/op (before) ProducerRecordBenchmark.constructorBenchmark avgt 15 14.904 ± 0.231 ns/op (after) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3233 from ijuma/remove-volatile-write-in-records-header-constructor	8 years ago
Ismael Juma	d7ec9feaf1	Revert "KAFKA-5098; ProducerRecord should validate the topic name" We need to verify that this doesn't cause a performance regression, so reverting until that is done. This reverts commit `941e2177c0`.	8 years ago
Ismael Juma	9b58372dcc	KAFKA-5371; Increase request timeout for producer used by testReachableServer 500ms is low for a shared Jenkins environment. Also removed the try/catch blocks that simply obscured the underlying error. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Apurva Mehta <apurva@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3225 from ijuma/kafka-5371-flaky-testReachableServer	8 years ago
huxihx	941e2177c0	KAFKA-5098; ProducerRecord should validate the topic name Author: huxihx <huxi_2b@hotmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3223 from huxihx/KAFKA-5098_Does_not_check_topic_name_before_sending	8 years ago
Mickael Maison	a598c4d26f	KAFKA-4291; TopicCommand --describe should show topics marked for deletion Developed with edoardocomar Author: Mickael Maison <mickael.maison@gmail.com> Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk> Closes #2011 from mimaison/KAFKA-4291	8 years ago
Jason Gustafson	8711e5937f	KAFKA-5019; Upgrades notes for idempotent/transactional features and new message format Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3212 from hachikuji/KAFKA-5019	8 years ago
Colin P. Mccabe	f389b71570	KAFKA-5374; Set allow auto topic creation to false when requesting node information only It avoids the need to handle protocol downgrades and it's safe (i.e. it will never cause the auto creation of topics). Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3220 from ijuma/kafka-5374-admin-client-metadata	8 years ago
Jason Gustafson	6a5a908b1c	KAFKA-5355; DelayedFetch should propagate isolation level to log read Tests will be added in a subsequent commit. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3221 from hachikuji/KAFKA-5355	8 years ago
Ismael Juma	eb5586102e	KAFKA-5272; Policy for Alter Configs (KIP-133) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3210 from ijuma/kafka-5272-improve-validation-for-describe-alter-configs	8 years ago
Xavier Léauté	1c786c589a	MINOR: Reuse decompression buffers in log cleaner Follow-up to KAFKA-5150, reuse decompression buffers in the log cleaner thread. Author: Xavier Léauté <xavier@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3180 from xvrl/logcleaner-decompression-buffers	8 years ago
Apurva Mehta	8104c0de27	KAFKA-5373; Revert breaking change to console consumer This patch reverts `b63e41ea78` since it broke the console consumer -- the consumer prints the addresses of the messages instead of the contents with that patch. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3218 from apurvam/KAFKA-5373-fix-console-consumer	8 years ago
Apurva Mehta	0b107e5f75	HOTFIX: Reinstate the placeholder for logPrefix in TransactionManager Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3216 from apurvam/HOTFIX-logging-bug-in-transaction-manager	8 years ago

1 2 3 4 5 ...

3748 Commits (ba07d828c545d542cc11db60249e660d88a20fbb) All Branches Search

3748 Commits (ba07d828c545d542cc11db60249e660d88a20fbb)

All Branches