src-kafka

Commit Graph

Author	SHA1	Message	Date
Sean McCauliff	4b37918ef4	KAFKA-5239; Allocate memory outside the lock in producer buffer pool Move byte buffer allocation out of lock. Add unit test for restoring count when OOM is thrown from byte buffer allocation. Author: Sean McCauliff <smccauliff@linkedin.com> Reviewers: Jiangjie Qin <becket.qin@gmail.com> Closes #3053 from smccauliff/kafka-5239	8 years ago
Rajini Sivaram	9934d28a34	KAFKA-5325; Avoid and handle exceptions for Kerberos re-login If producer creates a connection during Kerberos re-login (after logout, before login), there are no principals in the subject and `SaslClientAuthenticator.configure` may throw an exception while trying to determine the principal. A socket channel is created and its key registered with the selector, but the `RuntimeException` thrown leaves the key registered with the selector without adding the channel to the channel list. This results in an infinite loop of `NullPointerExceptions`. The PR applies two fixes: 1. Convert the `RuntimeException` to a meaningful `KafkaException` 2. Handle any exception in `buildChannel`, cleanup and throw `IOException`. Retries will take care of re-connections. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3208 from rajinisivaram/KAFKA-5325	8 years ago
Jason Gustafson	2bb48a4fa4	KAFKA-5376; Ensure aborted transactions are propagated in DelayedFetch Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3239 from hachikuji/KAFKA-5376	8 years ago
Ismael Juma	b119d04093	KAFKA-5112; Update compatibility system tests to include 0.10.2.1 Also update message format tests now that we have a third message format. Finally, set group.initial.rebalance.delay.ms=100. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Jason Gustafson <jason@confluent.io> Closes #2701 from ijuma/update-upgrade-tests-for-0.11	8 years ago
Ismael Juma	39eb31feae	MINOR: Optimise performance of `Topic.validate()` I included a JMH benchmark and the results follow. The implementation in this PR takes no more than 1/10th of the time when compared to trunk. I also included results for an alternative implementation that is a little slower than the one in the PR. Trunk: ```text TopicBenchmark.testValidate topic avgt 15 134.107 ± 3.956 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 316.241 ± 13.379 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 636.026 ± 30.272 ns/op ``` Implementation in the PR: ```text TopicBenchmark.testValidate topic avgt 15 13.153 ± 0.383 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 26.139 ± 0.896 ns/op TopicBenchmark.testValidate very-long-topic-name.with_more_text avgt 15 44.829 ± 1.390 ns/op ``` Alternative implementation where boolean validChar = Character.isLetterOrDigit(c) \|\| c == '.' \|\| c == '_' \|\| c == '-'; ```text TopicBenchmark.testValidate topic avgt 15 18.883 ± 1.044 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 36.696 ± 1.220 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 65.956 ± 0.669 ns/op ``` Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3234 from ijuma/optimise-topic-is-valid	8 years ago
Damian Guy	8e3ed7028e	KAFKA-5357: capture unexpected exception in txn marker sender thread and treat as fatal error Replace recursion in `TransactionMarkerChannelManager#appendToLogCallback` with retryQueue. Retry the enqueued log appends each time the InterBrokerSendThread runs Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com> Closes #3192 from dguy/kafka-5357	8 years ago
Bill Bejeck	1c70e7ac12	MINOR: Adding deprecated KTable methods to docs from KIP-114 Author: Bill Bejeck <bill@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3215 from bbejeck/kip114-doc	8 years ago
Ismael Juma	b62f156eb4	MINOR: Default GroupInitialRebalanceDelayMsProp to 0 and OffsetsTopicPartitionsProp to 5 in tests This should make tests faster. Tests that require specific values can override these values. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Damian Guy <damian.guy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3236 from ijuma/default-initial-rebalance-delay-and-offsets-topic-partitions	8 years ago
Matthias J. Sax	d4a070fcfa	MINOR: update docs with regard to KIP-123 Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3232 from mjsax/minor-update-docs-for-kip-123	8 years ago
Jason Gustafson	3557f097b8	KAFKA-5355; Test cases to ensure isolation level propagated in delayed fetch Include a few logging improvements. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3230 from hachikuji/KAFKA-5355-TESTS	8 years ago
Ismael Juma	c7bc8f7d8c	MINOR: Remove redundant volatile write in RecordHeaders The JMH benchmark included shows that the redundant volatile write causes the constructor of `ProducerRecord` to take more than 50% longer: ProducerRecordBenchmark.constructorBenchmark avgt 15 24.136 ± 1.458 ns/op (before) ProducerRecordBenchmark.constructorBenchmark avgt 15 14.904 ± 0.231 ns/op (after) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3233 from ijuma/remove-volatile-write-in-records-header-constructor	8 years ago
Ismael Juma	d7ec9feaf1	Revert "KAFKA-5098; ProducerRecord should validate the topic name" We need to verify that this doesn't cause a performance regression, so reverting until that is done. This reverts commit `941e2177c0`.	8 years ago
Ismael Juma	9b58372dcc	KAFKA-5371; Increase request timeout for producer used by testReachableServer 500ms is low for a shared Jenkins environment. Also removed the try/catch blocks that simply obscured the underlying error. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Apurva Mehta <apurva@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3225 from ijuma/kafka-5371-flaky-testReachableServer	8 years ago
huxihx	941e2177c0	KAFKA-5098; ProducerRecord should validate the topic name Author: huxihx <huxi_2b@hotmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3223 from huxihx/KAFKA-5098_Does_not_check_topic_name_before_sending	8 years ago
Mickael Maison	a598c4d26f	KAFKA-4291; TopicCommand --describe should show topics marked for deletion Developed with edoardocomar Author: Mickael Maison <mickael.maison@gmail.com> Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk> Closes #2011 from mimaison/KAFKA-4291	8 years ago
Jason Gustafson	8711e5937f	KAFKA-5019; Upgrades notes for idempotent/transactional features and new message format Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3212 from hachikuji/KAFKA-5019	8 years ago
Colin P. Mccabe	f389b71570	KAFKA-5374; Set allow auto topic creation to false when requesting node information only It avoids the need to handle protocol downgrades and it's safe (i.e. it will never cause the auto creation of topics). Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3220 from ijuma/kafka-5374-admin-client-metadata	8 years ago
Jason Gustafson	6a5a908b1c	KAFKA-5355; DelayedFetch should propagate isolation level to log read Tests will be added in a subsequent commit. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3221 from hachikuji/KAFKA-5355	8 years ago
Ismael Juma	eb5586102e	KAFKA-5272; Policy for Alter Configs (KIP-133) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3210 from ijuma/kafka-5272-improve-validation-for-describe-alter-configs	8 years ago
Xavier Léauté	1c786c589a	MINOR: Reuse decompression buffers in log cleaner Follow-up to KAFKA-5150, reuse decompression buffers in the log cleaner thread. Author: Xavier Léauté <xavier@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3180 from xvrl/logcleaner-decompression-buffers	8 years ago
Apurva Mehta	8104c0de27	KAFKA-5373; Revert breaking change to console consumer This patch reverts `b63e41ea78` since it broke the console consumer -- the consumer prints the addresses of the messages instead of the contents with that patch. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3218 from apurvam/KAFKA-5373-fix-console-consumer	8 years ago
Apurva Mehta	0b107e5f75	HOTFIX: Reinstate the placeholder for logPrefix in TransactionManager Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3216 from apurvam/HOTFIX-logging-bug-in-transaction-manager	8 years ago
Apurva Mehta	a318b15129	KAFKA-5322; Add `OPERATION_NOT_ATTEMPTED` error code to resolve AddPartitionsToTxn inconsistency In the `AddPartitionsToTxn` request handling, if even one partition fails authorization checks, the entire request is essentially failed. However, the `AddPartitionsToTxnResponse` today will only contain the error codes for the topics which failed authorization. It will have no error code for the topics which succeeded, making it inconsistent with other APIs. This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is returned for the successful partitions to indicate that they were not added to the transaction. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io> Closes #3204 from apurvam/KAFKA-5322-add-operation-not-attempted-for-add-partitions	8 years ago
Ismael Juma	4959444afc	KAFKA-5236; Increase the block/buffer size when compressing with Snappy or Gzip We had originally increased Snappy’s block size as part of KAFKA-3704. However, we had some issues with excessive memory usage in the producer and we reverted it in 7c6ee8d5e. After more investigation, we fixed the underlying reason why memory usage seemed to grow much more than expected via KAFKA-3747 (included in 0.10.0.1). In 0.10.2, we changed the broker to use the same classes as the producer and the broker’s block size for Snappy was changed from 32 KB to 1KB. As reported in KAFKA-5236, the on disk size is, in some cases, 50% larger when the data is compressed with 1 KB instead of 32 KB as the block size. As discussed in KAFKA-3704, it may be worth making this configurable and/or allocate the compression buffers from the producer pool. However, for 0.11.0.0, I think the simplest thing to do is to default to 32 KB for Snappy (the default if no block size is provided). I also increased the Gzip buffer size. 1 KB is too small and the default is smaller still (512 bytes). 8 KB (which is the default buffer size for BufferedOutputStream) seemed like a reasonable default. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3205 from ijuma/kafka-5236-snappy-block-size	8 years ago
Vahid Hashemian	b63e41ea78	KAFKA-3982: Fix processing order of some of the consumer properties This PR updates processing of console consumer's input properties. For both old and new consumer, the value provided for `auto.offset.reset` indirectly through `consumer.config` or `consumer.property` arguments will now take effect. For new consumer and for `key.deserializer` and `value.deserializer` properties, the precedence order is fixed to first the value directly provided as an argument, then the value provided indirectly via `consumer.property` and then `consumer.config`, and finally a default value. Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #1655 from vahidhashemian/KAFKA-3982	8 years ago
Hamidreza Afzali	0a8b10e27f	KAFKA-5368: Fix skipped record metrics to use rate of sum instead of rate of count This resolved the issue with Kafka Streams skipped records sensor reporting wrong values. Jira ticket: https://issues.apache.org/jira/browse/KAFKA-5368 The contribution is my original work and I license the work to the project under the project's open source license. Author: Hamidreza Afzali <hrafzali@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3206 from hrafzali/KAFKA-5368_skipped-records-sensor-bug	8 years ago
amethystic	d7d1196a0b	KAFKA-5327: ConsoleConsumer should manually commit offsets for records that are returned in receive() KAFKA-5327: ConsoleConsumer should manually commit offsets for those records it really consumed. Currently it leaves this job to the automatic offset commit scheme where some unread messages will be passed if `--max-messages` is set. Author: amethystic <huxi_2b@hotmail.com> Author: huxi <huxi_2b@hotmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3148 from amethystic/KAFKA-5327_ConsoleConsumer_distable_autocommit	8 years ago
Randall Hauch	af85e05b98	KAFKA-5164: Ensure SetSchemaMetadata updates key or value when Schema changes When the `SetSchemaMetadata` SMT is used to change the name and/or version of the key or value’s schema, any references to the old schema in the key or value must be changed to reference the new schema. Only keys or values that are `Struct` have such references, and so currently only these are adjusted. This is based on `trunk` since the fix is expected to be targeted to the 0.11.1 release. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3198 from rhauch/kafka-5164	8 years ago
Vahid Hashemian	f85c18032b	KAFKA-3264; Deprecate the old Scala consumer (KIP-109) Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Ismael Juma <ismael@juma.me.uk> This patch had conflicts when merged, resolved by Committer: Ismael Juma <ismael@juma.me.uk> Closes #2328 from vahidhashemian/KAFKA-3264	8 years ago
Vahid Hashemian	5d46348619	KAFKA-5282; Use a factory method to create producers/consumers and close them in tearDown Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> This patch had conflicts when merged, resolved by Committer: Ismael Juma <ismael@juma.me.uk> Closes #3129 from vahidhashemian/KAFKA-5282	8 years ago
Jason Gustafson	47a3066ccb	KAFKA-5365; Fix regression in compressed message iteration affecting magic v0 and v1 Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3203 from hachikuji/KAFKA-5365	8 years ago
Dale Peakall	1188db5657	KAFKA-5311: Support ExtendedDeserializer in Kafka Streams Author: Dale Peakall <dale@peakall.net> Reviewers: Michael André Pearce <michael.andre.pearce@me.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3199 from subnova/streams-extendeddeserializer	8 years ago
Apurva Mehta	673ab671e6	KAFKA-5364; Don't fail producer if drained partition is not yet in transaction Due to the async nature of the producer, it is possible to attempt to drain a messages whose partition hasn't been added to the transaction yet. Before this patch, we considered this a fatal error. However, it is only in error if the partition isn't in the queue to be sent to the coordinator. This patch updates the logic so that we only fail the producer if the partition would never be added to the transaction. If the partition of the batch is yet to be added, we will simply wait for the partition to be added to the transaction before sending the batch to the broker. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #3202 from apurvam/KAFKA-5364-ensure-partitions-added-to-txn-before-send	8 years ago
Jason Gustafson	1c882ee5fb	KAFKA-5283; Handle producer epoch/sequence overflow - Producer sequence numbers should wrap around - Generate a new producerId if the producer epoch would overflow Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3183 from hachikuji/KAFKA-5283	8 years ago
Jason Gustafson	0c3e466eb0	MINOR: Logging/debugging improvements for transactions Author: Jason Gustafson <jason@confluent.io> Author: Apurva Mehta <apurva.1618@gmail.com> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3185 from hachikuji/minor-transaction-logging-improvements	8 years ago
Guozhang Wang	4c8ec795a0	MINOR: Add upgrade notes for KAFKA-2358 Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3200 from guozhangwang/KMinor-KAFKA-2358-upgrade-notes	8 years ago
Matthias J. Sax	1e350b860a	MINOR: update Streams docs for KIP-123 Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Ismael Juma, Ewen Chesklack-Postava, Bill Bejeck, Guozhang Wang Closes #3194 from mjsax/minor-update-docs-for-kip-123	8 years ago
Rajini Sivaram	6b9bc3fd26	KAFKA-5345; Close KafkaClient when streams client is closed Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3195 from rajinisivaram/KAFKA-5345	8 years ago
Bharat Viswanadham	be392f95de	KAFKA-5229: Reflections logs excessive warnings when scanning classpaths changed the reflections log level to ERROR. And tested it, now the warning logs are not shown up during the start of Kafka connect. ewencp could you please review the changes. Author: Bharat Viswanadham <bharatv@us.ibm.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3072 from bharatviswa504/KAFKA-5229	8 years ago
Damian Guy	adb4ea74ab	MINOR: add upgrade not for group.initial.rebalance.delay.ms Add a new entry in upgrade.html for `group.initial.rebalance.delay.ms` Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com> Closes #3189 from dguy/relabance-delay	8 years ago
Guozhang Wang	e4d829c47b	set the rejoinNeeded in listener's onSuccess	8 years ago
Damian Guy	1b16acaaa1	KAFKA-5154: do not set rejoinNeeded in joinGroup response but in syncGroup response handler Scenario is as follows: 1. Consumer subscribes to topic t1 and begins consuming 2. heartbeat fails as the group is rebalancing 3. ConsumerCoordinator.onJoinGroupPrepare is called 3.1 onPartitionsRevoked is called 4. consumer becomes the group leader 5. sends sync group request 6. sync group is cancelled due to disconnection 7. fetch request is sent for partitions that have previously been revoked Author: Damian Guy <damian.guy@gmail.com> Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3181 from dguy/kafka-5154	8 years ago
Colin P. Mccabe	b3036c5861	KAFKA-5293; Do not apply exponential backoff if users have overridden… … reconnect.backoff.ms Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3174 from cmccabe/KAFKA-5293	8 years ago
Matthias J. Sax	39c1e6259c	KAFKA-5361: Add EOS integration tests for Streams API Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api	8 years ago
Apurva Mehta	1959835d9e	KAFKA-5281; System tests for transactions Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3149 from apurvam/KAFKA-5281-transactions-system-tests	8 years ago
Ismael Juma	8e8b3c5657	KAFKA-5360; Down-converted uncompressed batches should respect fetch offset More specifically, V2 messages are always batched (whether compressed or not) while V0/V1 are only batched if they are compressed. Clients like librdkafka expect to receive messages from the fetch offset when dealing with uncompressed V0/V1 messages. When converting from V2 to V0/1, we were returning all the messages in the V2 batch. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3191 from ijuma/kafka-5360-down-converted-uncompressed-respect-offset	8 years ago
Ewen Cheslack-Postava	a086346427	KAFKA-4459: Run rat checks in Jenkins script Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3186 from ewencp/kafka-4459-rat-jenkins	8 years ago
Rajini Sivaram	640082776b	KAFKA-4956: Verify client-side throttle time metrics in quota test Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jun Rao <junrao@gmail.com> Closes #3190 from rajinisivaram/KAFKA-4956-unittest	8 years ago
Konstantine Karantasis	e0150a25e8	MINOR: Traverse plugin path recursively in Connect (KIP-146) Author: Konstantine Karantasis <konstantine@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3173 from kkonstantine/MINOR-Traverse-plugin-path-recursively-in-Connect	8 years ago
Apurva Mehta	049abe7efa	KAFKA-5351: Reset pending state when returning an error in appendTransactionToLog Without this patch, future client retries would get the `CONCURRENT_TRANSACTIONS` error code indefinitely, since the pending state wouldn't be cleared when the append to the log failed. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3184 from apurvam/KAFKA-5351-clear-pending-state-on-retriable-error	8 years ago

... 4 5 6 7 8 ...

3970 Commits (1844bf2b2f4cdf5a8209d7ceccb6701fc7dcf768) All Branches Search

3970 Commits (1844bf2b2f4cdf5a8209d7ceccb6701fc7dcf768)

All Branches