src-kafka

Commit Graph

Author	SHA1	Message	Date
Ismael Juma	b62f156eb4	MINOR: Default GroupInitialRebalanceDelayMsProp to 0 and OffsetsTopicPartitionsProp to 5 in tests This should make tests faster. Tests that require specific values can override these values. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Damian Guy <damian.guy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3236 from ijuma/default-initial-rebalance-delay-and-offsets-topic-partitions	8 years ago
Matthias J. Sax	d4a070fcfa	MINOR: update docs with regard to KIP-123 Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3232 from mjsax/minor-update-docs-for-kip-123	8 years ago
Jason Gustafson	3557f097b8	KAFKA-5355; Test cases to ensure isolation level propagated in delayed fetch Include a few logging improvements. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3230 from hachikuji/KAFKA-5355-TESTS	8 years ago
Ismael Juma	c7bc8f7d8c	MINOR: Remove redundant volatile write in RecordHeaders The JMH benchmark included shows that the redundant volatile write causes the constructor of `ProducerRecord` to take more than 50% longer: ProducerRecordBenchmark.constructorBenchmark avgt 15 24.136 ± 1.458 ns/op (before) ProducerRecordBenchmark.constructorBenchmark avgt 15 14.904 ± 0.231 ns/op (after) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3233 from ijuma/remove-volatile-write-in-records-header-constructor	8 years ago
Ismael Juma	d7ec9feaf1	Revert "KAFKA-5098; ProducerRecord should validate the topic name" We need to verify that this doesn't cause a performance regression, so reverting until that is done. This reverts commit `941e2177c0`.	8 years ago
Ismael Juma	9b58372dcc	KAFKA-5371; Increase request timeout for producer used by testReachableServer 500ms is low for a shared Jenkins environment. Also removed the try/catch blocks that simply obscured the underlying error. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Apurva Mehta <apurva@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3225 from ijuma/kafka-5371-flaky-testReachableServer	8 years ago
huxihx	941e2177c0	KAFKA-5098; ProducerRecord should validate the topic name Author: huxihx <huxi_2b@hotmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3223 from huxihx/KAFKA-5098_Does_not_check_topic_name_before_sending	8 years ago
Mickael Maison	a598c4d26f	KAFKA-4291; TopicCommand --describe should show topics marked for deletion Developed with edoardocomar Author: Mickael Maison <mickael.maison@gmail.com> Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk> Closes #2011 from mimaison/KAFKA-4291	8 years ago
Jason Gustafson	8711e5937f	KAFKA-5019; Upgrades notes for idempotent/transactional features and new message format Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3212 from hachikuji/KAFKA-5019	8 years ago
Colin P. Mccabe	f389b71570	KAFKA-5374; Set allow auto topic creation to false when requesting node information only It avoids the need to handle protocol downgrades and it's safe (i.e. it will never cause the auto creation of topics). Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3220 from ijuma/kafka-5374-admin-client-metadata	8 years ago
Jason Gustafson	6a5a908b1c	KAFKA-5355; DelayedFetch should propagate isolation level to log read Tests will be added in a subsequent commit. Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3221 from hachikuji/KAFKA-5355	8 years ago
Ismael Juma	eb5586102e	KAFKA-5272; Policy for Alter Configs (KIP-133) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3210 from ijuma/kafka-5272-improve-validation-for-describe-alter-configs	8 years ago
Xavier Léauté	1c786c589a	MINOR: Reuse decompression buffers in log cleaner Follow-up to KAFKA-5150, reuse decompression buffers in the log cleaner thread. Author: Xavier Léauté <xavier@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3180 from xvrl/logcleaner-decompression-buffers	8 years ago
Apurva Mehta	8104c0de27	KAFKA-5373; Revert breaking change to console consumer This patch reverts `b63e41ea78` since it broke the console consumer -- the consumer prints the addresses of the messages instead of the contents with that patch. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3218 from apurvam/KAFKA-5373-fix-console-consumer	8 years ago
Apurva Mehta	0b107e5f75	HOTFIX: Reinstate the placeholder for logPrefix in TransactionManager Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3216 from apurvam/HOTFIX-logging-bug-in-transaction-manager	8 years ago
Apurva Mehta	a318b15129	KAFKA-5322; Add `OPERATION_NOT_ATTEMPTED` error code to resolve AddPartitionsToTxn inconsistency In the `AddPartitionsToTxn` request handling, if even one partition fails authorization checks, the entire request is essentially failed. However, the `AddPartitionsToTxnResponse` today will only contain the error codes for the topics which failed authorization. It will have no error code for the topics which succeeded, making it inconsistent with other APIs. This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is returned for the successful partitions to indicate that they were not added to the transaction. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io> Closes #3204 from apurvam/KAFKA-5322-add-operation-not-attempted-for-add-partitions	8 years ago
Ismael Juma	4959444afc	KAFKA-5236; Increase the block/buffer size when compressing with Snappy or Gzip We had originally increased Snappy’s block size as part of KAFKA-3704. However, we had some issues with excessive memory usage in the producer and we reverted it in `7c6ee8d5e`. After more investigation, we fixed the underlying reason why memory usage seemed to grow much more than expected via KAFKA-3747 (included in 0.10.0.1). In 0.10.2, we changed the broker to use the same classes as the producer and the broker’s block size for Snappy was changed from 32 KB to 1KB. As reported in KAFKA-5236, the on disk size is, in some cases, 50% larger when the data is compressed with 1 KB instead of 32 KB as the block size. As discussed in KAFKA-3704, it may be worth making this configurable and/or allocate the compression buffers from the producer pool. However, for 0.11.0.0, I think the simplest thing to do is to default to 32 KB for Snappy (the default if no block size is provided). I also increased the Gzip buffer size. 1 KB is too small and the default is smaller still (512 bytes). 8 KB (which is the default buffer size for BufferedOutputStream) seemed like a reasonable default. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3205 from ijuma/kafka-5236-snappy-block-size	8 years ago
Vahid Hashemian	b63e41ea78	KAFKA-3982: Fix processing order of some of the consumer properties This PR updates processing of console consumer's input properties. For both old and new consumer, the value provided for `auto.offset.reset` indirectly through `consumer.config` or `consumer.property` arguments will now take effect. For new consumer and for `key.deserializer` and `value.deserializer` properties, the precedence order is fixed to first the value directly provided as an argument, then the value provided indirectly via `consumer.property` and then `consumer.config`, and finally a default value. Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #1655 from vahidhashemian/KAFKA-3982	8 years ago
Hamidreza Afzali	0a8b10e27f	KAFKA-5368: Fix skipped record metrics to use rate of sum instead of rate of count This resolved the issue with Kafka Streams skipped records sensor reporting wrong values. Jira ticket: https://issues.apache.org/jira/browse/KAFKA-5368 The contribution is my original work and I license the work to the project under the project's open source license. Author: Hamidreza Afzali <hrafzali@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3206 from hrafzali/KAFKA-5368_skipped-records-sensor-bug	8 years ago
amethystic	d7d1196a0b	KAFKA-5327: ConsoleConsumer should manually commit offsets for records that are returned in receive() KAFKA-5327: ConsoleConsumer should manually commit offsets for those records it really consumed. Currently it leaves this job to the automatic offset commit scheme where some unread messages will be passed if `--max-messages` is set. Author: amethystic <huxi_2b@hotmail.com> Author: huxi <huxi_2b@hotmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3148 from amethystic/KAFKA-5327_ConsoleConsumer_distable_autocommit	8 years ago
Randall Hauch	af85e05b98	KAFKA-5164: Ensure SetSchemaMetadata updates key or value when Schema changes When the `SetSchemaMetadata` SMT is used to change the name and/or version of the key or value’s schema, any references to the old schema in the key or value must be changed to reference the new schema. Only keys or values that are `Struct` have such references, and so currently only these are adjusted. This is based on `trunk` since the fix is expected to be targeted to the 0.11.1 release. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3198 from rhauch/kafka-5164	8 years ago
Vahid Hashemian	f85c18032b	KAFKA-3264; Deprecate the old Scala consumer (KIP-109) Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Ismael Juma <ismael@juma.me.uk> This patch had conflicts when merged, resolved by Committer: Ismael Juma <ismael@juma.me.uk> Closes #2328 from vahidhashemian/KAFKA-3264	8 years ago
Vahid Hashemian	5d46348619	KAFKA-5282; Use a factory method to create producers/consumers and close them in tearDown Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> This patch had conflicts when merged, resolved by Committer: Ismael Juma <ismael@juma.me.uk> Closes #3129 from vahidhashemian/KAFKA-5282	8 years ago
Jason Gustafson	47a3066ccb	KAFKA-5365; Fix regression in compressed message iteration affecting magic v0 and v1 Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3203 from hachikuji/KAFKA-5365	8 years ago
Dale Peakall	1188db5657	KAFKA-5311: Support ExtendedDeserializer in Kafka Streams Author: Dale Peakall <dale@peakall.net> Reviewers: Michael André Pearce <michael.andre.pearce@me.com>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3199 from subnova/streams-extendeddeserializer	8 years ago
Apurva Mehta	673ab671e6	KAFKA-5364; Don't fail producer if drained partition is not yet in transaction Due to the async nature of the producer, it is possible to attempt to drain a messages whose partition hasn't been added to the transaction yet. Before this patch, we considered this a fatal error. However, it is only in error if the partition isn't in the queue to be sent to the coordinator. This patch updates the logic so that we only fail the producer if the partition would never be added to the transaction. If the partition of the batch is yet to be added, we will simply wait for the partition to be added to the transaction before sending the batch to the broker. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io> Closes #3202 from apurvam/KAFKA-5364-ensure-partitions-added-to-txn-before-send	8 years ago
Jason Gustafson	1c882ee5fb	KAFKA-5283; Handle producer epoch/sequence overflow - Producer sequence numbers should wrap around - Generate a new producerId if the producer epoch would overflow Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3183 from hachikuji/KAFKA-5283	8 years ago
Jason Gustafson	0c3e466eb0	MINOR: Logging/debugging improvements for transactions Author: Jason Gustafson <jason@confluent.io> Author: Apurva Mehta <apurva.1618@gmail.com> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3185 from hachikuji/minor-transaction-logging-improvements	8 years ago
Guozhang Wang	4c8ec795a0	MINOR: Add upgrade notes for KAFKA-2358 Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3200 from guozhangwang/KMinor-KAFKA-2358-upgrade-notes	8 years ago
Matthias J. Sax	1e350b860a	MINOR: update Streams docs for KIP-123 Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Ismael Juma, Ewen Chesklack-Postava, Bill Bejeck, Guozhang Wang Closes #3194 from mjsax/minor-update-docs-for-kip-123	8 years ago
Rajini Sivaram	6b9bc3fd26	KAFKA-5345; Close KafkaClient when streams client is closed Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #3195 from rajinisivaram/KAFKA-5345	8 years ago
Bharat Viswanadham	be392f95de	KAFKA-5229: Reflections logs excessive warnings when scanning classpaths changed the reflections log level to ERROR. And tested it, now the warning logs are not shown up during the start of Kafka connect. ewencp could you please review the changes. Author: Bharat Viswanadham <bharatv@us.ibm.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3072 from bharatviswa504/KAFKA-5229	8 years ago
Damian Guy	adb4ea74ab	MINOR: add upgrade not for group.initial.rebalance.delay.ms Add a new entry in upgrade.html for `group.initial.rebalance.delay.ms` Author: Damian Guy <damian.guy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com> Closes #3189 from dguy/relabance-delay	8 years ago
Guozhang Wang	e4d829c47b	set the rejoinNeeded in listener's onSuccess	8 years ago
Damian Guy	1b16acaaa1	KAFKA-5154: do not set rejoinNeeded in joinGroup response but in syncGroup response handler Scenario is as follows: 1. Consumer subscribes to topic t1 and begins consuming 2. heartbeat fails as the group is rebalancing 3. ConsumerCoordinator.onJoinGroupPrepare is called 3.1 onPartitionsRevoked is called 4. consumer becomes the group leader 5. sends sync group request 6. sync group is cancelled due to disconnection 7. fetch request is sent for partitions that have previously been revoked Author: Damian Guy <damian.guy@gmail.com> Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3181 from dguy/kafka-5154	8 years ago
Colin P. Mccabe	b3036c5861	KAFKA-5293; Do not apply exponential backoff if users have overridden… … reconnect.backoff.ms Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3174 from cmccabe/KAFKA-5293	8 years ago
Matthias J. Sax	39c1e6259c	KAFKA-5361: Add EOS integration tests for Streams API Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api	8 years ago
Apurva Mehta	1959835d9e	KAFKA-5281; System tests for transactions Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3149 from apurvam/KAFKA-5281-transactions-system-tests	8 years ago
Ismael Juma	8e8b3c5657	KAFKA-5360; Down-converted uncompressed batches should respect fetch offset More specifically, V2 messages are always batched (whether compressed or not) while V0/V1 are only batched if they are compressed. Clients like librdkafka expect to receive messages from the fetch offset when dealing with uncompressed V0/V1 messages. When converting from V2 to V0/1, we were returning all the messages in the V2 batch. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3191 from ijuma/kafka-5360-down-converted-uncompressed-respect-offset	8 years ago
Ewen Cheslack-Postava	a086346427	KAFKA-4459: Run rat checks in Jenkins script Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #3186 from ewencp/kafka-4459-rat-jenkins	8 years ago
Rajini Sivaram	640082776b	KAFKA-4956: Verify client-side throttle time metrics in quota test Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Jun Rao <junrao@gmail.com> Closes #3190 from rajinisivaram/KAFKA-4956-unittest	8 years ago
Konstantine Karantasis	e0150a25e8	MINOR: Traverse plugin path recursively in Connect (KIP-146) Author: Konstantine Karantasis <konstantine@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3173 from kkonstantine/MINOR-Traverse-plugin-path-recursively-in-Connect	8 years ago
Apurva Mehta	049abe7efa	KAFKA-5351: Reset pending state when returning an error in appendTransactionToLog Without this patch, future client retries would get the `CONCURRENT_TRANSACTIONS` error code indefinitely, since the pending state wouldn't be cleared when the append to the log failed. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3184 from apurvam/KAFKA-5351-clear-pending-state-on-retriable-error	8 years ago
Jason Gustafson	e4a6b50dec	KAFKA-5340; Batch splitting should preserve magic and transactional flag Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Jiangjie Qin <becket.qin@gmail.com>, Guozhang Wang <wangguoz@gmail.com> Closes #3162 from hachikuji/KAFKA-5340	8 years ago
Bill Bejeck	6360e04e70	KAFKA-5226; Fixes issue where adding topics matching a regex subscribed stream may not be detected by all followers until onJoinComplete returns. Author: Bill Bejeck <bill@confluent.io> Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3157 from bbejeck/KAFKA-5226_null_pointer_source_node_deserialize	8 years ago
Guozhang Wang	681c6fc1fb	KAFKA-5350: Modify unstable annotations in Streams API Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io> Closes #3172 from guozhangwang/K5350-compatibility-annotations	8 years ago
Ismael Juma	7311dcbc53	KAFKA-5291; AdminClient should not trigger auto creation of topics - Added a boolean `allow_auto_topic_creation` to MetadataRequest and bumped the protocol version to V4. - When connecting to brokers older than 0.11.0.0, the `allow_auto_topic_creation` field won't be considered, so we send a metadata request for all topics to keep the behavior consistent. - Set `allow_auto_topic_creation` to false in the new AdminClient and StreamsKafkaClient (which exists for the purpose of creating topics manually); set it to true everywhere else for now. Other clients will eventually rely on client-side auto topic creation, but that’s not there yet. - Add `allowAutoTopicCreation` field to `Metadata`, which is used by `DefaultMetadataUpdater`. This is not strictly needed for the new `AdminClient`, but it avoids surprises if it ever adds a topic to `Metadata` via `setTopics` or `addTopic`. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jun Rao <junrao@gmail.com> Closes #3098 from ijuma/kafka-5291-admin-client-no-auto-topic-creation	8 years ago
Ismael Juma	647afeff6a	KAFKA-5353; baseTimestamp should always have a create timestamp This makes the case where we build the records from scratch consistent with the case where update the batch header "in place". Thanks to edenhill who found the issue while testing librdkafka. The reason our tests don’t catch this is that we rely on the maxTimestamp to compute the record level timestamps if log append time is used. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3177 from ijuma/set-base-sequence-for-log-append-time	8 years ago
Ismael Juma	eeb8f67810	MINOR: Use `waitUntil` to fix transient failures of ControllerFailoverTest Without it, it's possible that the assertion is checked before the exception is thrown in the callback. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Closes #3182 from ijuma/fix-controller-failover-flakiness	8 years ago
Mario Molina	dc5bf4bd45	KAFKA-5218; New Short serializer, deserializer, serde Author: Mario Molina <mmolimar@gmail.com> Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>, Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3017 from mmolimar/KAFKA-5218	8 years ago

1 2 3 4 5 ...

3763 Commits (43e935a630eb0a7fa64c5a1a38bfee17f9b724dc) All Branches Search

3763 Commits (43e935a630eb0a7fa64c5a1a38bfee17f9b724dc)

All Branches