Removed the first two in favour of the latter.
Author: Kamal C <kamal.chandraprakash@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3350 from Kamal15/utilcleanup
Also remove broker-id tags as we generally use them to provide
additional context.
Finally, do a few clean-ups (could not resist).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3341 from ijuma/tweak-transaction-metrics
Headers are only preserved if the new consumer is
used since the old consumer does not support
them.
Add test case to verify the fix and to avoid regression.
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3322 from michaelandrepearce/KAFKA-5354
Before this patch, the `partitionErrors` was an immutable map. As a result if a single producer had a marker for multiple partitions, and if there were multiple response callbacks for a single append, we would get an `UnsupportedOperationException` in the `writeTxnMarker` handler.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3313 from apurvam/KAFKA-5438-fix-unsupportedoperationexception-in-writetxnmarker
The default client-id bandwidth quota config properties have been marked deprecated in the doc, but a warning may be useful before the property is removed in a future release.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3315 from rajinisivaram/MINOR-deprecate-staticquota
Skip topics that don't have any partitions in zkUtils.getAllPartitions()
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3295 from mimaison/KAFKA-5418
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3298 from hachikuji/KAFKA-5428
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3284 from hachikuji/minor-either-usage-cleanup
In `TransationStateManager`, we reset the pending state if an error occurred while appending to log; this is correct except that for the `TransactionMarkerChannelManager`, as it will retry appending to log and if eventually it succeeded, the transaction metadata's completing transition will throw an IllegalStateException since pending state is None, this will be thrown all the way to the `KafkaApis` and be swallowed.
1. Do not reset the pending state if the append will be retried (as is the case when write the complete transition).
2. A bunch of log4j improvements based the debugging experience. The main principle is to make sure all error codes that is about to sent to the client will be logged, and unnecessary log4j entries to be removed.
3. Also moved some log entries in ReplicationUtils.scala to `trace`: this is rather orthogonal to this PR but I found it rather annoying while debugging the logs.
4. A couple of unrelated bug fixes as pointed by hachikuji and apurvam.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3287 from guozhangwang/KHotfix-transaction-coordinator-append-callback
Record `apiThrottleTime` in RequestChannel.
junrao A trivial change. Please review. Thanks.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3265 from huxihx/KAFKA-5405
This assertion is hard to get right because the system time can roll backward on a host due to NTP (as shown in the ticket), and also because a transaction can start on one host and complete on another. Getting precise clock times across hosts is virtually impossible, and this check makes things fragile.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3286 from apurvam/KAFKA-5415-avoid-timestamp-check-in-completeTransition
This reverts commit d7d1196a0b.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3277 from hachikuji/KAFKA-5414
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3242 from guozhangwang/K5357-yammer-metrics
ijuma can you please review.
Author: Balint Molnar <balintmolnar91@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3245 from baluchicken/KAFKA-5391
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Colin P. Mccabe <cmccabe@confluent.io>, Jun Rao <junrao@gmail.com>
Closes#3257 from ijuma/kafka-5329-fix-order-of-replica-list-in-metadata-cache
* NetworkClient.java: when trace logging is enabled, show AbstractResponse Struct objects, rather than just a memory address of the AbstractResponse.
* AclOperation.java: add documentation of what ACLs imply other ACLs.
* Resource.java: add CLUSTER, CLUSTER_NAME constants.
* Reconcile the Java and Scala classes for ResourceType, OperationType, etc. Add unit tests to ensure they can be converted to each other.
* AclCommand.scala: we should be able to apply ACLs containing Alter and Describe operations to Cluster resources.
* SimpleAclAuthorizer: update the authorizer to handle the ACL inheritance rules described in AclOperation.java.
* KafkaApis.scala: update createAcls and deleteAcls to use ALTER on CLUSTER, as described in the KIP. describeAcls should use DESCRIBE on CLUSTER. Use fromJava methods instead of fromString methods to convert from Java objects to Scala ones.
* SaslSslAdminClientIntegrationTest.scala: do not use AllowEveryoneIfNoAclIsFound. Add a configureSecurityBeforeServerStart hook which installs the ACLs necessary for the tests. Add a test of ACL authorization ALLOW and DENY functionality.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3240 from cmccabe/KAFKA-5292
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3259 from hachikuji/group-coordinator-logging-improvements
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3248 from hachikuji/KAFKA-5378
This should fix transient failures due to timeouts caused by slow
auto creation of the offsets topic. The one exception is
`testDescribeGroupWithNewConsumerWithShortInitializationTimeout` where
we want initialisation to take longer than the timeout so we let
it be auto created. That test has also been a bit flaky and I reduced
the timeout to 1 ms.
Also:
- Simplified resource handling
- Removed usage of EasyMock that didn't make sense.
- `findCoordinator` should retry if `sendAnyNode` throws an exception
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3237 from ijuma/kafka-49480-describe-consumer-group-test-failure
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3239 from hachikuji/KAFKA-5376
Replace recursion in `TransactionMarkerChannelManager#appendToLogCallback` with retryQueue. Retry the enqueued log appends each time the InterBrokerSendThread runs
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#3192 from dguy/kafka-5357
Include a few logging improvements.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3230 from hachikuji/KAFKA-5355-TESTS
Developed with edoardocomar
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2011 from mimaison/KAFKA-4291
Tests will be added in a subsequent commit.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3221 from hachikuji/KAFKA-5355
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3210 from ijuma/kafka-5272-improve-validation-for-describe-alter-configs
Follow-up to KAFKA-5150, reuse decompression buffers in the log cleaner thread.
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3180 from xvrl/logcleaner-decompression-buffers
This patch reverts b63e41ea78
since it broke the console consumer -- the consumer prints
the addresses of the messages instead of the contents with
that patch.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3218 from apurvam/KAFKA-5373-fix-console-consumer
In the `AddPartitionsToTxn` request handling, if even one partition fails authorization checks, the entire request is essentially failed. However, the `AddPartitionsToTxnResponse` today will only contain the error codes for the topics which failed authorization. It will have no error code for the topics which succeeded, making it inconsistent with other APIs.
This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is returned for the successful partitions to indicate that they were not added to the transaction.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3204 from apurvam/KAFKA-5322-add-operation-not-attempted-for-add-partitions
This PR updates processing of console consumer's input properties.
For both old and new consumer, the value provided for `auto.offset.reset` indirectly through `consumer.config` or `consumer.property` arguments will now take effect.
For new consumer and for `key.deserializer` and `value.deserializer` properties, the precedence order is fixed to first the value directly provided as an argument, then the value provided indirectly via `consumer.property` and then `consumer.config`, and finally a default value.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1655 from vahidhashemian/KAFKA-3982
KAFKA-5327: ConsoleConsumer should manually commit offsets for those records it really consumed. Currently it leaves this job to the automatic offset commit scheme where some unread messages will be passed if `--max-messages` is set.
Author: amethystic <huxi_2b@hotmail.com>
Author: huxi <huxi_2b@hotmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3148 from amethystic/KAFKA-5327_ConsoleConsumer_distable_autocommit
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
This patch had conflicts when merged, resolved by
Committer: Ismael Juma <ismael@juma.me.uk>
Closes#2328 from vahidhashemian/KAFKA-3264
- Producer sequence numbers should wrap around
- Generate a new producerId if the producer epoch would overflow
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3183 from hachikuji/KAFKA-5283
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3193 from mjsax/kafka-5361-add-eos-integration-tests-for-streams-api
More specifically, V2 messages are always batched (whether compressed or not) while
V0/V1 are only batched if they are compressed.
Clients like librdkafka expect to receive messages from the fetch offset when dealing with uncompressed V0/V1 messages. When converting from V2 to V0/1, we were returning all the
messages in the V2 batch.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3191 from ijuma/kafka-5360-down-converted-uncompressed-respect-offset
Without this patch, future client retries would get the `CONCURRENT_TRANSACTIONS` error code indefinitely, since the pending state wouldn't be cleared when the append to the log failed.
Author: Apurva Mehta <apurva@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3184 from apurvam/KAFKA-5351-clear-pending-state-on-retriable-error
- Added a boolean `allow_auto_topic_creation` to MetadataRequest and
bumped the protocol version to V4.
- When connecting to brokers older than 0.11.0.0, the `allow_auto_topic_creation`
field won't be considered, so we send a metadata request for all topics
to keep the behavior consistent.
- Set `allow_auto_topic_creation` to false in the new AdminClient and
StreamsKafkaClient (which exists for the purpose of creating topics
manually); set it to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.
- Add `allowAutoTopicCreation` field to `Metadata`, which is used by
`DefaultMetadataUpdater`. This is not strictly needed for the new
`AdminClient`, but it avoids surprises if it ever adds a topic to `Metadata`
via `setTopics` or `addTopic`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3098 from ijuma/kafka-5291-admin-client-no-auto-topic-creation
Also introduce TopicConfig.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3120 from cmccabe/KAFKA-5265
Return UNSUPPORTED_MESSAGE_FORMAT in handleWriteTxnMarkers when a topic is not the correct message format.
Remove any TopicPartitions that have same error from those waiting for markers
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3152 from dguy/kafka-5308
Also update the test to be simpler since we can use a mock event to simulate the issue
more easily (thanks Jun for the suggestion). This should fix two issues:
1. A transient test failure due to a NPE in ControllerFailoverTest.testMetadataUpdate:
```text
Caused by: java.lang.NullPointerException
at kafka.controller.ControllerBrokerRequestBatch.addUpdateMetadataRequestForBrokers(ControllerChannelManager.scala:338)
at kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:975)
at kafka.controller.ControllerFailoverTest.testMetadataUpdate(ControllerFailoverTest.scala:141)
```
The test was creating an additional thread and it does not seem like it was doing the
appropriate synchronization (perhaps this became more of an issue after we changed
the Controller to be single-threaded and changed the locking)
2. Setting `activeControllerId.set(-1)` in `triggerControllerMove` causes `Reelect` not to invoke `onControllerResignation`. Among other things, this causes an `IllegalStateException` to be thrown when `KafkaScheduler.startup` is invoked for the second time without the corresponding `shutdown`. We now simply call `onControllerResignation` as part of `triggerControllerMove`.
Finally, I included a few clean-ups:
1. No longer update the broker state in `onControllerFailover`. This is no longer needed
since we removed the `RunningAsController` state (KAFKA-3761).
2. Trivial clean-ups in KafkaController
3. Removed unused parameter in `ZkUtils.getPartitionLeaderAndIsrForTopics`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#2935 from ijuma/on-controller-resignation-if-trigger-controller-move
Here is the sketch of this proposal:
1. When it is time to send the txn markers, only look for the leader node of the partition once instead of retrying, and if that information is not available, it means the partition is highly likely been removed since it was in the cache before. In this case, we just remove the partition from the metadata object and skip putting into the corresponding queue, and if all partitions' leader broker are non-available, complete this delayed operation to proceed to write the complete txn log entry.
2. If the leader id is unknown from the cache but the corresponding node object with the listener name is not available, it means that the leader is likely unavailable right now. Put it into a separate queue and let sender thread retry fetching its metadata again each time upon draining the queue.
One caveat of this approach is the delete-and-recreate case, and the argument is that since all the messages are deleted anyways when deleting the topic-partition, it does not matter whether the markers are on the log partitions or not.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Apurva Mehta <apurva@confluent.io>, Damian Guy <damian.guy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3130 from guozhangwang/K5202-handle-topic-deletion
KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error
Change all the OptionParser constructor in Kafka into "new OptionParser(false)"
Author: xinlihua <xin.lihua1@zte.com.cn>
Author: unknown <00067310@A23338408.zte.intra>
Author: auroraxlh <xin.lihua1@zte.com.cn>
Author: xin <xin.lihua1@zte.com.cn>
Reviewers: Damian Guy, Guozhang Wang
Closes#2349 from auroraxlh/fix_OptionParser_bug
Tom Bentley
Kamal C
Ismael Juma
Michael Andre Pearce
Apurva Mehta
Jason Gustafson
Rajini Sivaram
Mickael Maison
Guozhang Wang
huxihx
Balint Molnar
Colin P. Mccabe
Damian Guy
Xavier Léauté
Vahid Hashemian
Matthias J. Sax
Jorge Quilcate Otoya
xinlihua
Jiangjie Qin