After testing KAFKA-3160 a bit more, I found that the error code was not being set properly in ProduceResponse. This happened because the validation error is raised in the CompressionFactory constructor, which was not wrapped in a try / catch.
ijuma junrao
(This contribution is my original work and I license the work under Apache 2.0.)
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Gwen Shapira <cshapi@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1344 from dpkp/decompress_error_code
Removed the over pessimistic require and instead attempt to fill the dedup buffer. Use the (only) map until full;
this may allow to process all dirty segment (optimism) or may happen in the middle of a dirt segment.
In either case, do compaction using the map loaded that way.
This patch was developed with edoardocomar
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1332 from mimaison/KAFKA-3587
Test sends large request using multiple writes of length followed by request body. The first write should succeed, but since the server closes the connection on processing the length that is too big, subsequent writes may fail. Modified test to handle this exception.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1349 from rajinisivaram/KAFKA-3662
Add references to the new consumer property 'max.partition.fetch.bytes' along with the old consumer property 'fetch.message.max.bytes' in the corresponding warning messages of TopicCommand.
Also, create and leverage a static variable for the default value of the new consumer property.
Also, use 'DEFAULT_...' for default propoerty constant names in the code instead of '..._DEFAULT'.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Ashish Singh <asingh@cloudera.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1239 from vahidhashemian/KAFKA-3579
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1338 from ijuma/kafka-3670-controlled-shutdown-leader-selector-preferred-replica
This contribution is my original work and I license the work under Apache 2.0.
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1212 from dpkp/KAFKA-3160
…value for doLog
Author: Mark Grover <mark@apache.org>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1334 from markgrover/kafka-3669-trunk
Handle unsupported version of ApiVersionsRequest during SASL auth as well as normal operation by returning an error response.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1310 from rajinisivaram/KAFKA-3652
Author: William Thurston <wthurston@linkedin.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma, Guozhang Wang
Closes#1318 from ijuma/KAFKA-2236-offset-request-reply-segment-rolling-race
Tiny change to add a force option to the topic and config commands so they can be called programatically without requiring user input.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#351 from benstopford/CPKAFKA-61B
When an address is written textually, the zone index is appended to the address, separated by a percent sign (%). The actual syntax of zone indices depends on the operating system.
Author: Som Sahu <sosahu@microsoft.com>
Author: Soumyajit Sahu <soumyajit-sahu@users.noreply.github.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1305 from soumyajit-sahu/fixIPV6RegexPattern_trunk
Also:
* Remove redundant `time.milliseconds` call in `Sensor.record`
* Clean-up a number of tests and remove a manual test that is no longer required
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Liquan Pei <liquanpei@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1265 from ijuma/kafka-3128-zookeeper-metrics
Server-side implementation and tests for handling ApiVersionsRequest before SaslHandshakeRequest.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Gwen Shapira, Ismael Juma
Closes#1286 from rajinisivaram/KAFKA-3618
Even if a test calls stop() on console_consumer or verifiable_producer, it is still possible that producer/consumer will not shutdown cleanly, and will be killed forcefully after a timeout. It will be useful for some tests to know whether a clean shutdown happened or not. This PR adds methods to console_consumer and verifiable_producer to query whether clean shutdown happened or not.
hachikuji and/or granders Please review.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Jason Gustafson, Geoff Anderson, Gwen Shapira
Closes#1278 from apovzner/kafka-3597
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Onur Karaman <okaraman@linkedin.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1284 from hachikuji/KAFKA-3636
Unit tests for SASL authenticator, tests for SASL/PLAIN and multiple mechanisms, authorization test for SASL/PLAIN
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1273 from rajinisivaram/KAFKA-3617
The patch does the following.
1. Adds ApiVersionsRequest/Response.
2. Adds UNSUPPORTED_VERSION error and UnsupportedVersionException.
3. Adds broker side handling of ApiVersionsRequest.
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Gwen Shapira, Ismael Juma, Magnus Edenhill
Closes#986 from SinghAsDev/KAFKA-3307
…elds
- Adds boolean type to the protocol
- Allows protocol arrays to be null (optionally)
- Adds support to ask for no topics in the metadata request
- Adds new fields to the Metadata response protocol
- Adds server code to handle new fields
- Support no-topic metadata requests
- Track controller id in the metadata cache
- Check if a topic is considered internal
- Included rack information if present
- Include all replicas and ISRs, even if node is down
- Adds test code to test new functionality independent of the client
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira, Ismael Juma, Ashish Singh
Closes#1095 from granthenke/metadata-changes
Code changes corresponding to KIP-43 to enable review of the KIP.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@apache.org>, Ismael Juma <ismael@juma.me.uk>
Closes#812 from rajinisivaram/KAFKA-3149
with help from enothereska :)
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Jun Rao <junrao@apache.org>, Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#956 from benstopford/KAFKA-3270-ReassignPartitionsCommand-Tests
For enums and other constant strings, use locale independent case conversions to enable comparisons to work regardless of the default locale.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Manikumar Reddy, Ismael Juma, Guozhang Wang, Gwen Shapira
Closes#1220 from rajinisivaram/KAFKA-3548
RollingBounceTest is a system test that cannot be run reliably in unit tests and ReplicationTest is a superset of the
functionality: in addition to verifying that bouncing leaders eventually results in a new leader, ReplicationTest also
validates that data continues to be produced and consumed.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira
Closes#1242 from ewencp/minor-remove-rolling-bounce-integration-test
This patch fixes all occurances of two consecutive 'the's in the code comments.
Author: Ishita Mandhan (imandhaus.ibm.com)
Author: Ishita Mandhan <imandha@us.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1240 from imandhan/typofixes
* Use a fixed `Random` seed in `EndToEndLatency.scala` for determinism
* Add `compression_type` to and remove `consumer_fetch_max_wait` from `end_to_end_latency.py`. The latter was never used.
* Tweak logging of `end_to_end_latency.py` to be similar to `consumer_performance.py`.
* Add `compression_type` to `benchmark_test.py` methods and add `snappy` to `matrix` annotation
* Use randomly generated bytes from a restricted range for `ProducerPerformance` payload. This is a simple fix for now. It can be improved in the PR for KAFKA-3554.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1225 from ijuma/kafka-3558-add-compression_type-benchmark_test.py
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1217 from granthenke/close-consumers
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#1206 from hachikuji/KAFKA-3470
This should make Log.read act the same when startOffset is larger than maxOffset as it would if startOffset was larger than logEndOffset. The current behavior can result in an IllegalArgumentException from LogSegment if a consumer attempts to fetch an offset above the high watermark which is present in the leader's log. It seems more correct if Log.read presents the view of the log to consumers as if it simply ended at maxOffset (high watermark).
I've tried to describe an example scenario of this happening here https://issues.apache.org/jira/browse/KAFKA-725?focusedCommentId=15221673&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15221673
I'm not sure I understand why ReplicaManager sets maxOffset to the high watermark, and not high watermark + 1. Isn't the high watermark the last committed message, and readable by consumers?
Tests passed for me locally on second try, seems like it just hit a flaky test.
Author: Stig Rohde Døssing <sdo@it-minds.dk>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1178 from srdo/KAFKA-725
Fail unsent requests only when returning from KafkaConsumer.poll().
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1183 from rajinisivaram/KAFKA-3488
Allows the the maximum retires when writing to zookeeper to be overridden in tests and sets the value to Int.MaxValue to avoid transient failure.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1156 from granthenke/transient-acl-test
I also fixed a few issues in `SocketServerTest` and included a few clean-ups.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1172 from ijuma/kafka-3489-update-request-metrics-if-client-closes
* Make all fields accessed outside of a lock `volatile`
* Only allow mutation within the class
* Remove unnecessary `AtomicInteger` since mutation always happens inside a lock
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1188 from ijuma/kafka-3510-offset-index-thread-safety
Also removed the code for handling negative timeouts in `blockingReady` as `Selector.poll` has not supported that for a while.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1177 from ijuma/kafka-3495-blocking-send-and-receive-request-timeout
Use a different keytab for server and client in SASL tests
Also:
* Improve approach used to build the JAAS files programmatically
* Delete stale `kafka_jaas.conf` file
* Move `FourLetterWords` to its own file, add `Zk` prefix and clean-up its usage
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Harsha Chintalapani, Gwen Shapira
Closes#533 from ijuma/separate-keytabs-for-sasl-tests
Zookeeper clients that are not closed after the server is shutdown keep trying to reconnect, reloading JAAS configuration. This impacts subsequent tests which rely on JAAS config to be reset.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Flavio Junqueira <fpj@apache.org>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1171 from rajinisivaram/KAFKA-2910
This also fixes KAFKA-3453 and KAFKA-2866.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1155 from ijuma/kafka-3475-introduce-our-minikdc
Also:
* Rename remaining `CoreUtils.rm` to `delete` for consistency
* Use `try with resources` in `Utils` to simplify code
* Silence compiler warning due to exception catch clause in `TestUtils`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1153 from ijuma/remove-redundant-core-utils-rm
Dana Powers
Mickael Maison
Rajini Sivaram
Vahid Hashemian
Ismael Juma
Mark Grover
Kaufman Ng
Guozhang Wang
William Thurston
Jason Gustafson
Ben Stopford
Som Sahu
Anna Povzner
Liquan Pei
Ashish Singh
Manikumar reddy O
Grant Henke
Stig
Dionysis Grigoropoulos
Ewen Cheslack-Postava
Ishita Mandhan
Stig Rohde Døssing