Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1375 from becketqin/KAFKA-3698
Delete per-topic metrics when there are no replicas of any partitions of the topic on a broker.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Joel Koshy <jjkoshy.w@gmail.com>, Manikumar reddy O <manikumar.reddy@gmail.com>, Ashish Singh <asingh@cloudera.com, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#944 from rajinisivaram/KAFKA-3258
If you're pushing close to the network capacity, 0.10's additional 8 bytes per message can lead to overload of your network. We (Heroku Kafka) ran into this issue whilst benchmarking 0.10 RC and the ijuma suggested it belonged in the update note.
Comments/suggestions welcome.
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1389 from tcrayford/upgrade_note_about_increased_network_bandwidth
…e throughput drop due to the addition of timestamp field.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1372 from becketqin/KAFKA-3565
This fixes test_producer_throughput with compression_type=snappy.
Also: added heap dump on out of memory error to `producer_performance.py` and corrected the upgrade note related to the change in buffer size for compression streams.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1385 from ijuma/kafka-3713-test_producer_throughput-snappy-fail and squashes the following commits:
54c7962 [Ismael Juma] Correct upgrade note about buffer size for compression stream
515040b [Ismael Juma] Call `compressor.close()` to fix memory leak
5311e5b [Ismael Juma] Dump heap on out of memory error when running `producer_performance.py`
This PR loosens access restrictions on `WrapperSerde` to `protected` so that users can define a `Serdes` subclass that provides additional custom serde members following the same pattern as the parent class.
This is my own work and is compatible with Kafka's license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Guozhang Wang
Closes#1382 from jklukas/expose-wrapperserde
This PR is to fix synchronization issue between deleteOldSegments() and delete() method calls. log.deleteOldSegments() call throws NullPointerException after log.delete() method call.
cc ijuma junrao
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1367 from omkreddy/KAFKA-3584
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Geoff Anderson <geoff@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1364 from Ishiihara/add-quote-classpath
Initially proposed by ijuma in https://github.com/apache/kafka/pull/1362#issuecomment-218293662
mjsax commented:
> StreamThread.close() should be extended to call metrics.close() (the class need a private member to reference the Metrics object, too)
The `Metrics` instance is created in the `KafkaStreams` constructor and shared between all threads, so closing it within the threads doesn't seem like the right approach. This PR calls `Metrics.close()` in `KafkaStreams.close()` instead.
cc guozhangwang
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma, Guozhang Wang
Closes#1379 from jklukas/close-streams-metrics
This is a follow up of KAKFA-3421 to update the connect developer guide to include the configuration validation. Also includes a couple of minor fixes.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1366 from Ishiihara/connect-dev-doc
The information from this file is displayed in:
https://projects.apache.org/project.html?kafka
I removed the tag instead of fixing it because it's one more thing to maintain and neither Hadoop nor ZooKeeper have it (httpd does though).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1378 from ijuma/remove-release-from-doap
I removed the hamcrest matcher to unbreak the build, but we probably want to tweak the `import-control.xml` as it currently only allows it for `<subpackage name="integration">`, which is weird.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1380 from ijuma/fix-streams-config-test-checkstyle
This is an improved version of https://github.com/apache/kafka/pull/1374, where we include a unit test.
/cc ijuma and guozhangwang
Author: Guozhang Wang <wangguoz@gmail.com>
Author: Michael G. Noll <michael@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1377 from miguno/streamsconfig-multiple-bootstrap-servers
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1373 from ijuma/fix-producer-buffer-size-upgrade-note
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1358 from hachikuji/KAFKA-3583
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Geoff Anderson <geoff@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1365 from hachikuji/KAFKA-3694
Propagate IOException in SslTransportLayer channel.close to be consistent with PlaintextTransportLayer, close authenticator on channel close even if transport layer close fails.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1370 from rajinisivaram/minor-channelclose2
ZkUtils.getSequenceId() method is used to generate broker id sequence numbers. During startup, each broker updates the data at /brokers/seqid zk path and returns stat.getVersion as next sequence id.
stat.getVersion returns "1" for first data update. So ZkUtils.getSequenceId() should return "1" on first data update.
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Flavio Junqueira <fpj@apache.org>, Ismael Juma <ismael@juma.me.uk>
Closes#1224 from omkreddy/KAFKA-3525
We have had transient failures in this method when Jenkins is
overloaded.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1359 from ijuma/increase-producer-close-timeout-in-test
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1321 from guozhangwang/KStreamsJavaDoc
Add a test for changing SASL mechanism using rolling upgrade and a test for rolling upgrade from 0.9.0.x to 0.10.0 with SASL/GSSAPI.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ben Stopford <benstopford@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1290 from rajinisivaram/KAFKA-3634
_copyDependantTestLibs_ was added temporarily as a dependency of _jar_ task to enable SASL system tests to be run with MiniKdc without changing the automated system test runs which run _gradlew clean jar_. Since the build target _systemTestLibs_ is already in Kafka build.gradle, the Confluent automated test runs can now run _gradlew clean systemTestLibs_ instead. This PR provides the final change to remove _copyDependantTestLibs_ from the _jar_ task. This should be committed only after the Confluent automated sytem test build script is updated, to avoid breaking any builds.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#430 from rajinisivaram/minor-systemtestlibs
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1354 from hachikuji/minor-add-missing-assignor-tostrings
After testing KAFKA-3160 a bit more, I found that the error code was not being set properly in ProduceResponse. This happened because the validation error is raised in the CompressionFactory constructor, which was not wrapped in a try / catch.
ijuma junrao
(This contribution is my original work and I license the work under Apache 2.0.)
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Gwen Shapira <cshapi@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1344 from dpkp/decompress_error_code
Removed the over pessimistic require and instead attempt to fill the dedup buffer. Use the (only) map until full;
this may allow to process all dirty segment (optimism) or may happen in the middle of a dirt segment.
In either case, do compaction using the map loaded that way.
This patch was developed with edoardocomar
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1332 from mimaison/KAFKA-3587
Test sends large request using multiple writes of length followed by request body. The first write should succeed, but since the server closes the connection on processing the length that is too big, subsequent writes may fail. Modified test to handle this exception.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1349 from rajinisivaram/KAFKA-3662
Add references to the new consumer property 'max.partition.fetch.bytes' along with the old consumer property 'fetch.message.max.bytes' in the corresponding warning messages of TopicCommand.
Also, create and leverage a static variable for the default value of the new consumer property.
Also, use 'DEFAULT_...' for default propoerty constant names in the code instead of '..._DEFAULT'.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Manikumar reddy O <manikumar.reddy@gmail.com>, Ashish Singh <asingh@cloudera.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1239 from vahidhashemian/KAFKA-3579
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1338 from ijuma/kafka-3670-controlled-shutdown-leader-selector-preferred-replica
This contribution is my original work and I license the work under Apache 2.0.
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1212 from dpkp/KAFKA-3160
so that unresolvable DNS names are ignored and only throw an error if no other bootstrap servers are resolvable.
Author: Jonathan Bond <jbond@netflix.com>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#792 from bondj/KAFKA-3112
…value for doLog
Author: Mark Grover <mark@apache.org>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1334 from markgrover/kafka-3669-trunk
Jiangjie Qin
Rajini Sivaram
Tom Crayford
Ismael Juma
Jeff Klukas
Manikumar reddy O
Liquan Pei
Guozhang Wang
Jason Gustafson
Ben Stopford
Vahid Hashemian
Ashish Singh
Ewen Cheslack-Postava
Dana Powers
Mickael Maison
Kaufman Ng
Jonathan Bond
Chen Zhu
Mark Grover