guozhangwang
removing an unused class, FilteredIterator, and its test.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Gwen Shapira
Closes#816 from ymatsuda/remove_obsolete_class
Committing before inserting all records into the database
might lead to some records being lost.
I've changed the example to commit only after all records
returned by `poll` are inserted into the database.
Author: Dmitry Stratiychuk <dstratiychuk@yammer-inc.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#210 from shtratos/KAFKA-2478
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#773 from ijuma/kafka-3100-create-broker-version-check
Author: Konrad <konkalita@gmail.com>
Author: konradkalita <konkalita@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#749 from konradkalita/kafka-3076
Log compaction is supported on compressed topics as of 0.9.0, so update the docs to reflect that.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#807 from wushujames/KAFKA-3138
Improve the documentation by fixing typos, punctuations, and correcting the content.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#778 from vahidhashemian/typo05/fix_documentation_typos
Fix PatternSyntaxException and hand caused by it in MirrorMaker on passing invalid java regex string as whitelist
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#805 from SinghAsDev/KAFKA-3140
Given a schema with 2 versions (0 and 1), if you pass in a version of `2` you will get an `OutOfBoundsException` instead of an `IllegalArgumentException`.
This fixes the problem by changing the check from `>` to `>=`, which will now return true in the given scenario.
Author: Micah Zoltu <micah@zoltu.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#748 from Zoltu/patch-1
ewencp
This small change allows users to use Vagrantfile.local to specify a custom prefix for names of ec2 instances brought up with vagrant.
This makes management of multiple aws test clusters a little easier since individual clusters can be assigned different name prefixes.
if `ec2_instance_name_prefix` is not specified in `Vagrantfile.local`, behavior will be exactly the same as before this change.
Testing:
- aws: I verified worker nodes, broker nodes, zk nodes with and without the prefix override. Behavior is as expected
- locally: I verified that bringing up worker nodes, broker nodes, zk nodes on a local machine is not impacted by this change.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#801 from granders/minor-vagrant-aws-overrideable-prefix
TopicCommand provide a tool to add partitions for existing topics. It try to find the startIndex from existing partitions. There's a minor flaw in this process, it try to use the first partition fetched from zookeeper as the start partition, and use the first replica id in this partition as the startIndex.
One thing, the first partition fetched from zookeeper is not necessary to be the start partition. As partition id begin from zero, we should use partition with id zero as the start partition.
The other, broker id does not necessary begin from 0, so the startIndex is not necessary to be the first replica id in the start partition.
Author: chenshangan <chenshangan@meituan.com>
Reviewers: Guozhang Wang
Closes#329 from shangan/trunk-KAFKA-2146
The MessageFormatter being used was only introduced as of 0.9.0.0. The Kafka
version in some tests is changed dynamically, sometimes from trunk back to an
earlier version, so this option must be set based on the version used when the
service is started, not when it is created.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Geoff Anderson, Ismael Juma, Grant Henke
Closes#770 from ewencp/kafka-3080-system-test-console-consumer-version-failure
This PR replaces all occurrences of kafka.api.ProducerRequest/ProducerResponse by their common equivalents.
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#110 from dajac/KAFKA-2071
Also fix missing call to `sensors.record` on this error.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#791 from ijuma/fix-producer-memory-leak-on-authorization-exception
I just noticed that `MemoryRecords.sizeInBytes` throws NPE when MemoryRecords is non-writable. `compressor` is explicitly set to null when `writable` is false (L56) at the construction time, for instance when `MemoryRecords.readableRecords` is used.
guozhangwang Could you take a look when you have time?
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Guozhang Wang
Closes#786 from dajac/kafka-npe
Provides a configuration to opt out of broker id generation.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#762 from granthenke/id-generation
Add some basic documentation about the format, a link to get more detailed information and an example usage. I didn't want to make a huge section on the format since it documented elsewhere but I can expand is folks want.
https://issues.apache.org/jira/browse/KAFKA-3095
Author: Tom Graves <tgraves@yahoo-inc.com>
Reviewers: Gwen Shapira
Closes#776 from tgravescs/KAFKA-3095
It behaves better on Windows and provides more useful error messages.
Also:
* Minor inconsistency fix in `kafka.server.OffsetCheckpoint`.
* Remove delete from `streams.state.OffsetCheckpoint` constructor (similar to the change in `kafka.server.OffsetCheckpoint` in 836cb19633 (diff-2503b32f29cbbd61ed8316f127829455L29)).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#771 from ijuma/kafka-3105-use-atomic-move-with-fallback-instead-of-rename
Follow up PR as per comments in the ticket.
junrao It should be correct now as `curBrokers` included only live brokers and live/dead brokers are computed based on it. Could you take a look when you have time?
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#756 from dajac/KAFKA-3085
Buffer is rewound before the compression rate metric is updated which results in 0 compress rate. The fix is to include a new compressRate field to record the latest compression rate in ```close()``` and return it to sensor
Author: Xiao, Tao <taxiao@ebay.com>
Reviewers: Guozhang Wang
Closes#784 from xiaotao183/kafka-2993
guozhangwang
When ```WindowedSerializer``` is specified in ```to(...)``` or ```through(...)``` for a key, we use ```WindowedStreamPartitioner```.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#779 from ymatsuda/partitioner
topic level config delete config options
use --delete-config instead of --deleteConfig
Author: eamon <zhang.eamon@hotmail.com>
Reviewers: Ismael Juma, Grant Henke, Guozhang Wang
Closes#612 from EamonZhang/trunk
Remove deletion of tmp file in `OffsetCheckpoint`'s constructor. This delete causes unintuitive behaviour like `LogRecoveryTest` causing a `System.exit` because the test creates an instance of `OffsetCheckpoint` in order to call `read()` on it (while unexpectedly deleting a file being written by another instance of `OffsetCheckpoint`).
Also:
* Improve error-handling in `OffsetCheckpoint`
* Also include minor performance improvements in `read()`
* Minor clean-ups to `ReplicaManager` and `LogRecoveryTest`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#759 from ijuma/kafka-3063-log-recovery-test-exits-jvm
I'm also fixing a bug in the testChroot test case.
Author: Flavio Junqueira <fpj@apache.org>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#736 from fpj/KAFKA-3069
Note that KAFKA-3077 will be required to run these tests.
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#747 from SinghAsDev/KAFKA-3078
Yasuhiro Matsuda
Grant Henke
Vahid Hashemian
Dmitry Stratiychuk
Guozhang Wang
Ismael Juma
Konrad
James Cheng
Yifan Ying
Piotr Szwed
Ashish Singh
Micah Zoltu
Geoff Anderson
Chen Shangan
Ewen Cheslack-Postava
David Jacot
Atul Soman
Tom Graves
Jason Gustafson
Tao Xiao
Eamon Zhang
Ján Koščo
Flavio Junqueira