Also remove some unused imports.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#992 from guozhangwang/KSExamples
* Change `MessageFormat.writeTo` to take a `ConsumerRecord`
* Change `MessageReader.readMessage()` to use `ProducerRecord`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#972 from ijuma/kafka-3273-message-formatter-and-reader-resilient
The ability to specify a deserializer for keys and values was added in a recent commit (845c6eae1f), but it contained a few issues.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#987 from ijuma/console-consumer-cleanups
and remove TOTAL_RECORDS_TO_PROCESS
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#985 from ymatsuda/config_params
Observation: when doing "gradlew releaseTarGz" the streams jar was not included in the tarball. Adding a line to include it. ijuma guozhangwang could you please review. Thanks.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#984 from enothereska/trunk
guozhangwang
Author: Kim Christensen <kich@mvno.dk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#912 from kichristensen/KAFKA-3133
guozhangwang made the changes as requested, I reverted my original commit and that seems to have closed the other pull request - sorry if that mucks up the process a bit
Author: tomdearman <tom.dearman@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#978 from tomdearman/KAFKA-3278
Added CompressionTest that tests 4 producers, each using a different compression type and one not using compression.
Enabled VerifiableProducer to run producers with different compression types (passed in the constructor). This includes enabling each producer to output unique values, so that the verification process in ProduceConsumeValidateTest is correct (counts acks from all producers).
Also a fix for console consumer to raise an exception if it sees the incorrect consumer output (before we swallowed an exception, so was hard to debug the issue).
Author: Anna Povzner <anna@confluent.io>
Reviewers: Geoff Anderson, Jason Gustafson
Closes#958 from apovzner/kafka-3214
…list is not supported for new consumers
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#923 from SinghAsDev/KAFKA-3243
This is my original work and I license the work to the project under the project's open source license.
Author: Richard Whaling <rwhaling@spantree.net>
Reviewers: Jason Gustafson, Gwen Shapira
Closes#968 from rwhaling/docs/kafkaconsumer-heartbeat-doc-improvement
Author: Tom Lee <github@tomlee.co>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Jiangjie Qin <jiangjie@linkedin.com>, Grant Henke <ghenke@cloudera.com>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#962 from hachikuji/KAFKA-2698
guozhangwang
* add table aggregate to the system test
* actually create change log partition replica
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#966 from ymatsuda/enh_systest
There is a possibility that the state directory locking fails when another stream thread is taking long to close all tasks. Simple retries should alleviate the problem.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#899 from ymatsuda/minor2
This PR includes a number of clean-ups:
* Code style
* Documentation wording improvements
* Efficiency improvements
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#943 from ijuma/kafka-3259-kip-31-32-clean-ups
Also update `kafka-merge-pr.py` and `tests/kafkatest/__init__.py`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#963 from ijuma/update-trunk-0.10.0.0-SNAPSHOT
…able remote debugging to Kafka tools scripts
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#955 from christian-posta/ceposta-enable-jvm-debugging-opts
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#931 from hachikuji/KAFKA-3007
guozhangwang
My bad. I removed ZOOKEEPER_CONNECT_CONFIG from consumer's config by mistake. It is needed by our own partition assigner running in consumers.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#959 from ymatsuda/hotfix3
KAFKA-3242: minor rename / logging change to references to 'adding partitions' to indicate 'modifying partitions'
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Grant Henke
Closes#924 from benstopford/small_changes
StreamThread should keep going after a commit was failed due to a group rebalance.
Currently the thread just dies.
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#933 from ymatsuda/catch_commit_failure
Author: zhuchen1018 <amandazhu19620701@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Joel Koshy <jjkoshy.w@gmail.com>, Dong Lin <lindong28@gmail.com>
Closes#935 from zhuchen1018/minor-remove-unused-imports
This is the latest version in Maven even though HISTORY.md includes releases all the way to 4.5.0.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <ghenke@cloudera.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#937 from ijuma/update-rocks-db-for-streams
These are minor, but no reason to make our example code look worse than it has to.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke
Closes#940 from ijuma/example-style-improvements
Author: Frank Scholten <frank@frankscholten.nl>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#941 from frankscholten/tests/cluster-connection-states
… with the Vagranfile.local file
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Gwen Shapira
Closes#942 from christian-posta/ceposta-doco
See KIP-31 and KIP-32 for details.
A few notes on the patch:
1. This patch implements KIP-31 and KIP-32. The patch includes features in both KAFKA-3025, KAFKA-3026 and KAFKA-3036
2. All unit tests passed.
3. The unit tests were run with new and old message format.
4. When message format conversion occurs during consumption, the consumer will not be able to detect the message size too large situation. I did not try to fix this because the situation seems rare and only happen during migration phase.
Author: Jiangjie Qin <becket.qin@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Author: Jiangjie (Becket) Qin <becket.qin@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Anna Povzner <anna@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#764 from becketqin/KAFKA-3025
Removing streams' specific config params from producer/consumer configs to reduce warning messages.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#906 from ymatsuda/clean_config
1. Added a test case to prove commit() on SourceTask was not being called.
2. Added commitSourceTask() which logs potential exceptions.
3. Added after call to finishSuccessfulFlush().
Author: Jeremy Custenborder <jeremy@scarcemedia.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#909 from jcustenborder/KAFKA-3225
Author: zhuchen1018 <amandazhu19620701@gmail.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#911 from zhuchen1018/KAFKA-2757
…tionListener introduced as part of KAFKA-2211
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Reviewers: Flavio Junqueira <fpj@apache.org>, Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <mail@harsha.io>
Closes#679 from Parth-Brahmbhatt/KAFKA-2547 and squashes the following commits:
1722c76 [Parth Brahmbhatt] Addressing review comments.
376f77d [Parth Brahmbhatt] Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into KAFKA-2547
a13b963 [Parth Brahmbhatt] Addressing comments from Reviewers.
1007137 [Parth Brahmbhatt] KAFKA-2547: Make DynamicConfigManager to use the ZkNodeChangeNotificationListener introduced as part of KAFKA-2211
…uests equivalent
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <mail@harsha.io>
Closes#896 from granthenke/update-metadata and squashes the following commits:
2eb5d59 [Grant Henke] Address reviews
497258d [Grant Henke] KAFKA-2508: Replace UpdateMetadata{Request,Response} with o.a.k.c.requests equivalent
Guozhang Wang
Ismael Juma
Yasuhiro Matsuda
Eno Thereska
Kim Christensen
Parth Brahmbhatt
Tom Dearman
Anna Povzner
Ashish Singh
Richard Whaling
Jason Gustafson
Tom Lee
Gwen Shapira
Ewen Cheslack-Postava
Christian Posta
Xin Wang
Ben Stopford
zhuchen1018
Jiangjie Qin
Frank Scholten
Jun Rao
Jeremy Custenborder
Grant Henke