Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1786 from hachikuji/hotfix-ctrlchannelmgr-verbose-logging
Change cleanup.policy to accept a comma separated list of valid policies.
Updated LogCleaner.CleanerThread to also run deletion for any topics configured with compact,delete.
Ensure Log.deleteSegments only runs when delete is enabled.
Additional Integration and unit tests to cover new option
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1742 from dguy/kafka-4015
rocksdbjni version 4.9.0 now includes support for running on Windows; this PR updates Kafka Stream's dependency to that version. Tests pass locally, except for a timeout in testReprocessingFromScratchAfterReset that doesn't seem related; it happens with and without this change.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Mathieu Fenniak <mathieu.fenniak@replicon.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1783 from mfenniak/update-rocksdb-4.9
Also upgrade scoverage (required for compatibility) and remove usage of
`useAnt` which doesn't exist in Gradle 3.0
It turns out that one cannot even run `gradle` to download the project Gradle version if `useAnt` is used in the build. This is unfortunate (the SBT launcher has much saner behaviour).
Release notes: https://docs.gradle.org/3.0/release-notes
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#1774 from ijuma/kafka-4082-support-gradle-3.0
ijuma - Making the change against trunk based on your suggestions to have the stream closing handled in the private RecordIterator constructor which I understand is only to be used only if the block of message(s) are compressed.
Author: William Yu <wyu@unified.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1760 from wiyu/compressor_memory_leak_in_fetcher
Configure default classes using class objects instead of class names, enable configurable lists of classes to be specified as class objects, add tests for different classloader configurations.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1421 from rajinisivaram/KAFKA-3680
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1736 from guozhangwang/Kminor-topology-applicationID
handled by adding a catch all for any unhandled exception. Because the jira specifically mentions the InvalidReplicationFactor exception, a test was added for that specific case.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1739 from granthenke/create-errors
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <me@ewencp.org>
Closes#1746 from guozhangwang/K4049-RegexSourceIntegrationTest-failure
Use System.nanoseconds instead of System.currentTimeMillis in broker timer tasks to cope with changes to wall-clock time.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Gwen Shapira
Closes#1768 from rajinisivaram/KAFKA-4051
1. I think the instructions in step 2 of the security section which describe adding the CA to server/client truststores are swapped. That is, the instruction that says to add the CA to the server truststore adds it to the client truststore (and vice versa).
2. "clients keys" should be possessive ("clients' keys").
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1651 from ssaamm/trunk
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1763 from rajinisivaram/KAFKA-4066
As discussed in https://issues.apache.org/jira/browse/KAFKA-3894, this PR makes the log cleaner do a "partial" clean on a segment, whereby it builds a partial offset map up to a particular offset in a segment. Once cleaning resumes again, we will continue from the next dirty offset, which can now be located in the middle of a segment.
Prior to this PR, segments with overly numerous keys could crash the log cleaner thread, as it was required that the log cleaner had to fit at least a single segment in the offset map.
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1725 from tcrayford/dont_crash_log_cleaner_thread_if_segment_overflows_buffer
junrao Could you take a look when get a chance? Thanks.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1769 from becketqin/KAFKA-3163-follow-up
In case of unknown configs, only list the name without the value
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Jaikiran, Gwen Shapira, Grant Henke, Ryan Pridgeon, Dustin Cote
Closes#1759 from mimaison/KAFKA-4056
Add an optional configuration for the SecureRandom PRNG implementation, with the default behavior being the same (use the default implementation in the JDK/JRE).
Author: Todd Palino <Todd Palino>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#1747 from toddpalino/trunk
On Windows, the following output is seen when starting Zookeeper and Kafka servers:
```
'#' is not recognized as an internal or external command,
operable program or batch file.
```
This pull request makes a minor correction to the Windows `kafka-run-class.bat` script to replace the use of `#` with `rem`.
Author: P. Thorpe <paul.thorpe@uk.ibm.com>
Reviewers: Vahid Hashemian, Gwen Shapira
Closes#1740 from p-thorpe/trunk
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Dana Powers, Gwen Shapira
Closes#1732 from vahidhashemian/minor/clarify_producer_config_documentation
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1733 from ewencp/rest-api-retries
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#1627 from hachikuji/KAFKA-3888
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Greg Fodor <gfodor@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1530 from guozhangwang/K3769-per-thread-metrics
Author: Manikumar Reddy O <manikumar.reddy@gmail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Jun Rao <junrao@gmail.com>
Closes#1723 from omkreddy/KAFKA-4035
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1730 from granthenke/test-delete
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1720 from hachikuji/KAFKA-4034
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <me@ewencp.org>, Jun Rao <junrao@gmail.com>
Closes#1616 from granthenke/delete-wire-new
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson, Gwen Shapira
Closes#1727 from ewencp/kafka-3847-per-task-producers and squashes the following commits:
7d39724 [Ewen Cheslack-Postava] Add timeout for closing producers.
98ec7f6 [Ewen Cheslack-Postava] KAFKA-3847: Use a separate producer per source task
A small fix to check null before using the reference
Author: Som Sahu <sosahu@microsoft.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1718 from soumyajit-sahu/nullCheckForDirectBufferCleaner
guozhangwang enothereska mjsax miguno please take a look. A few things that need to be clarified
1. I've added StreamsConfig.USER_ENDPOINT_CONFIG, but should we have separate configs for host and port or is this one config ok?
2. `HostState` in the KIP has a byte[] field - not sure why and what it would be populated with
3. I've changed the API to return `List<KafkaStreamsInstance>` as opposed to `Map<HostInfo, Set<TaskMetadata>>` as i find this far more intuitive to work with.
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax, Michael G. Noll, Eno Thereska, Guozhang Wang
Closes#1576 from dguy/kafka-3914v2
Added non null checks to parameters supplied via the DSL and `TopologyBuilder`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Edward Ribeiro <edward.ribeiro@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#1711 from dguy/kafka-3936
kafka-run-class.sh now runs under Cygwin; paths and classpath are set up properly.
**WARNING:** The script was not tested on a Linux machine, only under Cygwin. Prior to merge it into trunk, if accepted, please run a quick test to ensure nothing broke. From my own code review, there should not be any problem, but we can never be too sure.
I do not have the environment to test it under Linux at this moment.
Author: deragon <32nx9812masakjds>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1487 from deragon/trunk
It previously hardcoded it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1613 from ijuma/kafka-3954-consumer-internal-topics-from-broker
Jason Gustafson
Matthias J. Sax
Damian Guy
sven0726
Mathieu Fenniak
Ismael Juma
William Yu
Konstantine Karantasis
Rajini Sivaram
Guozhang Wang
Grant Henke
Samuel Taylor
Tom Crayford
Jiangjie Qin
Eno Thereska
Shuai Zhang
Mickael Maison
Todd Palino
Ewen Cheslack-Postava
P. Thorpe
Vahid Hashemian
Manikumar Reddy O
Alexey Ozeritsky
Bryan Baugher
Som Sahu
sahil kharb
deragon