Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>
Closes#1656 from hachikuji/KAFKA-3977
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1670 from ewencp/kafka-3851-automate-release-notes
…ceful shutdown
The patch is pretty simple and the justification is explained in https://issues.apache.org/jira/browse/KAFKA-3924
I could not find Andrew Olson, who seems to be the contributor of this part of the code, in github so I am not sure whom I should ask to review the patch.
the contribution is my original work and that i license the work to the project under the project's open source license.
Author: Maysam Yabandeh <myabandeh@dropbox.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Andrew Olson <andrew.olson@cerner.com>, Jun Rao <junrao@gmail.com>
Closes#1634 from maysamyabandeh/KAFKA-3924
Also:
* Introduce a blocking variant to be used by `FileMessageSet.append`
* Add tests
* Minor clean-ups
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1669 from ijuma/kafka-3996-byte-buffer-message-set-write-to-non-blocking
They are now consistent with `waitUntilTrue`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1665 from ijuma/increase-default-wait-until-time
ijuma
As discussed in https://github.com/apache/kafka/pull/1645, this patch removes an extraneous line from several __init__.py files, and a few others as well
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1659 from granders/minor-cleanup-init-files
Fix the test by using a more liberal timeout and forcing more frequent SinkTask.put() calls. Also add some logging to aid future debugging.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1663 from ewencp/kafka-3935-fix-restart-system-test
Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.
Seeeing as `messageFormat > 0` always reads the full compressed message
set and is the default going forwards, we can use that behaviour to
always close the compressor when calling `deepIterator`
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1660 from tcrayford/dont_leak_native_memory_round_2
I’ve implemented my own custom Deserializer and been using it with `KStream.reduceByKey`; I observed that `reduceByKey` was passing null to my implementation, but it wasn’t clear to me what my implementation was expected to do in this case. So this attempts to clarify it.
This is my original work and I license this work to the Kafka project under Kafka’s open source license (the Apache License 2.0).
Author: Avi Flax <avi@aviflax.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1503 from aviflax/patch-1
(cherry picked from commit 6b1a6d9551)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Author: Alexey Romanchuk <al.romanchuk@2gis.ru>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1629 from 13h3r/kafka-3960
Add additional information to Acceptor debug message upon connection acceptance
Author: rnpridgeon <ryan.n.pridgeon@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1648 from rnpridgeon/trunk
5 seconds is probably enough when running tests locally, but
doesn't seem to be so for Jenkins when it is overloaded.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1589 from ijuma/increase-default-wait-time-for-wait-until-true
Better performance is always welcome:
"The Gradle build itself has seen a 50% reduction in configuration time. You'll see the biggest impact on multi-project builds"
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1644 from ijuma/update-gradle
Without this file the benchmark does not run nightly.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Geoff Anderson <geoff@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1645 from enothereska/hotfix-streams-test
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Roger Hoover, Matthias J. Sax, Guozhang Wang
Closes#1619 from enothereska/KAFKA-3858-print-topology
Add new config StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG to enable advanced
RocksDB users to override default RocksDB configuration
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Roger Hoover, Dan Norwood, Eno Thereska, Guozhang Wang
Closes#1640 from dguy/kafka-3740-listener
Currently, logs involving PartitionState are not very helpful.
```
Broker 449 cached leader info org.apache.kafka.common.requests.UpdateMetadataRequest$PartitionState3285d64a for partition <topic>-<partition> in response to UpdateMetadata request sent by controller 356 epoch 138 with correlation id 0
TRACE state.change.logger: Broker 449 received LeaderAndIsr request org.apache.kafka.common.requests.LeaderAndIsrRequest$PartitionState66d6a8eb correlation id 3 from controller 356 epoch 138 for partition [<topic>,<partition>]
```
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1609 from SinghAsDev/partitionState
…broker id
This is because the id passed into the MetadataCache is the value from the config before the real broker id is generated.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1632 from granthenke/metadata-id
The conversion is unsafe as the converted message size may be greater
than the message size limit. Updated `LogCleanerIntegrationTest` to test the max message size case for both V0 and the current version.
Also include a few minor clean-ups:
* Remove unused code branch in `LogCleaner.compressMessages`
* Avoid unintentional usage of `scala.collection.immutable.Stream` (`toSeq` on an `Iterator`)
* Add explicit result type in `FileMessageSet.iterator`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang
Closes#1643 from ijuma/kafka-3915-log-cleaner-io-buffers-message-conversion
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1635 from ijuma/remove-unused-parameter-in-check-if-partition-reassignment-succeeded and squashes the following commits:
f9ed930 [Ismael Juma] Code style improvements in `ReassignPartitionsCommand`
66c7541 [Ismael Juma] Fix comment in `KafkaController.onPartitionReassignment`
85288f3 [Ismael Juma] Remove unused parameter from `checkIfPartitionReassignmentSucceeded`
Merge of KAFKA-3812 caused a compilation error in StreamThreadStateStoreProviderTest
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1641 from dguy/fix-compile-error
Move all state directory creation/locking/unlocking/cleaning to a single class. Don't release the channel until the lock is released. Refactor code to make use of new class
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Ismael Juma, Guozhang Wang
Closes#1628 from dguy/kafka-3812
As kafka-streams is intended to be used by applications that may or may not wish to use log4j, kafka-streams itself should not have a dependency on a concrete log framework. This change adapts the dependencies to be API-only for compile, and framework-specific for the test runtime only.
I read through the [Contributing Code Guidelines](https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes) and interpreted this as a trivial change that doesn't require a Jira ticket. Please let me know if I've interpreted that wrongly.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Mathieu Fenniak <mathieu@encouragemarketing.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1639 from mfenniak/fix-slf4j-dependency-for-streams
guozhangwang enothereska please review
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Eno Thereska, Matthias J. Sax, Michael G. Noll, Guozhang Wang
Closes#1565 from dguy/kafka-3912
https://issues.apache.org/jira/browse/KAFKA-3922
KAFKA-3922 add copy-constructor to AbstractStream class
This copy-constructor allow to access protected variables from subclasses.
It should be used to extend KStreamImpl and KTableImpl classes by implementing a decorator pattern.
Author: Florian Hussonnois <florian.hussonnois@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1581 from fhussonnois/KAFKA-3922
Mark all public `TopologyBuilder` methods as synchronized as they can modify data-structures and these methods could be called from multiple threads
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1633 from dguy/kafka-3855
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1630 from ssaamm/trunk
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Geoff Anderson, Guozhang Wang, Ismael Juma
Closes#1621 from enothereska/simple-benchmark-streams-system-tests
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy, Matthias J. Sax, Michael G. Noll, Guozhang Wang
Closes#1526 from enothereska/expose-names-dsl
Author: Wan Wenli <wwl.990@hotmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1612 from swwl1992/ticket-KAFKA-3952-fix-consumer-rebalance-verifier
- Added validity checks for input parameters on subscribe, assign to avoid NPE, and provide an argument exception instead
- Updated behavior for subscription with null collection to be same as when subscription with emptyList.i.e., unsubscribes.
- Added tests on subscription, assign
Author: Rekha Joshi <rekhajoshm@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1601 from rekhajoshm/KAFKA-3905-1
Also move the initialization that restores from changelog to inner stores.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska, Dan Norwood
Closes#1610 from guozhangwang/K3941-avoid-eviction-listener
Full credit for figuring out the cause of these failures goes to hachikuji.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Guozhang Wang, Ismael Juma, Jason Gustafson
Closes#1594 from vahidhashemian/KAFKA-3931
Also made a pass over the streams unit tests, with the following changes:
1. Removed three integration tests as they are already covered by other integration tests.
2. Merged `KGroupedTableImplTest` into `KTableAggregateTest`.
3. Use mocks whenever possible to reduce code duplicates.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1604 from guozhangwang/Kminor-unit-tests-consolidation
Fix timing window in producer by holding onto cluster object while processing send requests so that changes to cluster during metadata refresh don't cause NPE if a topic is deleted.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1478 from rajinisivaram/KAFKA-3562
It was previously only deleting files/folders where the path started with /tmp. Changed it to delete from the value of the System Property `java.io.tmpdir`. Also changed the tests that were creating State dirs under /tmp to just use `TestUtils.tempDirectory(..)`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1600 from dguy/kafka-3942
I've updated the ops documentation with information on using the XFS filesystem, based on LinkedIn's testing (and subsequent switch from EXT4).
I've also added some information to clarify the potential risk to the suggested EXT4 options (again, based on my experience with a multiple broker failure situation).
Author: Todd Palino <tpalino@linkedin.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Dana Powers <dana.powers@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1605 from toddpalino/trunk
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1515 from SinghAsDev/KAFKA-3849
Author: Nafer Sanabria <nafr.snabr@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1595 from naferx/minor-typo
Since 0.9.0.1 Configuration parameter log.cleaner.enable is now true by default.
Author: Nihed MBAREK <nihedmm@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1592 from nihed/patch-1
Ismael Juma
Jason Gustafson
Ewen Cheslack-Postava
Matthias J. Sax
Maysam Yabandeh
Geoff Anderson
Tom Crayford
Avi Flax
Alexey Romanchuk
Ryan Pridgeon
Eno Thereska
Damian Guy
Ashish Singh
Grant Henke
Mathieu Fenniak
Florian Hussonnois
Samuel Taylor
Yuto Kawamura
Wan Wenli
Rekha Joshi
Guozhang Wang
Vahid Hashemian
Rajini Sivaram
Todd Palino
Nafer Sanabria
Nihed Bbarek