As pointed out by granthenke in #196, error can be wrapped in JShort directly to avoid second map.
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#714 from dajac/wrap-jshort
This fixes failure of the FileStreamSinkTaskTest unit test on Windows
Signed-off-by: glikson <gliksonil.ibm.com>
Author: glikson <glikson@il.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#708 from glikson/file-ut-newline
- Adds CheckStyle to core and examples modules
- Fixes any existing CheckStyle issues
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#703 from granthenke/checkstyle-core
Summary of code changes
------------------------------------
1) Added a new Checkstyle rule to flag any code using star imports
2) Fixed ALL existing code violations using star imports
Testing
-----------
Local build was successful
ALL JUnits ran successfully on local.
ewencp - Request you to please review changes. Thank you !
I state that the contribution is my original work and I license the work to the project under the project's open source license.
Author: manasvigupta <manasvigupta@yahoo.co.in>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#700 from manasvigupta/KAFKA-3009
DelayedFetchMetrics are loaded dynamically in function "onExpiration";
use assertNull(DelayedFetchMetrics) to initialize DelayedFetchMetrics explicitly;
Author: jinxing <jinxing@fenbi.com>
Author: ZoneMayor <jinxing6042@126.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#694 from ZoneMayor/trunk-KAFKA-2455
It turns that waitUntilMetadataIsPropagated is not enough;
in "onBrokerStartup", methods below will send send both LeaderAndIsrRequest and UpdateMetadataRequest to KafkaApis:
replicaStateMachine.handleStateChanges(allReplicasOnNewBrokers, OnlineReplica)
partitionStateMachine.triggerOnlinePartitionStateChange()
The two kinds of request are handled seperately and we are not sure about the order;
If UpdateMetadataRequest is handled first, metadataCache of kafkaApis will be updated, thus TestUtils.waitUntilMetadataIsPropagated will be satisfied, and consumer can(will) start fetching data;
But if the LeaderAndIsrRequest is not handled at this moment, "becomeLeaderOrFollower" cannot be called , thus structures like "leaderReplicaOpt" cannot be updated, which leads to failure of consumer's fetching data;
To fix above, consumer should start fetching data after partition's leaderReplica is refreshed, not just the leader is elected;
So added "TestUtils.waitUntilLeaderIsKnown(servers, topic, 0)"
Author: ZoneMayor <jinxing6042@126.com>
Author: jinxing <jinxing@fenbi.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#689 from ZoneMayor/trunk-KAFKA-2058
We always return a `Struct` from `Schema.read` and this means that
we can remove a large number of casts.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#684 from ijuma/schema-read-should-return-struct
The core of this PR is to ensure we evaluate enabling security in a running cluster where we have different broker and client protocols.
Also in this PR are some improvements to the validation process in produce_consume_validate.py which make it easier to work out where missing messages have been lost:
- Fail fast if producer or consumer stop running.
- If messages go missing, check in the data files to see if the cause was data loss or the consumer missing messages.
- Make it possible for the ConsoleConsumer to log both what it consumed and when it consumed it (and enable this feature in produce_consume_validate tests)
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Gwen Shapira, Geoff Anderson
Closes#667 from benstopford/security-rolling_upgrade-additions
The link to 'Producer Configs' section of the documentation is updated with this PR.
Author: vahidhashemian <vahidhashemian@us.ibm.com>
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Gwen Shapira
Closes#692 from vahidhashemian/typo03/fix_broken_doc_link
guozhangwang
At DAG level, `KTable<K,V>` sends (key, (new value, old value)) to down stream. This is done by wrapping the new value and the old value in an instance of `Change<V>` class and sending it as a "value" part of the stream. The old value is omitted (set to null) by default for optimization. When any downstream processor needs to use the old value, the framework should enable it (see `KTableImpl.enableSendingOldValues()` and implementations of `KTableProcessorSupplier.enableSensingOldValues()`).
NOTE: This is meant to be used by aggregation. But, if there is a use case like a SQL database trigger, we can add a new KTable method to expose this.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#672 from ymatsuda/trigger
Credit to Gwen for some of the text.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#678 from ijuma/mirror-maker-compatibility-note
We're seeing some GC pause issues in production, and during our investigation found that the thunks created during invocation of three trace statements guarded in this PR were responsible for ~98% of all allocations by object count and ~90% by size. While I'm not sure that this was actually the cause of our issue, it seems prudent to avoid useless allocations in a tight loop.
I realize that the trace() call does its own guarding internally, however it's insufficient to prevent allocation of the thunk.
This is my original work, and I license it to the Kafka project under the project's Apache license.
Author: Cory Kolbeck <cory.kolbeck@urbanairship.com>
Reviewers: Gwen Shapira
Closes#682 from ckolbeck/guard-trace-statements
Mention Java 7 requirement, fix and improve formatting and remove obsolete reference to SVN website.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Jason Gustafson <jason@confluent.io>
Closes#677 from ijuma/mention-java-requirement-in-readme
People are facing problems upgrading their clusters with configured broker IDs above 1000 due to `reserved.broker.max.id` which wasn't very well announced.
This PR attempts to improve that somewhat by fixing the broker config docs and adding a note to the upgrade documentation.
Author: Magnus Edenhill <magnus@edenhill.se>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#670 from edenhill/docs-reserved.broker.max.id
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#675 from hachikuji/KAFKA-2990
I can reproduced this transient failure, it seldom happen;
code is like below:
// rolling bounce brokers
for (i <- 0 until numServers) {
for (server <- servers) {
server.shutdown()
server.awaitShutdown()
server.startup()
Thread.sleep(2000)
}
// Make sure the producer do not see any exception
// in returned metadata due to broker failures
assertTrue(scheduler.failed == false)
// Make sure the leader still exists after bouncing brokers
(0 until numPartitions).foreach(partition => TestUtils.waitUntilLeaderIsElectedOrChanged(zkUtils, topic1, partition))
Brokers keep rolling restart, and producer keep sending messages;
In every loop, it will wait for election of partition leader;
But if the election is slow, more messages will be buffered in RecordAccumulator's BufferPool;
The limit for buffer is set to be 30000;
TimeoutException("Failed to allocate memory within the configured max blocking time") will show up when out of memory;
Since for every restart of the broker, it will sleep for 2000 ms, so this transient failure seldom happen;
But if I reduce the sleeping period, the bigger chance failure happens;
for example if the broker with role of controller suffered a restart, it will take time to select controller first, then select leader, which will lead to more messges blocked in KafkaProducer:RecordAccumulator:BufferPool;
In this fix, I just enlarge the producer's buffer size to be 1MB;
guozhangwang , Could you give some comments?
Author: jinxing <jinxing@fenbi.com>
Author: ZoneMayor <jinxing6042@126.com>
Reviewers: Guozhang Wang
Closes#648 from ZoneMayor/trunk-KAFKA-2837
guozhangwang
* a test for ktable state store creation
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#661 from ymatsuda/more_ktable_test
Moves test output from the project files and allows `gradle clean` to clean up the output.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#664 from granthenke/target
Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ben Stopford <ben@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#657 from granders/KAFKA-2927-reduce-log-footprint
Partition re-assignment tests with and without broker failure.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Ben Stopford <ben@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Geoff Anderson <geoff@confluent.io>
Closes#655 from apovzner/kafka_2896
Fixed version sanity checks by updated kafkatest version to match kafka version
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#656 from granders/KAFKA-2928-fix-version-sanity-checks
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#659 from ijuma/KAFKA-2578
A few issues found via static analysis.
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Sriharsha Chintalapani, Guozhang Wang
Closes#652 from ijuma/use-equals-instead-of-==
The change in `docs/design.html` is hard to catch in the diff -- a `tbe` is changed to `the`. All other changes show up clearly in the diff.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Alex Loddengaard <alexloddengaard@gmail.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#654 from alexlod/doc-typo-fixes
onurkaraman becketqin Do you have time to review this patch? It addresses the ticket that jjkoshy filed in KAFKA-2668.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Joel Koshy <jjkoshy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#328 from lindong28/KAFKA-2668
Grant Henke
David Jacot
glikson
Jason Gustafson
Vahid Hashemian
manasvigupta
Jin Xing
Jin Xing
Ismael Juma
Ben Stopford
Yasuhiro Matsuda
Cory Kolbeck
Gwen Shapira
Magnus Edenhill
Guozhang Wang
Xin Wang
Geoff Anderson
Anna Povzner
Edward Ribeiro
Alex Loddengaard
Dong Lin
unknown