Fix two minor typos in Kafka official website page.
Author: Gabriel Zhang <smalldirector@yahoo.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#742 from smalldirector/kafka-document-typos-fix
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#686 from granthenke/compaction
Deprecates ErrorMapping.scala in core in favor or Errors.java in common.
Duplicated exceptions in core are deprecated as well, to ensure the mapping is correct.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#616 from granthenke/error-mapping
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#196 from dajac/KAFKA-2072-part-2
The check was implemented in KAFKA-340 : If we are shutting down a broker when the ISR of a partition includes only that broker, we could lose some messages that have been previously committed. For clean shutdown, we need to guarantee that there is at least 1 other broker in ISR after the broker is shut down.
When we are deleting the topic, this check can be avoided.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Dong Lin <lindong28@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>, Jun Rao <junrao@gmail.com>
Closes#729 from MayureshGharat/kafka-2937
Author: Konrad <konkalita@gmail.com>
Author: konradkalita <konkalita@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#732 from konradkalita/Kafka-3058
I verified this by trying to produce to __consumer_offsets and the logged message looks like:
[2015-12-22 10:34:40,897] INFO [KafkaApi-0] Closing connection due to error during produce request with correlation id 1 from client id console-producer with ack=0
Topic and partition to exceptions: [__consumer_offsets,43] -> kafka.common.InvalidTopicException (kafka.server.KafkaApis)
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#709 from ijuma/kafka-3010-include-error-in-log-when-ack-0
Also include some trivial clean-ups in `ProducerConfig`and `BaseProducer`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#710 from ijuma/use-new-producer-properties-in-config
Fix it by making it possible to pass the `doLog` parameter to `AbstractConfig`. As explained in the code comments, this means that we can continue to benefit from ZK default settings as specified in `KafkaConfig` without having to duplicate code.
Also:
* Removed unused private methods from `KafkaConfig`
* Removed `case` modifier from `KafkaConfig` so that `hashCode`, `equals`
and `toString` from `AbstractConfig` are used.
* Made `props` a `val` and added `apply` method to `KafkaConfig` to
remain binary compatible.
* Call authorizer.close even if an exception is thrown during `configure`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang
Closes#725 from ijuma/kafka-3052-broker-properties-get-logged-twice-if-acl-enabled
In Kafka 0.9, request.required.acks=-1 which configration of producer is replaced by acks=all,
but this old config is remained in docs.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Gwen Shapira
Closes#716 from sasakitoa/acks_doc
As pointed out by granthenke in #196, error can be wrapped in JShort directly to avoid second map.
Author: David Jacot <david.jacot@gmail.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#714 from dajac/wrap-jshort
- Adds CheckStyle to core and examples modules
- Fixes any existing CheckStyle issues
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#703 from granthenke/checkstyle-core
DelayedFetchMetrics are loaded dynamically in function "onExpiration";
use assertNull(DelayedFetchMetrics) to initialize DelayedFetchMetrics explicitly;
Author: jinxing <jinxing@fenbi.com>
Author: ZoneMayor <jinxing6042@126.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#694 from ZoneMayor/trunk-KAFKA-2455
It turns that waitUntilMetadataIsPropagated is not enough;
in "onBrokerStartup", methods below will send send both LeaderAndIsrRequest and UpdateMetadataRequest to KafkaApis:
replicaStateMachine.handleStateChanges(allReplicasOnNewBrokers, OnlineReplica)
partitionStateMachine.triggerOnlinePartitionStateChange()
The two kinds of request are handled seperately and we are not sure about the order;
If UpdateMetadataRequest is handled first, metadataCache of kafkaApis will be updated, thus TestUtils.waitUntilMetadataIsPropagated will be satisfied, and consumer can(will) start fetching data;
But if the LeaderAndIsrRequest is not handled at this moment, "becomeLeaderOrFollower" cannot be called , thus structures like "leaderReplicaOpt" cannot be updated, which leads to failure of consumer's fetching data;
To fix above, consumer should start fetching data after partition's leaderReplica is refreshed, not just the leader is elected;
So added "TestUtils.waitUntilLeaderIsKnown(servers, topic, 0)"
Author: ZoneMayor <jinxing6042@126.com>
Author: jinxing <jinxing@fenbi.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#689 from ZoneMayor/trunk-KAFKA-2058
We always return a `Struct` from `Schema.read` and this means that
we can remove a large number of casts.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#684 from ijuma/schema-read-should-return-struct
The core of this PR is to ensure we evaluate enabling security in a running cluster where we have different broker and client protocols.
Also in this PR are some improvements to the validation process in produce_consume_validate.py which make it easier to work out where missing messages have been lost:
- Fail fast if producer or consumer stop running.
- If messages go missing, check in the data files to see if the cause was data loss or the consumer missing messages.
- Make it possible for the ConsoleConsumer to log both what it consumed and when it consumed it (and enable this feature in produce_consume_validate tests)
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Gwen Shapira, Geoff Anderson
Closes#667 from benstopford/security-rolling_upgrade-additions
We're seeing some GC pause issues in production, and during our investigation found that the thunks created during invocation of three trace statements guarded in this PR were responsible for ~98% of all allocations by object count and ~90% by size. While I'm not sure that this was actually the cause of our issue, it seems prudent to avoid useless allocations in a tight loop.
I realize that the trace() call does its own guarding internally, however it's insufficient to prevent allocation of the thunk.
This is my original work, and I license it to the Kafka project under the project's Apache license.
Author: Cory Kolbeck <cory.kolbeck@urbanairship.com>
Reviewers: Gwen Shapira
Closes#682 from ckolbeck/guard-trace-statements
People are facing problems upgrading their clusters with configured broker IDs above 1000 due to `reserved.broker.max.id` which wasn't very well announced.
This PR attempts to improve that somewhat by fixing the broker config docs and adding a note to the upgrade documentation.
Author: Magnus Edenhill <magnus@edenhill.se>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#670 from edenhill/docs-reserved.broker.max.id
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#675 from hachikuji/KAFKA-2990
I can reproduced this transient failure, it seldom happen;
code is like below:
// rolling bounce brokers
for (i <- 0 until numServers) {
for (server <- servers) {
server.shutdown()
server.awaitShutdown()
server.startup()
Thread.sleep(2000)
}
// Make sure the producer do not see any exception
// in returned metadata due to broker failures
assertTrue(scheduler.failed == false)
// Make sure the leader still exists after bouncing brokers
(0 until numPartitions).foreach(partition => TestUtils.waitUntilLeaderIsElectedOrChanged(zkUtils, topic1, partition))
Brokers keep rolling restart, and producer keep sending messages;
In every loop, it will wait for election of partition leader;
But if the election is slow, more messages will be buffered in RecordAccumulator's BufferPool;
The limit for buffer is set to be 30000;
TimeoutException("Failed to allocate memory within the configured max blocking time") will show up when out of memory;
Since for every restart of the broker, it will sleep for 2000 ms, so this transient failure seldom happen;
But if I reduce the sleeping period, the bigger chance failure happens;
for example if the broker with role of controller suffered a restart, it will take time to select controller first, then select leader, which will lead to more messges blocked in KafkaProducer:RecordAccumulator:BufferPool;
In this fix, I just enlarge the producer's buffer size to be 1MB;
guozhangwang , Could you give some comments?
Author: jinxing <jinxing@fenbi.com>
Author: ZoneMayor <jinxing6042@126.com>
Reviewers: Guozhang Wang
Closes#648 from ZoneMayor/trunk-KAFKA-2837
Moves test output from the project files and allows `gradle clean` to clean up the output.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#664 from granthenke/target
onurkaraman becketqin Do you have time to review this patch? It addresses the ticket that jjkoshy filed in KAFKA-2668.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Onur Karaman <okaraman@linkedin.com>, Joel Koshy <jjkoshy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#328 from lindong28/KAFKA-2668
`Iterator.continually` is more efficient (it doesn't allocate a `Cons` instance per element) and we don't need the extra functionality provided by `Stream.continually`.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava
Closes#106 from ijuma/kafka-2399-replace-stream-continually
It used to be used by MirrorMaker but its usage was removed in KAFKA-1997.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Jiangjie Qin
Closes#638 from granthenke/remove-dll
kafka-console-consumer.sh is showing an incorrect number of
messages processed, counting one more message than the actual
number of processed messages.
Author: Luciano Afranllie <luafran@gmail.com>
Reviewers: Guozhang Wang
Closes#617 from luafran/console-consumer-number-of-processed-messages
A few notes on the added test:
* I verified this test fails when changing between snappy 1.1.1.2 and 1.1.1.7 (per KAFKA-2189)
* The hard coded numbers are passing before and after lzo change
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#552 from granthenke/lz4
Author: Flavio Junqueira <fpj@apache.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ben Stopford <benstopford@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#410 from fpj/KAFKA-2732
Currently we don't log exceptions raised when reading from the local log which makes tracking down the cause of problems a bit tricky.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Guozhang Wang
Closes#593 from benstopford/small-patches
David Jacot
Grant Henke
Gabriel Zhang
Mayuresh Gharat
Konrad
Dong Lin
Ismael Juma
Flavio Junqueira
Sasaki Toru
ouyangliduo
Jin Xing
Jin Xing
Ben Stopford
Cory Kolbeck
Vahid Hashemian
Magnus Edenhill
Guozhang Wang
Jason Gustafson
Gwen Shapira
unknown
Luciano Afranllie