Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira <cshapi@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#2354 from ijuma/kafka-4565-separation-of-internal-and-external-traffic
Besides API and runtime changes, this PR also includes 2 data transformations (`InsertField`, `HoistToStruct`) and 1 routing transformation (`TimestampRouter`).
There is some gnarliness in `ConnectorConfig` / `ConfigDef` around creating, parsing and validating a dynamic `ConfigDef`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2299 from shikhar/smt-2017
KAFKA-4603 the argument of shell in doc wrong and command parsed error
In "7.6.2 Migrating clusters" of document security.html, the argument "--zookeeper.connection" of shell "zookeeper-security-migrat.sh" is wrong and the using of OptionParser is not correct
This patch corrected the doc and changed the OptionParser constructor
Author: auroraxlh <xin.lihua1@zte.com.cn>
Reviewers: Xi Hu <huxi_2b@hotmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#2322 from auroraxlh/my-issue
Changed caching in LoginManager to allow one LoginManager per client
JAAS configuration.
Added test to End2EndAuthorization for SASL Plain and GSSAPI with two
consumers with different credentials.
Developed with mimaison.
Author: Edoardo Comar <ecomar@uk.ibm.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2293 from edoardocomar/KAFKA-4180d
Kafka Streams: add granular metrics per node and per task, also expose ability to register non latency metrics in StreamsMetrics
Also added different recording levels to Metrics.
This is joint contribution from Eno Thereska and Aarti Gupta.
from https://github.com/apache/kafka/pull/1362#issuecomment-218326690-------
We can consider adding metrics for process / punctuate / commit rate at the granularity of each processor node in addition to the global rate mentioned above. This is very helpful in debugging.
We can consider adding rate / total cumulated metrics for context.forward indicating how many records were forwarded downstream from this processor node as well. This is helpful in debugging.
We can consider adding metrics for each stream partition timestamp.
This is helpful in debugging.
## Besides the latency metrics, we can also add throughput latency in terms of source records consumed.
More discussions here https://issues.apache.org/jira/browse/KAFKA-3715, KIP-104, KIP-105
Author: Eno Thereska <eno@confluent.io>
Author: Aarti Gupta <aartiguptaa@gmail.com>
Reviewers: Greg Fodor, Ismael Juma, Damian Guy, Guozhang Wang
Closes#1446 from aartigupta/trunk
The client should send older versions of requests to the broker if necessary.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2264 from cmccabe/KAFKA-4507
Adds a bunch of tests to unit tests to the assignment command.
Moves the Rack aware test into its own class as it makes use of ZooKeeperTestHarness and slows everything else down.
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1950 from benstopford/os-rebalance-extra-unit-testing
Shut down the group coordinator before shutting down the log manager to
ensure that any delayed operations are completed before the logs are closed.
Author: steve <sniemitz@twitter.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#2311 from steveniemitz/KAFKA-4523
Jason recently cleaned things up significantly by consolidating the Message/Record classes
into the common Java code in the clients module. While reviewing that, I noticed a few things
that could be improved a little more. To make reviewing easier, there will be multiple PRs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Jason Gustafson <jason@confluent.io>
Closes#2271 from ijuma/records-minor-fixes
I was able to reproduce the failure in less than 10 runs before the change. With the change,
the test passed 70 times consecutively.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang
Closes#2298 from ijuma/kafka-4528-fix-test-async-send-timeout
This makes it consistent with MirrorMaker with the old consumer.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2072 from amethystic/kafka-4351_Regex_behavior_change_for_new_consumer
Mx4jLoader.scala should explicitly `return true` if the class is successfully loaded and started, otherwise it will return false even if the class is loaded.
Author: Edward Ribeiro <edward.ribeiro@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2295 from eribeiro/mx4jloader-bug
During Acceptor initialization, if "Address already in use" error is encountered,
the shutdown latch in each Processor is never counted down. Consequently,
the Kafka server hangs when `Processor.shutdown` is called.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2156 from amethystic/kafka-4428_Kafka_noexit_for_port_already_use
adding a LogConfig value validator. gwenshap or junrao would you mind taking a look?
Author: Dustin Cote <dustin@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1796 from cotedm/retentionbytesvalidation
The current output for the `--generate` option looks like this
```
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[0]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[1]}]}
```
This PR simply changes it to
```
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[0]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t1","partition":0,"replicas":[1]}]}
```
to make it more readable.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1868 from vahidhashemian/minor/improve_output_format_of_reassign_partitions
This was changed in b58b6a1bef and caused the `ReplicaVerificationToolTest.test_replica_lags`
system test to start failing.
I also added a unit test and a couple of other minor clean-ups.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2280 from ijuma/kafka-4554-fix-replica-buffer-verify-checksum
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Colin P. Mccabe <cmccabe@confluent.io>, Dana Powers <dana.powers@gmail.com>, Gwen Shapira <cshapi@gmail.com>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1251 from SinghAsDev/KAFKA-3600
The latter return `Iterable` instead of `Iterator` so that enhanced foreach can be used
in Java.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2261 from ijuma/deepEntries-shallowEntries
Fix OffsetIndex overflow when replicating a highly compacted topic.
https://issues.apache.org/jira/browse/KAFKA-4451
Author: Michael Schiff <schiff.michael@gmail.com>
Author: Michael Schiff <michael.schiff@tubemogul.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#2210 from michaelschiff/bug/4451
Author: Dong Lin <lindong28@gmail.com>
Author: Dong Lin <dolin@linkedin.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>
Closes#2241 from lindong28/KAFKA-4521
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2140 from hachikuji/KAFKA4390
Collecting socket server metrics during shutdown may throw NullPointerException
Author: Xavier Léauté <xavier@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2221 from xvrl/fix-metrics-npe-on-shutdown
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2170 from lindong28/KAFAK-4445
Fix possible integer overflow.
Author: Kim Christensen <kich@mvno.dk>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2200 from kichristensen/MiscalculatedOffsetRetention
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma, Jun Rao, Jiangjie Qin, Guozhang Wang
Closes#2195 from hachikuji/KAFKA-3994-linked-queue
NPE was caused by `log.logSegments.toArray` resulting in array containing `null` values. The exact reason still remains somewhat a mystery to me, but it seems that the culprit is `JavaConverters` in combination with concurrent data structure access.
Here's a simple code example to prove that:
```scala
import java.util.concurrent.ConcurrentSkipListMap
// Same as `JavaConversions`, but allows explicit conversions via `asScala`/`asJava` methods.
import scala.collection.JavaConverters._
case object Value
val m = new ConcurrentSkipListMap[Int, Value.type]
new Thread { override def run() = { while (true) m.put(9000, Value) } }.start()
new Thread { override def run() = { while (true) m.remove(9000) } }.start()
new Thread { override def run() = { while (true) { println(m.values.asScala.toArray.headOption) } } }.start()
```
Running the example will occasionally print `Some(null)` indicating that there's something shady going on during `toArray` conversion.
`null`s magically disappear by making the following change:
```diff
- println(m.values.asScala.toArray.headOption)
+ println(m.values.asScala.toSeq.headOption)
```
Author: Anton Karamanov <ataraxer@yandex-team.ru>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#2204 from ataraxer/KAFKA-4205
Jiangjie Qin
Ismael Juma
Rajini Sivaram
Shikhar Bhushan
Lihua Xin
Edoardo Comar
Eno Thereska
Colin P. Mccabe
Mickael Maison
Ben Stopford
Magnus Reftel
steve
Vahid Hashemian
Andrew Olson
Balint Molnar
Himani Arora
huxi
Edward Ribeiro
Jeff Widman
Dustin Cote
Manikumar Reddy O
Dong Lin
Jason Gustafson
Rekha Joshi
Json Tu
Ashish Singh
Michael Schiff
Xavier Léauté
Kim Christensen
Anton Karamanov
