Renames `HoistToStruct` SMT to `HoistField`.
Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`
Adds HTML doc generation and updates to `connect.html`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2374 from shikhar/more-smt
Besides API and runtime changes, this PR also includes 2 data transformations (`InsertField`, `HoistToStruct`) and 1 routing transformation (`TimestampRouter`).
There is some gnarliness in `ConnectorConfig` / `ConfigDef` around creating, parsing and validating a dynamic `ConfigDef`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2299 from shikhar/smt-2017
dguy guozhangwang This is a new PR for KAFKA-4060.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Author: Hojjat Jafarpour <hojjat@HojjatJpoursMBP.attlocal.net>
Reviewers: Damian Guy, Matthias J. Sax, Isamel Juma, Guozhang Wang
Closes#1884 from hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-new
There were a couple of important issues fixed in Gradle 3.2.1:
* [GRADLE-3582] - Gradle wrapper fails to escape arguments with nested quotes
* [GRADLE-3583] - Newlines in JAVA_OPTS breaks application plugin shell script in Gradle 3.2
And a lot of important issues fixed in Scala 2.12.1:
* http://www.scala-lang.org/news/2.12.1
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
Closes#2216 from ijuma/gradle-3.2.1-and-scala-2.12.1
We suspect that the test suite hangs we have been seeing are
due to PermGen exhaustion. It is a common reason for
hard JVM lock-ups.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1926 from ijuma/test-jvm-params
Also upgrade scoverage (required for compatibility) and remove usage of
`useAnt` which doesn't exist in Gradle 3.0
It turns out that one cannot even run `gradle` to download the project Gradle version if `useAnt` is used in the build. This is unfortunate (the SBT launcher has much saner behaviour).
Release notes: https://docs.gradle.org/3.0/release-notes
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Sriharsha Chintalapani <harsha@hortonworks.com>
Closes#1774 from ijuma/kafka-4082-support-gradle-3.0
ijuma said that it would make sense to split out this work from KAFKA-3234, since KAFKA-3234 had both a mechanical change (generating docs) as well as a change requiring discussion (deprecating/renaming config options).
jjkoshy, I hope you don't mind that I took over this work. It's been 3 months since the last activity on KAFKA-3234, so I thought it would be okay to take over.
This work is essentially is the first 5-6 commits from Joel's https://github.com/apache/kafka/pull/907. However, since I'm not very experienced with git, I didn't do a direct merge/rebase, but instead largely hand-merged it. I did some minor cleanup. All credit goes to Joel, all blame goes to me. :)
For reference, I attached the auto-generated configuration.html file (as a PDF, because github won't let me attache html).
[configuration.pdf](https://github.com/apache/kafka/files/323901/configuration.pdf)
This is my first time writing Scala, so let me know if there are any changes needed.
I don't know who is the right person to review this. ijuma, can you help me redirect this to the appropriate person? Thanks.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1527 from wushujames/generate_topic_docs
This is a regression caused by 0bb1d3ae.
After that commit, Streams no longer has a direct dependency on slf4j-log4j12, but zkclient
has a dependency on an older version of slf4j-log4j12, so we get a transitive dependency on
the older version.
The fix is to simply exclude the undesired dependencies from the zkclient dependency.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1704 from ijuma/kafka-4018-streams-duplicate-slf4j-log4j
moved streams application reset tool from tools to core
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1685 from mjsax/moveResetTool
(cherry picked from commit f2405a73ea)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Better performance is always welcome:
"The Gradle build itself has seen a 50% reduction in configuration time. You'll see the biggest impact on multi-project builds"
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1644 from ijuma/update-gradle
As kafka-streams is intended to be used by applications that may or may not wish to use log4j, kafka-streams itself should not have a dependency on a concrete log framework. This change adapts the dependencies to be API-only for compile, and framework-specific for the test runtime only.
I read through the [Contributing Code Guidelines](https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes) and interpreted this as a trivial change that doesn't require a Jira ticket. Please let me know if I've interpreted that wrongly.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Mathieu Fenniak <mathieu@encouragemarketing.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1639 from mfenniak/fix-slf4j-dependency-for-streams
Fix timing window in producer by holding onto cluster object while processing send requests so that changes to cluster during metadata refresh don't cause NPE if a topic is deleted.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1478 from rajinisivaram/KAFKA-3562
Currently javadoc doesn't specify charset.
This pull reqeust will set this to UTF-8.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1413 from sasakitoa/javadoc_garbled
The task is called `aggregatedJavadoc` and the generated html will be under `<project.dir>/build/docs/javadoc/`.
I also disabled javadoc for `tools` and `log4j-appender` as they are not public API.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1398 from ijuma/kafka-3717-aggregate-javadoc
These dependencies are unnecessary and they are acquired
transitively via zkclient (jline, netty) and reflections (annotations).
Ewen did the hard work in figuring out why we have unexpected
additional dependencies since 0.9.x.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava, Guozhang Wang, Gwen Shapira
Closes#1396 from ijuma/exclude-jline-netty-deps-in-streams and squashes the following commits:
3aa366f [Ismael Juma] Exclude findbugs annotations due to LGPL license
2d3d714 [Ismael Juma] Use local exclusion for `jline` and `netty`
482b6c0 [Ismael Juma] Exclude `jline` and `netty` dependencies in the `streams` project
_copyDependantTestLibs_ was added temporarily as a dependency of _jar_ task to enable SASL system tests to be run with MiniKdc without changing the automated system test runs which run _gradlew clean jar_. Since the build target _systemTestLibs_ is already in Kafka build.gradle, the Confluent automated test runs can now run _gradlew clean systemTestLibs_ instead. This PR provides the final change to remove _copyDependantTestLibs_ from the _jar_ task. This should be committed only after the Confluent automated sytem test build script is updated, to avoid breaking any builds.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#430 from rajinisivaram/minor-systemtestlibs
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma, Damian Guy, Michael G. Noll, Guozhang Wang
Closes#1260 from enothereska/KAFKA-3612-integration-tests
There are a few improvements in 2.12 and 2.13. I am particularly interested in the performance improvements:
* 2.12: "This release brings support for compile only dependencies, improved build script compilation speed and even better IDE support."
* 2.13: "We've achieved performance improvements during Gradle's configuration and execution phase, where we have measured up to 25% improvements to build time in our performance tests. No changes to your build script are necessary to start taking advantage of these improvements."
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1271 from ijuma/gradle-2.13
This also fixes KAFKA-3453 and KAFKA-2866.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1155 from ijuma/kafka-3475-introduce-our-minikdc
…ibraries
This ensures duplicates are not copied in the distribution without rewriting all of the tar'ing logic. A larger improvement could be made to the packaging code, but that should be tracked by another jira.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira, Ismael Juma
Closes#1075 from granthenke/libs-duplicates
* Fix and suppress number of unchecked warnings (except for Kafka Streams)
* Add `SafeVarargs` annotation to fix warnings
* Suppress unfixable deprecation warnings
* Replace deprecated by non-deprecated usage where possible
* Avoid reflective calls via structural types in Scala
* Tweak compiler settings for scalac and javac
Once we drop Java 7 and Scala 2.10, we can tweak the compiler settings further so that they warn us about more things.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke, Gwen Shapira, Guozhang Wang
Closes#1042 from ijuma/kafka-3375-suppress-depreccated-tweak-compiler
- Moves all generated docs under /docs/generated
- Generates docs for Protocol, Errors, and ApiKeys
- Adds new protocol.html page
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#970 from granthenke/protocol-doc-wip
Adds a gradle task to generate a report of outdate release dependencies:
`gradle dependencyUpdates`
Updates a few minor versions.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma, Gwen Shapira
Closes#973 from granthenke/outdated-deps
Without this change `./gradlew releaseTarGz` (and its variants) will not include the RocksDB jar, which is required for Kafka Streams, in Kafka's `libs/` folder. The impact is that any Streams job will fail when it runs against a broker that was installed via a release tarball.
guozhangwang junrao : please review.
Author: Michael G. Noll <michael@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1007 from miguno/trunk-rocksdb-fixes
Also remove some unused imports.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#992 from guozhangwang/KSExamples
Observation: when doing "gradlew releaseTarGz" the streams jar was not included in the tarball. Adding a line to include it. ijuma guozhangwang could you please review. Thanks.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#984 from enothereska/trunk
Patch version bumps for bouncy castle, minikdc, snappy, slf4j, scalatest and powermock. Notable fixes:
* Snappy: fixes a resource leak
* Bouncy castle: security fixes
Also update Gradle to 2.11 (where the notable change is improved IDE integration) and the grgit build dependency.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#903 from ijuma/kafka-3227-conservative-update-of-kafka-deps
This is handy when debugging certain kinds of Jenkins failures.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#739 from ijuma/gradle-show-standard-streams
guozhangwang added .git/refs/heads/ file existence check.
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#209 from omkreddy/KAFKA-1901
Ismael Juma
Shikhar Bhushan
Hojjat Jafarpour
Ben Stopford
Raghav Kumar Gautam
Bernard Leach
Guozhang Wang
Ewen Cheslack-Postava
James Cheng
Matthias J. Sax
Mathieu Fenniak
Rajini Sivaram
Sasaki Toru
Eno Thereska
Grant Henke
Liquan Pei
Michael G. Noll
Yasuhiro Matsuda
Manikumar reddy O
Gwen Shapira
