Removed ignore annotations from the upgrade tests. This PR includes the following changes for updating the upgrade tests:
* Uploaded new versions 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, and 2.0.0 (in the associated scala versions) to kafka-packages
* Update versions in version.py, Dockerfile, base.sh
* Added new versions to StreamsUpgradeTest.test_upgrade_downgrade_brokers including version 2.0.0
* Added new versions StreamsUpgradeTest.test_simple_upgrade_downgrade test excluding version 2.0.0
* Version 2.0.0 is excluded from the streams upgrade/downgrade test as StreamsConfig needs an update for the new version, requiring a KIP. Once the community votes the KIP in, a minor follow-up PR can be pushed to add the 2.0.0 version to the upgrade test.
* Fixed minor bug in kafka-run-class.sh for classpath in upgrade/downgrade tests across versions.
* Follow on PRs for 0.10.2x, 0.11.0x, 1.0.x, 1.1.x, and 2.0.x will be pushed soon with the same updates required for the specific version.
Reviewers: Eno Thereska <eno.thereska@gmail.com>, John Roesler <vvcephei@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
- Removed Scala consumers (`SimpleConsumer` and `ZooKeeperConsumerConnector`)
and their tests.
- Removed Scala request/response/message classes.
- Removed any mention of new consumer or new producer in the code
with the exception of MirrorMaker where the new.consumer option was
never deprecated so we have to keep it for now. The non-code
documentation has not been updated either, that will be done
separately.
- Removed a number of tools that only made sense in the context
of the Scala consumers (see upgrade notes).
- Updated some tools that worked with both Scala and Java consumers
so that they only support the latter (see upgrade notes).
- Removed `BaseConsumer` and related classes apart from `BaseRecord`
which is used in `MirrorMakerMessageHandler`. The latter is a pluggable
interface so effectively public API.
- Removed `ZkUtils` methods that were only used by the old consumers.
- Removed `ZkUtils.registerBroker` and `ZKCheckedEphemeral` since
the broker now uses the methods in `KafkaZkClient` and no-one else
should be using that method.
- Updated system tests so that they don't use the Scala consumers except
for multi-version tests.
- Updated LogDirFailureTest so that the consumer offsets topic would
continue to be available after all the failures. This was necessary for it
to work with the Java consumer.
- Some multi-version system tests had not been updated to include
recently released Kafka versions, fixed it.
- Updated findBugs and checkstyle configs not to refer to deleted
classes and packages.
Reviewers: Dong Lin <lindong28@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Fix ServiceLoader issue with PluginClassLoader and add basic-auth-extension packaging & classpath
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
Author: Magesh Nandakumar <magesh.n.kumar@gmail.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5135 from mageshn/KAFKA-6991
We need to match to the end of the line to make it work with Java 10 as explained in the expanded comment.
Tested manually for all supported versions:
```shell
echo $(.../jdk1.8.0_152.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
1
echo $(.../jdk-9.0.4.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
9
echo $(.../jdk-10.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10
echo $(.../jdk-10.0.1.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10
```
To build jar you need to specify `scalaVersion` instead of `scala_version`.
Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
- Add capability to create delegation token
- Add authentication based on delegation token.
- Add capability to renew/expire delegation tokens.
- Add units tests and integration tests
Author: Manikumar Reddy <manikumar.reddy@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3616 from omkreddy/KAFKA-4541
Different fix for problem addressed by https://github.com/apache/kafka/pull/1953. Should prevent the CLASSPATH environment variable from being prefixed by a single colon before the JVM is invoked in the run-class script, which will then prevent the current working directory from being unintentionally included in the classpath when using the Reflections library.
If the current working directory should still be included in the classpath, it just needs to be explicitly specified either with its fully-qualified pathname or as a single dot (".").
Author: Chris Egerton <chrise@confluent.io>
Reviewers: Randall Hauch <rhauch@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#4406 from C0urante/fix-colon-prefixed-classpath
(WIP: this commit isn't ready to be reviewed yet. I was checking the travis-ci build with the configuration changes in my account and opened the PR prematurely against trunk. I will make it consistent with Contribution guidelines once it's well tested.)
https://issues.apache.org/jira/browse/KAFKA-5473
Design:
`zookeeper.connection.retry.timeout.ms` => this determines how long to wait before triggering the shutdown. The default is 60000ms.
Currently the implementation only handles the `handleSessionEstablishmentError` by waiting for the sessionTimeout.
Author: Prasanna Gautam <prasannagautam@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#3990 from prasincs/KAFKA-5473
Missed a few places in the previous commit.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4200 from ijuma/scala-2.11.12-part2
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3036 from mimaison/KAFKA-3356
When we install kafka on path with spaces, batch files were failing, this PR is trying to fix this issue.
Author: Vladimír Kleštinec <klestinec@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#2649 from klesta490/trunk
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Colin P. Mccabe <cmccabe@confluent.io>
Closes#3621 from lindong28/KAFKA-5694
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3699 from cmccabe/trogdor-review
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.
It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
Added kafka-broker-api-versions.bat for Windows platform
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian, Ismael Juma, Guozhang Wang
Closes#2943 from amethystic/kafka-5143_offer_broker_api_versions_bat
Add a default log retention policy to keep GC logs from growing too large
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1431 from rnpridgeon/KAFKA-3754
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2476 from lindong28/KAFKA-4586
handle existing classpath with spaces
list each lib dependency individually
Author: Jon Freedman <jon.freedman@zoho.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2433 from jonfreedman/trunk
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2184 from cmccabe/KAFKA-4457
Besides API and runtime changes, this PR also includes 2 data transformations (`InsertField`, `HoistToStruct`) and 1 routing transformation (`TimestampRouter`).
There is some gnarliness in `ConnectorConfig` / `ConfigDef` around creating, parsing and validating a dynamic `ConfigDef`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2299 from shikhar/smt-2017
Some of the recent changes to `kafka-run-class.sh` have not been applied to `kafka-run-class.bat`.
These recent changes include setting proper streams or connect classpaths. So any streams or connect use case that leverages `kafka-run-class.bat` would fail with an error like
```
Error: Could not find or load main class org.apache.kafka.streams.???
```
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2238 from vahidhashemian/minor/sync_up_kafka-run-class.bat
Without this fix the new consumer fails to run on a 32-bit Windows OS.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#2189 from vahidhashemian/KAFKA-4271
Will allow users to set one fewer environment variable if they need to
change scala version. Still, SCALA_BINARY_VERSION can be explicitly set.
Author: Konstantine Karantasis <konstantine@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2130 from kkonstantine/MINOR-Extract-SCALA_BINARY_VERSION-from-SCALA_VERSION
add file 'kafka-consumer-groups.bat' for Windows platform
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#2037 from amethystic/kafka-4310_add_consumer_groups_script
## Problem
Current connect scripts (`connect-distributed.sh`, `connect-standalone.sh`) do not support `-daemon` flag even if users specify the flag
since `kafka-run-class.sh` requires that the`-daemon` flag should precede other arguments (e.g. class name)
## Solution
Do the same thing like in `kafka-server-start.sh`
- Parse a command
- Add `-daemon` to `$EXTRA_ARGS` if exists
Author: 1ambda <1amb4a@gmail.com>
Reviewers: Gwen Shapira
Closes#1717 from 1ambda/KAFKA-3742-connect-running-as-daemon
On Windows, the following output is seen when starting Zookeeper and Kafka servers:
```
'#' is not recognized as an internal or external command,
operable program or batch file.
```
This pull request makes a minor correction to the Windows `kafka-run-class.bat` script to replace the use of `#` with `rem`.
Author: P. Thorpe <paul.thorpe@uk.ibm.com>
Reviewers: Vahid Hashemian, Gwen Shapira
Closes#1740 from p-thorpe/trunk
kafka-run-class.sh now runs under Cygwin; paths and classpath are set up properly.
**WARNING:** The script was not tested on a Linux machine, only under Cygwin. Prior to merge it into trunk, if accepted, please run a quick test to ensure nothing broke. From my own code review, there should not be any problem, but we can never be too sure.
I do not have the environment to test it under Linux at this moment.
Author: deragon <32nx9812masakjds>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1487 from deragon/trunk
moved streams application reset tool from tools to core
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1685 from mjsax/moveResetTool
(cherry picked from commit f2405a73ea2dd4b636832b7f8729fb06a04de1d5)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Geoff Anderson <geoff@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1364 from Ishiihara/add-quote-classpath
bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2.
Author: Dana Powers <dana.powers@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1302 from dpkp/KAFKA-3615-fix
granders hachikuji Can you take a look when you have time? Appreciate your time to review.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1263 from Ishiihara/classpath-no-test-jar
ewencp granders Can you take a look? Thanks!
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1259 from Ishiihara/fix-warning
I believe this addresses KAFKA-3384.
The POSIX kill manpage is at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html
Author: Matt McClure <mlm@aya.yale.edu>
Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1148 from matthewlmcclure/KAFKA-3384
Stop scritps such as kafka-server-stop.sh log messages of kill command's error when processes aren't running.
This PR changes this message to "No kafka server to stop".
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Gwen Shapira
Closes#971 from sasakitoa/stop_scripts_says_not_good_message
Bill Bejeck
Ismael Juma
Magesh Nandakumar
Manikumar Reddy O
Sasaki Toru
Matthias J. Sax
hmcl
Jimin Hsieh
Jason Gustafson
Chris Egerton
Prasanna Gautam
Arjun Satish
Vahid Hashemian
Mickael Maison
Vladimír Kleštinec
Dong Lin
Colin P. Mccabe
Xavier Léauté
Evgeny Veretennikov
amethystic
Ryan P
Jon Freedman
Xi Hu
Shikhar Bhushan
Konstantine Karantasis
1ambda
P. Thorpe
Grant Henke
deragon
Matthias J. Sax
Liquan Pei
Dana Powers
Matt McClure
Yasuhiro Matsuda
Sasaki Toru