The following files are available in https://s3-us-west-2.amazonaws.com/kafka-packages/:
kafka-streams-3.3.1-test.jar
kafka_2.12-3.3.1.tgz
kafka_2.13-3.3.1.tgz
Reviewers: Colin P. McCabe <cmccabe@apache.org>
Update the ZooKeeper version to v3.6.3. This requires adding dropwizard
as a new dependency.
Also, add Kafka v2.8.0 to the ducktape system test image.
Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>
Two tests using 50k replicas on 8 brokers:
* Do a rolling restart with clean shutdown, delete topics
* Run produce bench and consumer bench on a subset of topics
Reviewed-By: David Jacot <djacot@confluent.io>, Vikas Singh <vikas@confluent.io>, Jason Gustafson <jason@confluent.io>
* Add rate limiting to tc
* Feedback from PR
* Add a sanity test for tc
* Add iperf to vagrant scripts
* Dynamically determine the network interface
* Add some temp code for testing on AWS
* Temp: use hostname instead of external IP
* Temp: more AWS debugging
* More AWS WIP
* More AWS temp
* Lower latency some
* AWS wip
* Trying this again now that ping should work
* Add cluster decorator to tests
* Fix broken import
* Fix device name
* Fix decorator arg
* Remove errant import
* Increase timeouts
* Fix tbf command, relax assertion on latency test
* Fix log line
* Final bit of cleanup
* Newline
* Revert Trogdor retry count
* PR feedback
* More PR feedback
* Feedback from PR
* Remove unused argument
When Trogdor wants to clear all the faults injected to Kibosh, it sends the empty JSON object {}. However, Kibosh expects {"faults":[]} instead. Kibosh should handle the empty JSON object, since that's consistent with how Trogdor handles empty JSON fields in general (if they're empty, they can be omitted). We should also have a test for this.
Reviewers: David Arthur <mumrah@gmail.com>, Stanislav Kozlovski <stanislav_kozlovski@outlook.com>
These are important to ensure we don't break compatibility.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#6794 from ijuma/update-version-compat-tests
The following command in vagrant/README.md doesn't work,
since `--zookeeper` option has been unsuppored from v2.0.0.
This PR updates its command options to fix it.
```
bin/kafka-console-consumer.sh --zookeeper zk1:2181 --topic sandbox --from-beginning
```
Reviewers: Jason Gustafson <jason@confluent.io>
Contrary to the previous explanation, a command example in
vagrant/README.md lacks the option to specify the aws provider.
Author: Kengo Seki <sekikn@apache.org>
Reviewers: Gwen Shapira
Closes#6702 from sekikn/add-missing-option
For now, `vagrant/vagrant-up.sh --aws` fails because
the `vagrant hostmanager` command in that script lacks
the `--aws` option. This PR adds it.
I ran `vagrant/vagrant-up.sh --aws` with and without
`--no-parallel` option and confirmed both worked
as expected.
Author: Kengo Seki <sekikn@apache.org>
Reviewers: Gwen Shapira
Closes#6703 from sekikn/KAFKA-8344
Verified that the https links work.
I didn't update the license header in this PR since that touches
so many files. Will file a separate one for that.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Upgrade from 171 to 202. Unpack and install directly from a cached tgz rather than going via the installer deb from webupd8. The installer is still on 8u919 while we want 202.
Testing via kafka branch builder job
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2305/
Author: Jarek Rudzinski <jarek@confluent.io>
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Alex Diachenko <sansanichfb@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#6165 from jarekr/trunk-jdk8-from-tgz
The StreamsUpgradeTest::test_upgrade_downgrade_brokers used sleep calls in the test which led to flaky test performance and as a result, we placed an @ignore annotation on the test. This PR uses log events instead of the sleep calls hence we can now remove the @ignore setting.
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Pricing for m3.xlarge: On-Demand is at $0.266. Reserved is at about $0.16 (40% discount). And Spot is at $0.0627 (76% discount relative to On-Demand, or 60% discount relative to Reserved). Insignificant fluctuation in the past 3 months.
Ran on branch builder and works as expected -- each worker is created using spot instances (https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1982/console)
This can be safely backported to 0.10.2 (tested using https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1983/)
Author: Max Zheng <maxzheng.os@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#5707 from maxzheng/minor-switch@trunk
Removed ignore annotations from the upgrade tests. This PR includes the following changes for updating the upgrade tests:
* Uploaded new versions 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, and 2.0.0 (in the associated scala versions) to kafka-packages
* Update versions in version.py, Dockerfile, base.sh
* Added new versions to StreamsUpgradeTest.test_upgrade_downgrade_brokers including version 2.0.0
* Added new versions StreamsUpgradeTest.test_simple_upgrade_downgrade test excluding version 2.0.0
* Version 2.0.0 is excluded from the streams upgrade/downgrade test as StreamsConfig needs an update for the new version, requiring a KIP. Once the community votes the KIP in, a minor follow-up PR can be pushed to add the 2.0.0 version to the upgrade test.
* Fixed minor bug in kafka-run-class.sh for classpath in upgrade/downgrade tests across versions.
* Follow on PRs for 0.10.2x, 0.11.0x, 1.0.x, 1.1.x, and 2.0.x will be pushed soon with the same updates required for the specific version.
Reviewers: Eno Thereska <eno.thereska@gmail.com>, John Roesler <vvcephei@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
* Set --source, --target and --release to 1.8.
* Build Scala 2.12 by default.
* Remove some conditionals in the build file now that Java 8
is the minimum version.
* Bump the version of Jetty, Jersey and Checkstyle (the newer
versions require Java 8).
* Fixed issues uncovered by the new version if Checkstyle.
* A couple of minor updates to handle an incompatible source
change in the new version of Jetty.
* Add dependency to jersey-hk2 to fix failing tests caused
by Jersey upgrade.
* Update release script to use Java 8 and to take into account
that Scala 2.12 is now built by default.
* While we're at it, bump the version of Gradle, Gradle plugins,
ScalaLogging, JMH and apache directory api.
* Minor documentation updates including the readme and upgrade
notes. A number of Streams Java 7 examples can be removed
subsequently.
Debian installer packages are no longer available for Java 7.
Also upgrade AMI to latest ubuntu/trusty 14 amd64 as the older
one is no longer available.
Note that this only changes the JDK used to build and run
the system tests. We still have Jenkins jobs that compile
and run the JUnit tests with Java 7 so that we don't use
features that are only available in newer Java versions.
Some changes required to get the Streams system tests working via Docker
To test:
TC_PATHS="tests/kafkatest/tests/streams" bash tests/docker/run_tests.sh
That command will take about 3.5 hours, and should pass. Note there are a couple of ignored tests.
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Bill Bejeck <bill@confluent.io>
Fix an omission where Kibosh was not getting installed on Vagrant
instances running in AWS.
Fix an issue where the Dockerfile was unable to download old Apache
Kafka releases. See the discussion on KAFKA-6233.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#4240 from cmccabe/KAFKA-6247
Author: Ewen Cheslack-Postava <ewen@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Eno Thereska <eno.thereska@gmail.com>
Closes#3546 from ewencp/fix-rsync
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>
Closes#3454 from ijuma/test-upgrades-from-0.11.0.x
-Tests for rolling upgrades for a streams app (keeping broker config fixed)
-Tests for rolling upgrades of brokers (keeping streams app config fixed)
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3411 from enothereska/KAFKA-5487-upgrade-test-streams
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Joseph Rea <jrea@users.noreply.github.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2575 from mjsax/minor-update-system-test-readme
See https://github.com/mitchellh/vagrant/issues/7531. The core of the issue is that vagrant rsync uses a fixed set of 1000 possible temp file entries for SSH ControlMaster files to cache SSH connections for rsyncing. A few notes:
* We can't break down the steps further and maintain performance due to various limitations in vagrant/vagrant-aws (rsync is only executed on `vagrant up`/`vagrant reload`/`vagrant rsync`, you can't enable/disable and rsync shared folder only during some of those stages, and provisioning only runs in parallel with vagrant-aws during `vagrant up`).
* We need to isolate each of the serialized rsync calls. (If we assumed `parallel` was available, we actually could get the parallelism back.) This is required because even across calls they could randomly choose the same temporary file.
* If there's a chance multiple instances were running on the same server at the same or nearly the same time, they can conflict since the same temp file entries are used globally. This means anything running on shared CI servers might end up syncing data between different CI jobs (!!), which could lead to some very strange results. Especially weird if they aren't even for the same type of job.
* Provisioning error check needs to be removed because it is catching rsync errors, but those can still happen in the initial `vagrant up` rsync step before the `vagrant up` provisioning step. It seems likely this bug was the cause of missing files anyway so this check might not be as valuable anymore.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3380 from ewencp/deparallelize-rsync
Satish Duggana
Divij Vaidya
Mickael Maison
José Armando García Sancio
Manikumar Reddy
Tom Bentley
Bruno Cadonna
David Jacot
Ron Dagostino
A. Sophie Blee-Goldman
Bill Bejeck
Luke Chen
John Roesler
Randall Hauch
Boyang Chen
David Arthur
Colin Patrick McCabe
Matthias J. Sax
Ismael Juma
Kengo Seki
Jarek Rudzinski
Max Zheng
Matthias J. Sax
Jarek Rudzinski
Colin P. Mccabe
Ewen Cheslack-Postava
Eno Thereska
Ewen Cheslack-Postava