tchekai704
/
src-kafka
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

MINOR: Provide valid examples in README page. (#10259) 

* MINOR: Provide valid examples in README page.

- `testMetadataUpdateWaitTime` method is removed from MetadataTest class.
-  Removed the travis CI documentation.

Reviewers: Luke Chen <showuon@gmail.com>
pull/10239/merge
Kamal Chandraprakash 3 years ago committed by GitHub
parent
06ca4850c5
commit
496aa1f84b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 1 additions and 304 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 2
      README.md
  2. 303
      tests/README.md

2
README.md
Unescape View File

@ -46,7 +46,7 @@ Follow instructions in https://kafka.apache.org/quickstart @@ -46,7 +46,7 @@ Follow instructions in https://kafka.apache.org/quickstart
### Running a particular test method within a unit/integration test ###
./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testTimeToNextUpdate
### Running a particular unit/integration test with log4j output ###
Change the log4j setting in either `clients/src/test/resources/log4j.properties` or `core/src/test/resources/log4j.properties`

303
tests/README.md
Unescape View File

@ -127,309 +127,6 @@ REBUILD="t" bash tests/docker/run_tests.sh @@ -127,309 +127,6 @@ REBUILD="t" bash tests/docker/run_tests.sh
will attach it to the TopicCommand process running in the docker image.
Examining CI run
----------------
* Set BUILD_ID is travis ci's build id. E.g. build id is 169519874 for the following build
```bash
https://travis-ci.org/apache/kafka/builds/169519874
```
* Getting number of tests that were actually run
```bash
for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | grep -cE 'RunnerClient: Loading test'
```
* Getting number of tests that passed
```bash
for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | grep -cE 'RunnerClient.*PASS'
```
* Getting all the logs produced from a run
```bash
for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done
```
* Explanation of curl calls to travis-ci & jq commands
- We get json information of the build using the following command
```bash
curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874
```
This produces a json about the build which looks like:
```json
{
"id": 169519874,
"repository_id": 6097916,
"number": "19",
"config": {
"sudo": "required",
"dist": "trusty",
"language": "java",
"env": [
"TC_PATHS=\"tests/kafkatest/tests/client\"",
"TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"",
"TC_PATHS=\"tests/kafkatest/tests/mirror_maker\"",
"TC_PATHS=\"tests/kafkatest/tests/replication\"",
"TC_PATHS=\"tests/kafkatest/tests/upgrade\"",
"TC_PATHS=\"tests/kafkatest/tests/security\"",
"TC_PATHS=\"tests/kafkatest/tests/core\""
],
"jdk": [
"oraclejdk8"
],
"before_install": null,
"script": [
"./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh"
],
"services": [
"docker"
],
"before_cache": [
"rm -f $HOME/.gradle/caches/modules-2/modules-2.lock",
"rm -fr $HOME/.gradle/caches/*/plugin-resolution/"
],
"cache": {
"directories": [
"$HOME/.m2/repository",
"$HOME/.gradle/caches/",
"$HOME/.gradle/wrapper/"
]
},
".result": "configured",
"group": "stable"
},
"state": "finished",
"result": null,
"status": null,
"started_at": "2016-10-21T13:35:43Z",
"finished_at": "2016-10-21T14:46:03Z",
"duration": 16514,
"commit": "7e583d9ea08c70dbbe35a3adde72ed203a797f64",
"branch": "trunk",
"message": "respect _DUCK_OPTIONS",
"committed_at": "2016-10-21T00:12:36Z",
"author_name": "Raghav Kumar Gautam",
"author_email": "raghav@apache.org",
"committer_name": "Raghav Kumar Gautam",
"committer_email": "raghav@apache.org",
"compare_url": "https://github.com/raghavgautam/kafka/compare/cc788ac99ca7...7e583d9ea08c",
"event_type": "push",
"matrix": [
{
"id": 169519875,
"repository_id": 6097916,
"number": "19.1",
"config": {
"sudo": "required",
"dist": "trusty",
"language": "java",
"env": "TC_PATHS=\"tests/kafkatest/tests/client\"",
"jdk": "oraclejdk8",
"before_install": null,
"script": [
"./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh"
],
"services": [
"docker"
],
"before_cache": [
"rm -f $HOME/.gradle/caches/modules-2/modules-2.lock",
"rm -fr $HOME/.gradle/caches/*/plugin-resolution/"
],
"cache": {
"directories": [
"$HOME/.m2/repository",
"$HOME/.gradle/caches/",
"$HOME/.gradle/wrapper/"
]
},
".result": "configured",
"group": "stable",
"os": "linux"
},
"result": null,
"started_at": "2016-10-21T13:35:43Z",
"finished_at": "2016-10-21T14:24:50Z",
"allow_failure": false
},
{
"id": 169519876,
"repository_id": 6097916,
"number": "19.2",
"config": {
"sudo": "required",
"dist": "trusty",
"language": "java",
"env": "TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"",
"jdk": "oraclejdk8",
"before_install": null,
"script": [
"./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh"
],
"services": [
"docker"
],
"before_cache": [
"rm -f $HOME/.gradle/caches/modules-2/modules-2.lock",
"rm -fr $HOME/.gradle/caches/*/plugin-resolution/"
],
"cache": {
"directories": [
"$HOME/.m2/repository",
"$HOME/.gradle/caches/",
"$HOME/.gradle/wrapper/"
]
},
".result": "configured",
"group": "stable",
"os": "linux"
},
"result": 1,
"started_at": "2016-10-21T13:35:46Z",
"finished_at": "2016-10-21T14:22:05Z",
"allow_failure": false
},
...
]
}
```
- By passing this through jq filter `.matrix` we extract the matrix part of the json
```bash
curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix'
```
The resulting json looks like:
```json
[
{
"id": 169519875,
"repository_id": 6097916,
"number": "19.1",
"config": {
"sudo": "required",
"dist": "trusty",
"language": "java",
"env": "TC_PATHS=\"tests/kafkatest/tests/client\"",
"jdk": "oraclejdk8",
"before_install": null,
"script": [
"./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh"
],
"services": [
"docker"
],
"before_cache": [
"rm -f $HOME/.gradle/caches/modules-2/modules-2.lock",
"rm -fr $HOME/.gradle/caches/*/plugin-resolution/"
],
"cache": {
"directories": [
"$HOME/.m2/repository",
"$HOME/.gradle/caches/",
"$HOME/.gradle/wrapper/"
]
},
".result": "configured",
"group": "stable",
"os": "linux"
},
"result": null,
"started_at": "2016-10-21T13:35:43Z",
"finished_at": "2016-10-21T14:24:50Z",
"allow_failure": false
},
{
"id": 169519876,
"repository_id": 6097916,
"number": "19.2",
"config": {
"sudo": "required",
"dist": "trusty",
"language": "java",
"env": "TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"",
"jdk": "oraclejdk8",
"before_install": null,
"script": [
"./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh"
],
"services": [
"docker"
],
"before_cache": [
"rm -f $HOME/.gradle/caches/modules-2/modules-2.lock",
"rm -fr $HOME/.gradle/caches/*/plugin-resolution/"
],
"cache": {
"directories": [
"$HOME/.m2/repository",
"$HOME/.gradle/caches/",
"$HOME/.gradle/wrapper/"
]
},
".result": "configured",
"group": "stable",
"os": "linux"
},
"result": 1,
"started_at": "2016-10-21T13:35:46Z",
"finished_at": "2016-10-21T14:22:05Z",
"allow_failure": false
},
...
]
```
- By further passing this through jq filter `map(.id)` we extract the id of
the builds for each of the splits
```bash
curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix|map(.id)'
```
The resulting json looks like:
```json
[
169519875,
169519876,
169519877,
169519878,
169519879,
169519880,
169519881
]
```
- To use these ids in for loop we want to get rid of `[]` which is done by
passing it through `.[]` filter
```bash
curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix|map(.id)|.[]'
```
And we get
```text
169519875
169519876
169519877
169519878
169519879
169519880
169519881
```
- In the for loop we have made calls to fetch logs
```bash
curl -sSL "https://api.travis-ci.org/jobs/169519875/log.txt?deansi=true" | tail
```
which gives us
```text
[INFO:2016-10-21 14:21:12,538]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: test 16 of 28
[INFO:2016-10-21 14:21:12,538]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: setting up
[INFO:2016-10-21 14:21:30,810]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: running
[INFO:2016-10-21 14:24:35,519]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: PASS
[INFO:2016-10-21 14:24:35,519]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: tearing down
The job exceeded the maximum time limit for jobs, and has been terminated.
```
* Links
- [Travis-CI REST api documentation](https://docs.travis-ci.com/api)
- [jq Manual](https://stedolan.github.io/jq/manual/)
Local Quickstart
----------------
This quickstart will help you run the Kafka system tests on your local machine. Note this requires bringing up a cluster of virtual machines on your local computer, which is memory intensive; it currently requires around 10G RAM.

Write Preview
Loading…
Cancel
Save