From 496aa1f84b0e75ed1c89559e41d75f69fe1ac71f Mon Sep 17 00:00:00 2001 From: Kamal Chandraprakash Date: Mon, 21 Feb 2022 12:18:24 +0530 Subject: [PATCH] MINOR: Provide valid examples in README page. (#10259) * MINOR: Provide valid examples in README page. - `testMetadataUpdateWaitTime` method is removed from MetadataTest class. - Removed the travis CI documentation. Reviewers: Luke Chen --- README.md | 2 +- tests/README.md | 303 ------------------------------------------------ 2 files changed, 1 insertion(+), 304 deletions(-) diff --git a/README.md b/README.md index 1f35df48a67..5e409f8dada 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Follow instructions in https://kafka.apache.org/quickstart ### Running a particular test method within a unit/integration test ### ./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic - ./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime + ./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testTimeToNextUpdate ### Running a particular unit/integration test with log4j output ### Change the log4j setting in either `clients/src/test/resources/log4j.properties` or `core/src/test/resources/log4j.properties` diff --git a/tests/README.md b/tests/README.md index e54c60e9f72..78b10a701dc 100644 --- a/tests/README.md +++ b/tests/README.md @@ -127,309 +127,6 @@ REBUILD="t" bash tests/docker/run_tests.sh will attach it to the TopicCommand process running in the docker image. -Examining CI run ----------------- -* Set BUILD_ID is travis ci's build id. E.g. build id is 169519874 for the following build -```bash -https://travis-ci.org/apache/kafka/builds/169519874 -``` - -* Getting number of tests that were actually run -```bash -for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | grep -cE 'RunnerClient: Loading test' -``` - -* Getting number of tests that passed -```bash -for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done | grep -cE 'RunnerClient.*PASS' -``` -* Getting all the logs produced from a run -```bash -for id in $(curl -sSL https://api.travis-ci.org/builds/$BUILD_ID | jq '.matrix|map(.id)|.[]'); do curl -sSL "https://api.travis-ci.org/jobs/$id/log.txt?deansi=true" ; done -``` -* Explanation of curl calls to travis-ci & jq commands - - We get json information of the build using the following command -```bash -curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 -``` -This produces a json about the build which looks like: -```json -{ - "id": 169519874, - "repository_id": 6097916, - "number": "19", - "config": { - "sudo": "required", - "dist": "trusty", - "language": "java", - "env": [ - "TC_PATHS=\"tests/kafkatest/tests/client\"", - "TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"", - "TC_PATHS=\"tests/kafkatest/tests/mirror_maker\"", - "TC_PATHS=\"tests/kafkatest/tests/replication\"", - "TC_PATHS=\"tests/kafkatest/tests/upgrade\"", - "TC_PATHS=\"tests/kafkatest/tests/security\"", - "TC_PATHS=\"tests/kafkatest/tests/core\"" - ], - "jdk": [ - "oraclejdk8" - ], - "before_install": null, - "script": [ - "./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh" - ], - "services": [ - "docker" - ], - "before_cache": [ - "rm -f $HOME/.gradle/caches/modules-2/modules-2.lock", - "rm -fr $HOME/.gradle/caches/*/plugin-resolution/" - ], - "cache": { - "directories": [ - "$HOME/.m2/repository", - "$HOME/.gradle/caches/", - "$HOME/.gradle/wrapper/" - ] - }, - ".result": "configured", - "group": "stable" - }, - "state": "finished", - "result": null, - "status": null, - "started_at": "2016-10-21T13:35:43Z", - "finished_at": "2016-10-21T14:46:03Z", - "duration": 16514, - "commit": "7e583d9ea08c70dbbe35a3adde72ed203a797f64", - "branch": "trunk", - "message": "respect _DUCK_OPTIONS", - "committed_at": "2016-10-21T00:12:36Z", - "author_name": "Raghav Kumar Gautam", - "author_email": "raghav@apache.org", - "committer_name": "Raghav Kumar Gautam", - "committer_email": "raghav@apache.org", - "compare_url": "https://github.com/raghavgautam/kafka/compare/cc788ac99ca7...7e583d9ea08c", - "event_type": "push", - "matrix": [ - { - "id": 169519875, - "repository_id": 6097916, - "number": "19.1", - "config": { - "sudo": "required", - "dist": "trusty", - "language": "java", - "env": "TC_PATHS=\"tests/kafkatest/tests/client\"", - "jdk": "oraclejdk8", - "before_install": null, - "script": [ - "./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh" - ], - "services": [ - "docker" - ], - "before_cache": [ - "rm -f $HOME/.gradle/caches/modules-2/modules-2.lock", - "rm -fr $HOME/.gradle/caches/*/plugin-resolution/" - ], - "cache": { - "directories": [ - "$HOME/.m2/repository", - "$HOME/.gradle/caches/", - "$HOME/.gradle/wrapper/" - ] - }, - ".result": "configured", - "group": "stable", - "os": "linux" - }, - "result": null, - "started_at": "2016-10-21T13:35:43Z", - "finished_at": "2016-10-21T14:24:50Z", - "allow_failure": false - }, - { - "id": 169519876, - "repository_id": 6097916, - "number": "19.2", - "config": { - "sudo": "required", - "dist": "trusty", - "language": "java", - "env": "TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"", - "jdk": "oraclejdk8", - "before_install": null, - "script": [ - "./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh" - ], - "services": [ - "docker" - ], - "before_cache": [ - "rm -f $HOME/.gradle/caches/modules-2/modules-2.lock", - "rm -fr $HOME/.gradle/caches/*/plugin-resolution/" - ], - "cache": { - "directories": [ - "$HOME/.m2/repository", - "$HOME/.gradle/caches/", - "$HOME/.gradle/wrapper/" - ] - }, - ".result": "configured", - "group": "stable", - "os": "linux" - }, - "result": 1, - "started_at": "2016-10-21T13:35:46Z", - "finished_at": "2016-10-21T14:22:05Z", - "allow_failure": false - }, - - ... - ] -} - -``` - - By passing this through jq filter `.matrix` we extract the matrix part of the json -```bash -curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix' -``` -The resulting json looks like: -```json -[ - { - "id": 169519875, - "repository_id": 6097916, - "number": "19.1", - "config": { - "sudo": "required", - "dist": "trusty", - "language": "java", - "env": "TC_PATHS=\"tests/kafkatest/tests/client\"", - "jdk": "oraclejdk8", - "before_install": null, - "script": [ - "./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh" - ], - "services": [ - "docker" - ], - "before_cache": [ - "rm -f $HOME/.gradle/caches/modules-2/modules-2.lock", - "rm -fr $HOME/.gradle/caches/*/plugin-resolution/" - ], - "cache": { - "directories": [ - "$HOME/.m2/repository", - "$HOME/.gradle/caches/", - "$HOME/.gradle/wrapper/" - ] - }, - ".result": "configured", - "group": "stable", - "os": "linux" - }, - "result": null, - "started_at": "2016-10-21T13:35:43Z", - "finished_at": "2016-10-21T14:24:50Z", - "allow_failure": false - }, - { - "id": 169519876, - "repository_id": 6097916, - "number": "19.2", - "config": { - "sudo": "required", - "dist": "trusty", - "language": "java", - "env": "TC_PATHS=\"tests/kafkatest/tests/connect tests/kafkatest/tests/streams tests/kafkatest/tests/tools\"", - "jdk": "oraclejdk8", - "before_install": null, - "script": [ - "./gradlew systemTestLibs && /bin/bash ./tests/travis/run_tests.sh" - ], - "services": [ - "docker" - ], - "before_cache": [ - "rm -f $HOME/.gradle/caches/modules-2/modules-2.lock", - "rm -fr $HOME/.gradle/caches/*/plugin-resolution/" - ], - "cache": { - "directories": [ - "$HOME/.m2/repository", - "$HOME/.gradle/caches/", - "$HOME/.gradle/wrapper/" - ] - }, - ".result": "configured", - "group": "stable", - "os": "linux" - }, - "result": 1, - "started_at": "2016-10-21T13:35:46Z", - "finished_at": "2016-10-21T14:22:05Z", - "allow_failure": false - }, - - ... -] - -``` - - By further passing this through jq filter `map(.id)` we extract the id of - the builds for each of the splits -```bash -curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix|map(.id)' -``` -The resulting json looks like: -```json -[ - 169519875, - 169519876, - 169519877, - 169519878, - 169519879, - 169519880, - 169519881 -] -``` - - To use these ids in for loop we want to get rid of `[]` which is done by - passing it through `.[]` filter -```bash -curl -sSL https://api.travis-ci.org/apache/kafka/builds/169519874 | jq '.matrix|map(.id)|.[]' -``` -And we get -```text -169519875 -169519876 -169519877 -169519878 -169519879 -169519880 -169519881 -``` - - In the for loop we have made calls to fetch logs -```bash -curl -sSL "https://api.travis-ci.org/jobs/169519875/log.txt?deansi=true" | tail -``` -which gives us -```text -[INFO:2016-10-21 14:21:12,538]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: test 16 of 28 -[INFO:2016-10-21 14:21:12,538]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: setting up -[INFO:2016-10-21 14:21:30,810]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: running -[INFO:2016-10-21 14:24:35,519]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: PASS -[INFO:2016-10-21 14:24:35,519]: SerialTestRunner: kafkatest.tests.client.consumer_test.OffsetValidationTest.test_consumer_bounce.clean_shutdown=False.bounce_mode=rolling: tearing down - - -The job exceeded the maximum time limit for jobs, and has been terminated. - -``` -* Links - - [Travis-CI REST api documentation](https://docs.travis-ci.com/api) - - [jq Manual](https://stedolan.github.io/jq/manual/) - Local Quickstart ---------------- This quickstart will help you run the Kafka system tests on your local machine. Note this requires bringing up a cluster of virtual machines on your local computer, which is memory intensive; it currently requires around 10G RAM.