src-kafka

Commit Graph

Author	SHA1	Message	Date
Anna Povzner	4e0ae79d5c	KAFKA-3214: Added system tests for compressed topics Added CompressionTest that tests 4 producers, each using a different compression type and one not using compression. Enabled VerifiableProducer to run producers with different compression types (passed in the constructor). This includes enabling each producer to output unique values, so that the verification process in ProduceConsumeValidateTest is correct (counts acks from all producers). Also a fix for console consumer to raise an exception if it sees the incorrect consumer output (before we swallowed an exception, so was hard to debug the issue). Author: Anna Povzner <anna@confluent.io> Reviewers: Geoff Anderson, Jason Gustafson Closes #958 from apovzner/kafka-3214	9 years ago
Geoff Anderson	18d7abd899	KAFKA-2927; reduce system test storage footprint Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default. Author: Geoff Anderson <geoff@confluent.io> Reviewers: Ben Stopford <ben@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #657 from granders/KAFKA-2927-reduce-log-footprint	9 years ago
Ben Stopford	6b4cc2ea2b	KAFKA-2771: Added rolling upgrade system test (ducktape) for Secured Cluster Tests rolling upgrade from PLAINTEXT to SSL Author: Ben Stopford <benstopford@gmail.com> Reviewers: Geoff Anderson, Ismael Juma Closes #496 from benstopford/security-upgrade-test	9 years ago
Rajini Sivaram	69a1cced49	KAFKA-2643: Run mirror maker ducktape tests with SSL and SASL Run tests with SSL, SASL_PLAINTEXT and SASL_SSL. Same security protocol is used for source and target Kafka. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Geoff Andreson, Ben Stopford Closes #559 from rajinisivaram/KAFKA-2643	9 years ago
Geoff Anderson	2e91806db4	KAFKA-2820: systest log level Restores control over log level in system test service class KafkaService. Author: Geoff Anderson <geoff@confluent.io> Reviewers: Ismael Juma, Ewen Cheslack-Postava Closes #538 from granders/KAFKA-2820-systest-log-level	9 years ago
Jason Gustafson	bce664b42a	KAFKA-2274: verifiable consumer and integration testing Author: Jason Gustafson <jason@confluent.io> Reviewers: Guozhang Wang, Geoff Anderson Closes #465 from hachikuji/KAFKA-2274	9 years ago
Geoff Anderson	7073fa7efe	KAFKA-2258: add failover to mirrormaker test This PR adds failover to simple end to end mirror maker test Marked as WIP for 2 reasons: - We may want to add a couple more test cases where kafka is being used to store offsets - There appears to be a test failure in the hard failover case Author: Geoff Anderson <geoff@confluent.io> Reviewers: Ewen Cheslack-Postava Closes #427 from granders/KAFKA-2258-mirrormaker-test	9 years ago
Rajini Sivaram	98db5ea94f	KAFKA-2644; Run relevant ducktape tests with SASL_PLAINTEXT and SASL_SSL Run sanity check, replication tests and benchmarks with SASL/Kerberos using MiniKdc. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Geoff Anderson <geoff@confluent.io>, Jun Rao <junrao@gmail.com> Closes #358 from rajinisivaram/KAFKA-2644	9 years ago
Geoff Anderson	e6b343302f	KAFKA-1888: rolling upgrade test ewencp gwenshap This needs some refactoring to avoid the duplicated code between replication test and upgrade test, but in shape for initial feedback. I'm interested in feedback on the added `KafkaConfig` class and `kafka_props` file. This addition makes it: - easier to attach different configs to different nodes (e.g. during broker upgrade process) - easier to reason about the configuration of a particular node Notes: - in the default values in the KafkaConfig class, I removed many properties which were in kafka.properties before. This is because most of those properties were set to what is already the default value. - when running non-trunk VerifiableProducer, I append the trunk tools jar to the classpath, and run it with the non-trunk kafka-run-class.sh script Author: Geoff Anderson <geoff@confluent.io> Reviewers: Dong Lin, Ewen Cheslack-Postava Closes #229 from granders/KAFKA-1888-upgrade-test	9 years ago
Rajini Sivaram	dd514b2bb8	KAFKA-2581: Run some existing ducktape tests with SSL Parametrize console consumer sanity test, replication tests and benchmarks tests to run with both PLAINTEXT and SSL. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Geoff Anderson, Ewen Cheslack-Postava, Guozhang Wang Closes #271 from rajinisivaram/KAFKA-2581	9 years ago
Geoff Anderson	1d2ae89c5a	KAFKA-2439; Add MirrorMaker service class for system tests Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0->0.3.0 Author: Geoff Anderson <geoff@confluent.io> Reviewers: Ewen Cheslack-Postava, Gwen Shapira Closes #148 from granders/KAFKA-2439 and squashes the following commits: `c7c3ebd` [Geoff Anderson] MirrorMaker now can run as multi-node service. Added kill -9 to various clean_node methods. `1e806f2` [Geoff Anderson] Various cleanups per review. `1b4b049` [Geoff Anderson] Added MirrorMaker service and a few corresponding sanity checks, as well as necessary config template files. A few additional updates to accomodate the change in wait_until from ducktape0.2.0->0.3.0	9 years ago
Geoff Anderson	e43c9aff92	KAFKA-2276; KIP-25 initial patch Initial patch for KIP-25 Note that to install ducktape, do not use pip to install ducktape. Instead: ``` $ git clone gitgithub.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` Author: Geoff Anderson <geoff@confluent.io> Author: Geoff <granders@gmail.com> Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen, Gwen, Jun, Guozhang Closes #70 from granders/KAFKA-2276 and squashes the following commits: `a62fb6c` [Geoff Anderson] fixed checkstyle errors `a70f0f8` [Geoff Anderson] Merged in upstream trunk. `8b62019` [Geoff Anderson] Merged in upstream trunk. `47b7b64` [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j. `a9e6a14` [Geoff Anderson] Merged in upstream changes `d18db7b` [Geoff Anderson] fixed :rat errors (needed to add licenses) `321fdf8` [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task `795fc75` [Geoff Anderson] Merged in changes from upstream trunk. `1d93f06` [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316 `2ea4e29` [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer `0eb6fdc` [Geoff Anderson] Merged in system-tests `69dd7be` [Geoff Anderson] Merged in trunk `4034dd6` [Geoff Anderson] Merged in upstream trunk `ede6450` [Geoff] Merge pull request #4 from confluentinc/move_muckrake `7751545` [Geoff Anderson] Corrected license headers `e6d532f` [Geoff Anderson] java 7 -> java 6 `8c61e2d` [Geoff Anderson] Reverted jdk back to 6 `f14c507` [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup. `98b7253` [Geoff Anderson] Updated consumer tests to pre-populate kafka logs `e6a41f1` [Geoff Anderson] removed stray println `b15b24f` [Geoff Anderson] leftover KafkaBenchmark in super call `0f75187` [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test `f469f84` [Geoff Anderson] Tweaked readme, added example Vagrantfile.local `3d73857` [Geoff Anderson] Merged downstream changes `42dcdb1` [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast `7f7c3e0` [Geoff Anderson] Updated setup.py for kafkatest `c60125c` [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency `4f476fe` [Geoff Anderson] Moved aws scripts to vagrant directory `5af88fc` [Geoff Anderson] Updated README to include aws quickstart `e5edf03` [Geoff Anderson] Updated example aws Vagrantfile.local `96533c3` [Geoff] Update aws-access-keys-commands `25a413d` [Geoff] Update aws-example-Vagrantfile.local `884b20e` [Geoff Anderson] Moved a bunch of files to kafkatest directory `fc7c81c` [Geoff Anderson] added setup.py `632be12` [Geoff] Merge pull request #3 from confluentinc/verbose-client `51a94fd` [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. `a80a428` [Geoff Anderson] Added shell program for VerifiableProducer. `d586fb0` [Geoff Anderson] Updated comments to reflect that throttler is not message-specific `6842ed1` [Geoff Anderson] left out a file from last commit `1228eef` [Geoff Anderson] Renamed throttler `9100417` [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. `0a5de8e` [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. `475423b` [Geoff Anderson] Convert class to string before adding to json object. `bc009f2` [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients) `c0526fe` [Geoff Anderson] Updates per review comments. `8b4b1f2` [Geoff Anderson] Minor updates to VerboseProducer `2777712` [Geoff Anderson] Added some metadata to producer output. `da94b8c` [Geoff Anderson] Added number of messages option. `07cd1c6` [Geoff Anderson] Added simple producer which prints status of produced messages to stdout. `a278988` [Geoff Anderson] fixed typos `f1914c3` [Liquan Pei] Merge pull request #2 from confluentinc/system_tests `81e4156` [Liquan Pei] Bootstrap Kafka system tests	9 years ago

12 Commits (9c34df1511a769b272893b75ec1ed90d38cc9576)