src-kafka

Commit Graph

Author	SHA1	Message	Date
Guozhang Wang	3e9d1c1411	KAFKA-8106: Skipping ByteBuffer allocation of key / value / headers in logValidator (#6785 ) * KAFKA-8106:Reducing the allocation and copying of ByteBuffer when logValidator do validation. * KAFKA-8106:Reducing the allocation and copying of ByteBuffer when logValidator do validation. * github comments * use batch.skipKeyValueIterator * cleanups * no need to skip kv for uncompressed iterator * checkstyle fixes * fix findbugs * adding unit tests * reuse decompression buffer; and using streaming iterator * checkstyle * add unit tests * remove reusing buffer supplier * fix unit tests * add unit tests * use streaming iterator * minor refactoring * rename * github comments * github comments * reuse buffer at DefaultRecord caller * some further optimization * major refactoring * further refactoring * update comment * github comments * minor fix * add jmh benchmarks * update jmh * github comments * minor fix * github comments	5 years ago
Lifei Chen	3322439d98	MINOR: Document improvement (#6682 ) Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>	6 years ago
Ismael Juma	7d9e93ac6d	MINOR: Use https instead of http in links (#6477 ) Verified that the https links work. I didn't update the license header in this PR since that touches so many files. Will file a separate one for that. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>	6 years ago
Ismael Juma	12f310d50e	KAFKA-7612: Fix javac warnings and enable warnings as errors (#5900 ) - Use Xlint:all with 3 exclusions (filed KAFKA-7613 to remove the exclusions) - Use the same javac options when compiling tests (seems accidental that we didn't do this before) - Replaced several deprecated method calls with non-deprecated ones: - `KafkaConsumer.poll(long)` and `KafkaConsumer.close(long)` - `Class.newInstance` and `new Integer/Long` (deprecated since Java 9) - `scala.Console` (deprecated in Scala 2.11) - `PartitionData` taking a timestamp (one of them seemingly a bug) - `JsonMappingException` single parameter constructor - Fix unnecessary usage of raw types in several places. - Add @SuppressWarnings for deprecations, unchecked and switch fallthrough in several places. - Scala clean-ups (var -> val, ETA expansion warnings, avoid reflective calls) - Use lambdas to simplify code in a few places - Add @SafeVarargs, fix varargs usage and remove unnecessary `Utils.mkList` method Reviewers: Matthias J. Sax <mjsax@apache.org>, Manikumar Reddy <manikumar.reddy@gmail.com>, Randall Hauch <rhauch@gmail.com>, Bill Bejeck <bill@confluent.io>, Stanislav Kozlovski <stanislav_kozlovski@outlook.com>	6 years ago
Armin Braun	346d0ca538	MINOR: Fix needless GC + Result time unit in JMH Fixes two issues with the JMH benchmark example: * Trivial: The output should be in `ops/ms` for readability reasons (it's in the millions of operations per second) * Important: The benchmark is not actually measuring the LRU Cache performance as most of the time in each run is wasted on concatenating `key + counter` as well as `value + counter`. Fixed by pre-generating 10k K-V pairs (100x the cache capacity) and iterating over them. This brings the performance up by a factor of more than 5 on a standard 4 core i7 (`~6k/ms` before goes to `~35k/ms`). Author: Armin Braun <me@obrown.io> Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #2903 from original-brownbear/fix-jmh-example	7 years ago
Bill Bejeck	b6adb2dc89	KAFKA-3989; MINOR: follow-up: update script to run from kafka root …h-benchmarks/jmh.sh Author: bbejeck <bbejeck@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com> Closes #2654 from bbejeck/KAFKA-3989_follow_up	7 years ago
Jason Gustafson	f49697a279	KAFKA-5456; Ensure producer handles old format large compressed messages More specifically, fix the case where a compressed V0 or V1 message is larger than the producer batch size. Author: Jason Gustafson <jason@confluent.io> Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #3356 from hachikuji/KAFKA-5456	8 years ago
Ismael Juma	39eb31feae	MINOR: Optimise performance of `Topic.validate()` I included a JMH benchmark and the results follow. The implementation in this PR takes no more than 1/10th of the time when compared to trunk. I also included results for an alternative implementation that is a little slower than the one in the PR. Trunk: ```text TopicBenchmark.testValidate topic avgt 15 134.107 ± 3.956 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 316.241 ± 13.379 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 636.026 ± 30.272 ns/op ``` Implementation in the PR: ```text TopicBenchmark.testValidate topic avgt 15 13.153 ± 0.383 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 26.139 ± 0.896 ns/op TopicBenchmark.testValidate very-long-topic-name.with_more_text avgt 15 44.829 ± 1.390 ns/op ``` Alternative implementation where boolean validChar = Character.isLetterOrDigit(c) \|\| c == '.' \|\| c == '_' \|\| c == '-'; ```text TopicBenchmark.testValidate topic avgt 15 18.883 ± 1.044 ns/op TopicBenchmark.testValidate longer-topic-name avgt 15 36.696 ± 1.220 ns/op TopicBenchmark.testValidate very-long-topic-name_with_more_text avgt 15 65.956 ± 0.669 ns/op ``` Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #3234 from ijuma/optimise-topic-is-valid	8 years ago
Ismael Juma	c7bc8f7d8c	MINOR: Remove redundant volatile write in RecordHeaders The JMH benchmark included shows that the redundant volatile write causes the constructor of `ProducerRecord` to take more than 50% longer: ProducerRecordBenchmark.constructorBenchmark avgt 15 24.136 ± 1.458 ns/op (before) ProducerRecordBenchmark.constructorBenchmark avgt 15 14.904 ± 0.231 ns/op (after) Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3233 from ijuma/remove-volatile-write-in-records-header-constructor	8 years ago
Xavier Léauté	c060c48285	KAFKA-5150; Reduce LZ4 decompression overhead - reuse decompression buffers in consumer Fetcher - switch lz4 input stream to operate directly on ByteBuffers - avoids performance impact of catching exceptions when reaching the end of legacy record batches - more tests with both compressible / incompressible data, multiple blocks, and various other combinations to increase code coverage - fixes bug that would cause exception instead of invalid block size for invalid incompressible blocks - fixes bug if incompressible flag is set on end frame block size Overall this improves LZ4 decompression performance by up to 40x for small batches. Most improvements are seen for batches of size 1 with messages on the order of ~100B. We see at least 2x improvements for for batch sizes of < 10 messages, containing messages < 10kB This patch also yields 2-4x improvements on v1 small single message batches for other compression types. Full benchmark results can be found here https://gist.github.com/xvrl/05132e0643513df4adf842288be86efd Author: Xavier Léauté <xavier@confluent.io> Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #2967 from xvrl/kafka-5150	8 years ago
bbejeck	79f85039d7	KAFKA-3989; Initial support for adding a JMH benchmarking module Author: bbejeck <bbejeck@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes #1712 from bbejeck/KAFKA-3989_create_jmh_benchmarking_module	8 years ago

11 Commits (41e1c13a52082521b3702a8466f6b8a484cc6ddf)