Mirror of Apache Kafka

kafkascala

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Tree: 4b97e50eea

0.10.0

0.10.1

0.10.2

0.11.0

0.7

0.7.0

0.7.1

0.7.2

0.8

0.8.0-beta1-candidate1

0.8.1

0.8.2

0.9.0

1.0

1.1

2.0

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

2.x

3.0

3.1

3.2

3.3

3.4

3.5

3.6

KAFKA-14367-join-group

KAFKA-14496-fix-oauth-encoder

KAFKA-15311

add-assignor-log-generation

cmccabe_2023-04-11_improve_controller_logging

cmccabe_2023-05-10_cleanup

cmccabe_2023-06-21_some_minor_fixes

cmccabe_kip_919

hekai-study-v2.8

john-disable-12049

kafka-10867-improved-task-idling-nolog

kip-866-zk-migration-to-kraft

metashell

minor-alter-isr-scheduling

printer

repro-task-idling-problem

revert-13391-kafka-14561

temp-8436

trunk

txn1

0.10.0.0

0.10.0.0-rc1

0.10.0.0-rc2

0.10.0.0-rc3

0.10.0.0-rc4

0.10.0.0-rc5

0.10.0.0-rc6

0.10.0.1

0.10.0.1-rc0

0.10.0.1-rc1

0.10.0.1-rc2

0.10.1.0

0.10.1.0-rc0

0.10.1.0-rc1

0.10.1.0-rc2

0.10.1.0-rc3

0.10.1.1

0.10.1.1-rc0

0.10.1.1-rc1

0.10.2.0

0.10.2.0-KAFKA-5526

0.10.2.0-rc0

0.10.2.0-rc1

0.10.2.0-rc2

0.10.2.1

0.10.2.1-rc0

0.10.2.1-rc1

0.10.2.1-rc2

0.10.2.1-rc3

0.10.2.2

0.10.2.2-rc0

0.10.2.2-rc1

0.11.0.0

0.11.0.0-rc0

0.11.0.0-rc1

0.11.0.0-rc2

0.11.0.1

0.11.0.1-rc0

0.11.0.2

0.11.0.2-rc0

0.11.0.3-rc0

0.8.0

0.8.0-beta1

0.8.0-beta1-candidate1

0.8.1

0.8.1.0

0.8.1.1

0.8.2-beta

0.8.2.0

0.8.2.0-cp

0.8.2.1

0.8.2.2

0.9.0.0

0.9.0.1

1.0.0

1.0.0-rc0

1.0.0-rc1

1.0.0-rc2

1.0.0-rc3

1.0.0-rc4

1.0.1

1.0.1-rc0

1.0.1-rc1

1.0.1-rc2

1.0.2

1.0.2-rc0

1.0.2-rc1

1.1.0

1.1.0-rc0

1.1.0-rc1

1.1.0-rc2

1.1.0-rc3

1.1.0-rc4

1.1.1

1.1.1-rc0

1.1.1-rc1

1.1.1-rc2

1.1.1-rc3

2.0.0

2.0.0-rc0

2.0.0-rc1

2.0.0-rc2

2.0.0-rc3

2.0.1

2.0.1-rc0

2.1.0

2.1.0-rc0

2.1.0-rc1

2.1.1

2.1.1-rc0

2.1.1-rc1

2.1.1-rc2

2.2.0

2.2.0-rc0

2.2.0-rc1

2.2.0-rc2

2.2.1

2.2.1-rc0

2.2.1-rc1

2.2.2

2.2.2-rc1

2.2.2-rc2

2.3.0

2.3.0-rc1

2.3.0-rc2

2.3.0-rc3

2.3.1

2.3.1-rc0

2.3.1-rc1

2.3.1-rc2

2.4.0

2.4.0-rc0

2.4.0-rc1

2.4.0-rc2

2.4.0-rc3

2.4.0-rc4

2.4.1

2.4.1-rc0

2.5.0

2.5.0-rc0

2.5.0-rc1

2.5.0-rc2

2.5.0-rc3

2.5.1

2.5.1-rc0

2.6.0

2.6.0-rc0

2.6.0-rc1

2.6.0-rc2

2.6.1

2.6.1-rc0

2.6.1-rc1

2.6.1-rc2

2.6.1-rc3

2.6.2

2.6.2-rc0

2.6.2-rc1

2.6.3

2.6.3-rc0

2.7.0

2.7.0-rc0

2.7.0-rc1

2.7.0-rc2

2.7.0-rc3

2.7.0-rc4

2.7.0-rc5

2.7.0-rc6

2.7.1

2.7.1-rc0

2.7.1-rc1

2.7.1-rc2

2.7.2

2.7.2-rc0

2.8.0

2.8.0-rc0

2.8.0-rc1

2.8.0-rc2

2.8.1

2.8.1-rc0

2.8.1-rc1

2.8.2

2.8.2-rc0

3.0.0

3.0.0-rc0

3.0.0-rc1

3.0.0-rc2

3.0.1

3.0.1-rc0

3.0.2

3.0.2-rc0

3.1.0

3.1.0-rc0

3.1.0-rc1

3.1.1

3.1.1-rc0

3.1.1-rc1

3.1.2

3.1.2-rc0

3.2.0

3.2.0-rc0

3.2.0-rc1

3.2.1

3.2.1-rc2

3.2.1-rc3

3.2.2

3.2.2-rc0

3.2.3

3.2.3-rc0

3.3.0

3.3.0-rc1

3.3.0-rc2

3.3.1

3.3.1-rc0

3.3.2

3.3.2-rc0

3.3.2-rc1

3.4.0

3.4.0-rc0

3.4.0-rc1

3.4.0-rc2

3.4.1

3.4.1-rc0

3.4.1-rc1

3.4.1-rc2

3.4.1-rc3

3.5.0

3.5.0-rc0

3.5.0-rc1

3.5.1

3.5.1-rc0

3.5.1-rc1

3.6.0

3.6.0-rc0

3.6.0-rc1

3.6.0-rc2

fetch

kafka-0.7.0-incubating-candidate-9

kafka-0.7.1-incubating-candidate-1

kafka-0.7.1-incubating-candidate-2

kafka-0.7.1-incubating-candidate-3

kafka-0.7.2-incubating-candidate-1

kafka-0.7.2-incubating-candidate-2

kafka-0.7.2-incubating-candidate-3

kafka-0.7.2-incubating-candidate-4

kafka-0.7.2-incubating-candidate-5

show

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from '4b97e50eea'

${ noResults }

src-kafka/jmh-benchmarks/src/main/java/org/apache/kafka/jmh

History

Lucas Bradstreet cfc34cace5 MINOR: reduce allocations in log start and recovery checkpoints (#8467) ... For brokers with replica counts > 4000, allocations from logsByDir become substantial. logsByDir is called often by LogManager.checkpointLogRecoveryOffsets and LogManager.checkpointLogStartOffsets. The approach used is similar to the one from the checkpointHighwatermarks change in https://github.com/apache/kafka/pull/6741. Are there better ways to structure out data structure to avoid creating logsByDir on demand for each checkpoint iteration? This micro-optimization will help as is, but if we can avoid doing this completely it'd be better. JMH benchmark results: ``` Before: Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 15 2.233 ± 0.013 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 100 thrpt 15 477.097 ± 49.731 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 100 thrpt 15 246083.007 ± 33.052 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 100 thrpt 15 475.683 ± 55.569 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 100 thrpt 15 245474.040 ± 14968.328 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 100 thrpt 15 0.001 ± 0.001 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 100 thrpt 15 0.341 ± 0.268 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 100 thrpt 15 129.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 100 thrpt 15 52.000 ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 15 0.572 ± 0.004 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 1000 thrpt 15 1360.240 ± 150.539 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 1000 thrpt 15 2750221.257 ± 891.024 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 1000 thrpt 15 1362.908 ± 148.799 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 1000 thrpt 15 2756395.092 ± 44671.843 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 1000 thrpt 15 0.017 ± 0.008 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 1000 thrpt 15 33.611 ± 14.401 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 1000 thrpt 15 273.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 1000 thrpt 15 186.000 ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 15 0.266 ± 0.002 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 2000 thrpt 15 1342.557 ± 171.260 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 2000 thrpt 15 5877881.729 ± 3695.086 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 2000 thrpt 15 1343.965 ± 186.069 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 2000 thrpt 15 5877788.561 ± 168540.343 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 2000 thrpt 15 0.081 ± 0.043 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 2000 thrpt 15 351.277 ± 167.006 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 2000 thrpt 15 253.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 2000 thrpt 15 231.000 ms JMH benchmarks done After: CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 15 2.809 ± 0.129 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 100 thrpt 15 211.248 ± 25.953 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 100 thrpt 15 86533.838 ± 3763.989 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 100 thrpt 15 211.512 ± 38.669 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 100 thrpt 15 86228.552 ± 9590.781 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 100 thrpt 15 ≈ 10⁻³ MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 100 thrpt 15 0.140 ± 0.111 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 100 thrpt 15 57.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 100 thrpt 15 25.000 ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 15 1.046 ± 0.030 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 1000 thrpt 15 524.597 ± 74.793 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 1000 thrpt 15 582898.889 ± 37552.262 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 1000 thrpt 15 519.675 ± 89.754 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 1000 thrpt 15 576371.150 ± 55972.955 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 1000 thrpt 15 0.009 ± 0.005 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 1000 thrpt 15 9.920 ± 5.375 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 1000 thrpt 15 111.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 1000 thrpt 15 56.000 ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 15 0.617 ± 0.007 ops/ms CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate 3 2000 thrpt 15 573.061 ± 95.931 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.alloc.rate.norm 3 2000 thrpt 15 1092098.004 ± 75140.633 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space 3 2000 thrpt 15 572.448 ± 97.960 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Eden_Space.norm 3 2000 thrpt 15 1091290.460 ± 85946.164 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen 3 2000 thrpt 15 0.010 ± 0.012 MB/sec CheckpointBench.measureCheckpointLogStartOffsets:·gc.churn.G1_Old_Gen.norm 3 2000 thrpt 15 19.990 ± 24.407 B/op CheckpointBench.measureCheckpointLogStartOffsets:·gc.count 3 2000 thrpt 15 109.000 counts CheckpointBench.measureCheckpointLogStartOffsets:·gc.time 3 2000 thrpt 15 67.000 ms JMH benchmarks done ``` For the 2000 topic, 3 partition case, we see a reduction in normalized allocations from 5877881B/op to 1284190.774B/op, a reduction of 78%. Some allocation profiles from a mid sized broker follow. I have seen worse, but these add up to around 3.8% on a broker that saw GC overhead in CPU time of around 30%. You could argue that this is relatively small, but it seems worthwhile for a low risk change.   Reviewers: Ismael Juma <ismael@juma.me.uk>		5 years ago
..
acl	KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer	5 years ago
cache	MINOR: Fix integer overflow in LRUCacheBenchmark (#7270)	5 years ago
common	MINOR: Optimise performance of `Topic.validate()`	8 years ago
fetcher	MINOR: Improve performance of checkpointHighWatermarks, patch 1/2 (#6741)	5 years ago
fetchsession	KAFKA-9039: Optimize ReplicaFetcher fetch path (#7443)	5 years ago
metadata	KAFKA-9670; Reduce allocations in Metadata Response preparation (#8236)	5 years ago
partition	MINOR: Improve performance of checkpointHighWatermarks, patch 1/2 (#6741)	5 years ago
producer	KAFKA-7612: Fix javac warnings and enable warnings as errors (#5900)	6 years ago
record	MINOR: fix inaccurate RecordBatchIterationBenchmark.measureValidation benchmark (#8428)	5 years ago
server	MINOR: reduce allocations in log start and recovery checkpoints (#8467)	5 years ago