KAFKA-8841; Reduce overhead of ReplicaManager.updateFollowerFetchState (#7324)
This PR makes two changes to code in the ReplicaManager.updateFollowerFetchState path, which is in the hot path for follower fetches. Although calling ReplicaManager.updateFollowerFetch state is inexpensive on its own, it is called once for each partition every time a follower fetch occurs.
1. updateFollowerFetchState no longer calls maybeExpandIsr when the follower is already in the ISR. This avoid repeated expansion checks.
2. Partition.maybeIncrementLeaderHW is also in the hot path for ReplicaManager.updateFollowerFetchState. Partition.maybeIncrementLeaderHW calls Partition.remoteReplicas four times each iteration, and it performs a toSet conversion. maybeIncrementLeaderHW now avoids generating any intermediate collections when updating the HWM.
**Benchmark results for Partition.updateFollowerFetchState on a r5.xlarge:**
Old:
```
1288.633 ±(99.9%) 1.170 ns/op [Average]
(min, avg, max) = (1287.343, 1288.633, 1290.398), stdev = 1.037
CI (99.9%): [1287.463, 1289.802] (assumes normal distribution)
```
New (when follower fetch offset is updated):
```
261.727 ±(99.9%) 0.122 ns/op [Average]
(min, avg, max) = (261.565, 261.727, 261.937), stdev = 0.114
CI (99.9%): [261.605, 261.848] (assumes normal distribution)
```
New (when follower fetch offset is the same):
```
68.484 ±(99.9%) 0.025 ns/op [Average]
(min, avg, max) = (68.446, 68.484, 68.520), stdev = 0.023
CI (99.9%): [68.460, 68.509] (assumes normal distribution)
```
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
5 years ago
|
|
|
<!DOCTYPE import-control PUBLIC
|
|
|
|
|
"-//Puppy Crawl//DTD Import Control 1.1//EN"
|
|
|
|
|
"http://www.puppycrawl.com/dtds/import_control_1_1.dtd">
|
|
|
|
|
<!--
|
|
|
|
|
// Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
|
|
// contributor license agreements. See the NOTICE file distributed with
|
|
|
|
|
// this work for additional information regarding copyright ownership.
|
|
|
|
|
// The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
|
// (the "License"); you may not use this file except in compliance with
|
|
|
|
|
// the License. You may obtain a copy of the License at
|
|
|
|
|
//
|
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
//
|
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
|
// limitations under the License.
|
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
<import-control pkg="org.apache.kafka.jmh">
|
|
|
|
|
<allow pkg="java"/>
|
|
|
|
|
<allow pkg="scala"/>
|
|
|
|
|
<allow pkg="javax.management"/>
|
|
|
|
|
<allow pkg="org.slf4j"/>
|
|
|
|
|
<allow pkg="org.openjdk.jmh.annotations"/>
|
|
|
|
|
<allow pkg="org.openjdk.jmh.runner"/>
|
|
|
|
|
<allow pkg="org.openjdk.jmh.infra"/>
|
|
|
|
|
<allow pkg="java.security"/>
|
|
|
|
|
<allow pkg="javax.net.ssl"/>
|
|
|
|
|
<allow pkg="javax.security"/>
|
|
|
|
|
<allow pkg="org.apache.kafka.common"/>
|
|
|
|
|
<allow pkg="org.apache.kafka.clients.producer"/>
|
|
|
|
|
<allow pkg="kafka.cluster"/>
|
|
|
|
|
<allow pkg="kafka.log"/>
|
|
|
|
|
<allow pkg="kafka.server"/>
|
|
|
|
|
<allow pkg="kafka.api"/>
|
|
|
|
|
<allow pkg="kafka.controller"/>
|
|
|
|
|
<allow pkg="kafka.coordinator"/>
|
|
|
|
|
<allow pkg="kafka.network"/>
|
|
|
|
|
<allow pkg="kafka.utils"/>
|
|
|
|
|
<allow pkg="kafka.zk"/>
|
|
|
|
|
<allow class="kafka.utils.Pool"/>
|
KAFKA-8841; Reduce overhead of ReplicaManager.updateFollowerFetchState (#7324)
This PR makes two changes to code in the ReplicaManager.updateFollowerFetchState path, which is in the hot path for follower fetches. Although calling ReplicaManager.updateFollowerFetch state is inexpensive on its own, it is called once for each partition every time a follower fetch occurs.
1. updateFollowerFetchState no longer calls maybeExpandIsr when the follower is already in the ISR. This avoid repeated expansion checks.
2. Partition.maybeIncrementLeaderHW is also in the hot path for ReplicaManager.updateFollowerFetchState. Partition.maybeIncrementLeaderHW calls Partition.remoteReplicas four times each iteration, and it performs a toSet conversion. maybeIncrementLeaderHW now avoids generating any intermediate collections when updating the HWM.
**Benchmark results for Partition.updateFollowerFetchState on a r5.xlarge:**
Old:
```
1288.633 ±(99.9%) 1.170 ns/op [Average]
(min, avg, max) = (1287.343, 1288.633, 1290.398), stdev = 1.037
CI (99.9%): [1287.463, 1289.802] (assumes normal distribution)
```
New (when follower fetch offset is updated):
```
261.727 ±(99.9%) 0.122 ns/op [Average]
(min, avg, max) = (261.565, 261.727, 261.937), stdev = 0.114
CI (99.9%): [261.605, 261.848] (assumes normal distribution)
```
New (when follower fetch offset is the same):
```
68.484 ±(99.9%) 0.025 ns/op [Average]
(min, avg, max) = (68.446, 68.484, 68.520), stdev = 0.023
CI (99.9%): [68.460, 68.509] (assumes normal distribution)
```
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
5 years ago
|
|
|
<allow class="kafka.utils.KafkaScheduler"/>
|
|
|
|
|
<allow class="org.apache.kafka.clients.FetchSessionHandler"/>
|
KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates unused iterator (#8422)
https://github.com/apache/kafka/commit/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0 introduced skipKeyValueIterator(s) which were intended to be used, but in this case were created but were not used in offset validation.
A subset of the benchmark results follow. Looks like a 20% improvement in validation performance and a 40% reduction in garbage allocation for 1-2 batch sizes.
**# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 1, messageSize = 1000, messageVersion = 2)**
Before:
Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
64851.837 ±(99.9%) 944.248 ops/s [Average]
(min, avg, max) = (64505.317, 64851.837, 65114.359), stdev = 245.218
CI (99.9%): [63907.589, 65796.084] (assumes normal distribution)
"org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
164088.003 ±(99.9%) 0.004 B/op [Average]
(min, avg, max) = (164088.001, 164088.003, 164088.004), stdev = 0.001
CI (99.9%): [164087.998, 164088.007] (assumes normal distribution)
After:
Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
78910.273 ±(99.9%) 707.024 ops/s [Average]
(min, avg, max) = (78785.486, 78910.273, 79234.007), stdev = 183.612
CI (99.9%): [78203.249, 79617.297] (assumes normal distribution)
"org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
96440.002 ±(99.9%) 0.001 B/op [Average]
(min, avg, max) = (96440.002, 96440.002, 96440.002), stdev = 0.001
CI (99.9%): [96440.002, 96440.003] (assumes normal distribution)
**# Parameters: (bufferSupplierStr = NO_CACHING, bytes = RANDOM, compressionType = LZ4, maxBatchSize = 2, messageSize = 1000, messageVersion = 2)**
Before:
Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
64815.364 ±(99.9%) 639.309 ops/s [Average]
(min, avg, max) = (64594.545, 64815.364, 64983.305), stdev = 166.026
CI (99.9%): [64176.056, 65454.673] (assumes normal distribution)
"org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
163944.003 ±(99.9%) 0.001 B/op [Average]
(min, avg, max) = (163944.002, 163944.003, 163944.003), stdev = 0.001
CI (99.9%): [163944.002, 163944.004] (assumes normal distribution)
After:
Result "org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation":
77075.096 ±(99.9%) 201.092 ops/s [Average]
(min, avg, max) = (77021.537, 77075.096, 77129.693), stdev = 52.223
CI (99.9%): [76874.003, 77276.188] (assumes normal distribution)
"org.apache.kafka.jmh.record.RecordBatchIterationBenchmark.measureValidation:·gc.alloc.rate.norm":
96504.002 ±(99.9%) 0.003 B/op [Average]
(min, avg, max) = (96504.001, 96504.002, 96504.003), stdev = 0.001
CI (99.9%): [96503.999, 96504.005] (assumes normal distribution)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@juma.me.uk>
5 years ago
|
|
|
<allow pkg="kafka.common"/>
|
|
|
|
|
<allow pkg="kafka.message"/>
|
KAFKA-8841; Reduce overhead of ReplicaManager.updateFollowerFetchState (#7324)
This PR makes two changes to code in the ReplicaManager.updateFollowerFetchState path, which is in the hot path for follower fetches. Although calling ReplicaManager.updateFollowerFetch state is inexpensive on its own, it is called once for each partition every time a follower fetch occurs.
1. updateFollowerFetchState no longer calls maybeExpandIsr when the follower is already in the ISR. This avoid repeated expansion checks.
2. Partition.maybeIncrementLeaderHW is also in the hot path for ReplicaManager.updateFollowerFetchState. Partition.maybeIncrementLeaderHW calls Partition.remoteReplicas four times each iteration, and it performs a toSet conversion. maybeIncrementLeaderHW now avoids generating any intermediate collections when updating the HWM.
**Benchmark results for Partition.updateFollowerFetchState on a r5.xlarge:**
Old:
```
1288.633 ±(99.9%) 1.170 ns/op [Average]
(min, avg, max) = (1287.343, 1288.633, 1290.398), stdev = 1.037
CI (99.9%): [1287.463, 1289.802] (assumes normal distribution)
```
New (when follower fetch offset is updated):
```
261.727 ±(99.9%) 0.122 ns/op [Average]
(min, avg, max) = (261.565, 261.727, 261.937), stdev = 0.114
CI (99.9%): [261.605, 261.848] (assumes normal distribution)
```
New (when follower fetch offset is the same):
```
68.484 ±(99.9%) 0.025 ns/op [Average]
(min, avg, max) = (68.446, 68.484, 68.520), stdev = 0.023
CI (99.9%): [68.460, 68.509] (assumes normal distribution)
```
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
5 years ago
|
|
|
<allow pkg="org.mockito"/>
|
|
|
|
|
<allow pkg="kafka.security.authorizer"/>
|
|
|
|
|
<allow pkg="org.apache.kafka.server"/>
|
|
|
|
|
<allow pkg="org.apache.kafka.clients"/>
|
KAFKA-8841; Reduce overhead of ReplicaManager.updateFollowerFetchState (#7324)
This PR makes two changes to code in the ReplicaManager.updateFollowerFetchState path, which is in the hot path for follower fetches. Although calling ReplicaManager.updateFollowerFetch state is inexpensive on its own, it is called once for each partition every time a follower fetch occurs.
1. updateFollowerFetchState no longer calls maybeExpandIsr when the follower is already in the ISR. This avoid repeated expansion checks.
2. Partition.maybeIncrementLeaderHW is also in the hot path for ReplicaManager.updateFollowerFetchState. Partition.maybeIncrementLeaderHW calls Partition.remoteReplicas four times each iteration, and it performs a toSet conversion. maybeIncrementLeaderHW now avoids generating any intermediate collections when updating the HWM.
**Benchmark results for Partition.updateFollowerFetchState on a r5.xlarge:**
Old:
```
1288.633 ±(99.9%) 1.170 ns/op [Average]
(min, avg, max) = (1287.343, 1288.633, 1290.398), stdev = 1.037
CI (99.9%): [1287.463, 1289.802] (assumes normal distribution)
```
New (when follower fetch offset is updated):
```
261.727 ±(99.9%) 0.122 ns/op [Average]
(min, avg, max) = (261.565, 261.727, 261.937), stdev = 0.114
CI (99.9%): [261.605, 261.848] (assumes normal distribution)
```
New (when follower fetch offset is the same):
```
68.484 ±(99.9%) 0.025 ns/op [Average]
(min, avg, max) = (68.446, 68.484, 68.520), stdev = 0.023
CI (99.9%): [68.460, 68.509] (assumes normal distribution)
```
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
5 years ago
|
|
|
|
|
|
|
|
<subpackage name="cache">
|
|
|
|
|
</subpackage>
|
|
|
|
|
</import-control>
|
