Mirror of Apache Kafka
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

463 lines
19 KiB

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Spotbugs filtering.
Spotbugs is a static code analysis tool run as part of the "check" phase of the build.
This file dictates which categories of bugs and individual false positives that we suppress.
For a detailed description of spotbugs bug categories, see https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html
-->
<FindBugsFilter>
<!-- false positive in Java 11, see https://github.com/spotbugs/spotbugs/issues/756 -->
<Match>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE"/>
</Match>
<!-- false positive in Java 11, see https://github.com/spotbugs/spotbugs/issues/756 -->
<Match>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE"/>
</Match>
<Match>
<!-- Disable warnings about mutable objects and the use of public fields.
EI_EXPOSE_REP: May expose internal representation by returning reference to mutable object
EI_EXPOSE_REP2: May expose internal representation by incorporating reference to mutable object
MS_PKGPROTECT: Field should be package protected -->
<Bug pattern="EI_EXPOSE_REP,EI_EXPOSE_REP2,MS_PKGPROTECT"/>
</Match>
<Match>
<!-- Disable warnings about System.exit, until we decide to stop using it.
DM_EXIT: Method invokes System.exit -->
<Bug pattern="DM_EXIT"/>
</Match>
<Match>
<!-- Disable warnings about the lack of equals() when compareTo() is implemented.
EQ_COMPARETO_USE_OBJECT_EQUALS: This class defines a compareTo method but no equals() method -->
<Bug pattern="EQ_COMPARETO_USE_OBJECT_EQUALS"/>
</Match>
<Match>
<!-- Spotbugs tends to work a little bit better with Java than with Scala. We suppress
some categories of bug reports when using Scala, since spotbugs generates huge
numbers of false positives in some of these categories when examining Scala code.
NP_LOAD_OF_KNOWN_NULL_VALUE: The variable referenced at this point is known to be null
due to an earlier check against null.
NP_NULL_PARAM_DEREF: Method call passes null for non-null parameter.
NP_NULL_ON_SOME_PATH: Possible null pointer dereference
SE_BAD_FIELD: Non-transient non-serializable instance field in serializable class.
DM_STRING_CTOR: Method invokes inefficient new String(String) constructor.
DM_NEW_FOR_GETCLASS: Method allocates an object, only to get the class object.
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD: Write to static field from instance method.
DM_NUMBER_CTOR: Method invokes inefficient Number constructor; use static valueOf instead.
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE: Nullcheck of value previously dereferenced.
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE: Redundant nullcheck of value known to be non-null.
RCN_REDUNDANT_NULLCHECK_OF_NULL_VALUE: Redundant nullcheck of value known to be null.
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT: Return value of method without side effect is ignored.
NM_CLASS_NAMING_CONVENTION: Class names should start with an upper case letter.
NM_METHOD_NAMING_CONVENTION: Method names should start with a lower case letter.
EC_NULL_ARG: Call to equals(null)
NP_ALWAYS_NULL: Null pointer dereference
MINOR: Make the build compile with Scala 2.13 (#6989) Scala 2.13 support was added to build via #5454. This PR adjusts the code so that it compiles with 2.11, 2.12 and 2.13. Changes: * Add `scala-collection-compat` dependency. * Import `scala.collection.Seq` in a number of places for consistent behavior between Scala 2.11, 2.12 and 2.13. * Remove wildcard imports that were causing the Java classes to have priority over the Scala ones, related Scala issue: https://github.com/scala/scala/pull/6589. * Replace parallel collection usage with `Future`. The former is no longer included by default in the standard library. * Replace val _: Unit workaround with one that is more concise and works with Scala 2.13 * Replace `filterKeys` with `filter` when we expect a `Map`. `filterKeys` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `mapValues` with `map` or add a `toMap` as an additional transformation when we expect a `Map`. `mapValues` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `breakOut` with `iterator` and `to`, `breakOut` was removed in Scala 2.13. * Replace to() with toMap, toIndexedSeq and toSet * Replace `mutable.Buffer.--` with `filterNot`. * ControlException is an abstract class in Scala 2.13. * Variable arguments can only receive arrays or immutable.Seq in Scala 2.13. * Use `Factory` instead of `CanBuildFrom` in DecodeJson. `CanBuildFrom` behaves a bit differently in Scala 2.13 and it's been deprecated. `Factory` has the behavior we need and it's available via the compat library. * Fix failing tests due to behavior change in Scala 2.13, "Map.values.map is not strict in Scala 2.13" (https://github.com/scala/bug/issues/11589). * Use Java collections instead of Scala ones in StreamResetter (a Java class). * Adjust CheckpointFile.write to take an `Iterable` instead of `Seq` to avoid unnecessary collection copies. * Fix DelayedElectLeader to use a Map instead of Set and avoid `to` call that doesn't work in Scala 2.13. * Use unordered map for mapping in SimpleAclAuthorizer, mapping of ordered maps require an `Ordering` in Scala 2.13 for safety reasons. * Adapt `ConsumerGroupCommand` to compile with Scala 2.13. * CoreUtils.min takes an `Iterable` instead of `TraversableOnce`, the latter does not exist in Scala 2.13. * Replace `Unit` with `()` in a couple places. Scala 2.13 is stricter when it expects a value instead of a type. * Fix bug in CustomQuotaCallbackTest where we did not necessarily set `partitionRatio` correctly, `forall` can terminate early. * Add a couple of spotbugs exclusions that are needed by code generated by Scala 2.13 * Remove unused variables, simplify some code and remove procedure syntax in a few places. * Remove unused `CoreUtils.JSONEscapeString`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, José Armando García Sancio <jsancio@users.noreply.github.com>
5 years ago
MS_CANNOT_BE_FINAL: Field isn't final and can't be protected from malicious code
IC_INIT_CIRCULARITY: Initialization circularity
SE_NO_SUITABLE_CONSTRUCTOR: Class is Serializable but its superclass doesn't define a void constructor -->
<Source name="~.*\.scala" />
<Or>
<Bug pattern="NP_LOAD_OF_KNOWN_NULL_VALUE"/>
<Bug pattern="NP_NULL_ON_SOME_PATH"/>
<Bug pattern="NP_NULL_PARAM_DEREF"/>
<Bug pattern="SE_BAD_FIELD"/>
<Bug pattern="DM_STRING_CTOR"/>
<Bug pattern="DM_NEW_FOR_GETCLASS"/>
<Bug pattern="ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD"/>
<Bug pattern="DM_NUMBER_CTOR"/>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE"/>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE"/>
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NULL_VALUE"/>
<Bug pattern="RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT"/>
<Bug pattern="NM_CLASS_NAMING_CONVENTION"/>
<Bug pattern="NM_METHOD_NAMING_CONVENTION"/>
<Bug pattern="EC_NULL_ARG"/>
<Bug pattern="NP_ALWAYS_NULL"/>
<Bug pattern="MS_CANNOT_BE_FINAL"/>
MINOR: Make the build compile with Scala 2.13 (#6989) Scala 2.13 support was added to build via #5454. This PR adjusts the code so that it compiles with 2.11, 2.12 and 2.13. Changes: * Add `scala-collection-compat` dependency. * Import `scala.collection.Seq` in a number of places for consistent behavior between Scala 2.11, 2.12 and 2.13. * Remove wildcard imports that were causing the Java classes to have priority over the Scala ones, related Scala issue: https://github.com/scala/scala/pull/6589. * Replace parallel collection usage with `Future`. The former is no longer included by default in the standard library. * Replace val _: Unit workaround with one that is more concise and works with Scala 2.13 * Replace `filterKeys` with `filter` when we expect a `Map`. `filterKeys` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `mapValues` with `map` or add a `toMap` as an additional transformation when we expect a `Map`. `mapValues` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `breakOut` with `iterator` and `to`, `breakOut` was removed in Scala 2.13. * Replace to() with toMap, toIndexedSeq and toSet * Replace `mutable.Buffer.--` with `filterNot`. * ControlException is an abstract class in Scala 2.13. * Variable arguments can only receive arrays or immutable.Seq in Scala 2.13. * Use `Factory` instead of `CanBuildFrom` in DecodeJson. `CanBuildFrom` behaves a bit differently in Scala 2.13 and it's been deprecated. `Factory` has the behavior we need and it's available via the compat library. * Fix failing tests due to behavior change in Scala 2.13, "Map.values.map is not strict in Scala 2.13" (https://github.com/scala/bug/issues/11589). * Use Java collections instead of Scala ones in StreamResetter (a Java class). * Adjust CheckpointFile.write to take an `Iterable` instead of `Seq` to avoid unnecessary collection copies. * Fix DelayedElectLeader to use a Map instead of Set and avoid `to` call that doesn't work in Scala 2.13. * Use unordered map for mapping in SimpleAclAuthorizer, mapping of ordered maps require an `Ordering` in Scala 2.13 for safety reasons. * Adapt `ConsumerGroupCommand` to compile with Scala 2.13. * CoreUtils.min takes an `Iterable` instead of `TraversableOnce`, the latter does not exist in Scala 2.13. * Replace `Unit` with `()` in a couple places. Scala 2.13 is stricter when it expects a value instead of a type. * Fix bug in CustomQuotaCallbackTest where we did not necessarily set `partitionRatio` correctly, `forall` can terminate early. * Add a couple of spotbugs exclusions that are needed by code generated by Scala 2.13 * Remove unused variables, simplify some code and remove procedure syntax in a few places. * Remove unused `CoreUtils.JSONEscapeString`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, José Armando García Sancio <jsancio@users.noreply.github.com>
5 years ago
<Bug pattern="IC_INIT_CIRCULARITY"/>
<Bug pattern="SE_NO_SUITABLE_CONSTRUCTOR"/>
</Or>
</Match>
<Match>
KAFKA-8991: Enable scalac optimizer (#7452) The scalac optimizer is able to inline methods to avoid lambda allocations, eliminating the runtime cost of higher order functions in many cases. The compilation parameters we are using here were introduced in 2.12.x, so we don't enable them for Scala 2.11. Also, we enable a more aggressive inlining policy for the `core` project since it's not meant to be used as a library. See https://www.lightbend.com/blog/scala-inliner-optimizer for more information about the optimizer. I verified that the lambda allocation in the code below (from LogCleaner.scala) went away after this change with Scala 2.12 and 2.13. ```scala private def consumeAbortedTxnsUpTo(offset: Long): Unit = { while (abortedTransactions.headOption.exists(_.firstOffset <= offset)) { val abortedTxn = abortedTransactions.dequeue() ongoingAbortedTxns.getOrElseUpdate(abortedTxn.producerId, new AbortedTransactionMetadata(abortedTxn)) } } ``` The relevant part of the bytecode when compiled with Scala 2.13 looks like: ```text private void consumeAbortedTxnsUpTo(long); Code: 0: aload_0 1: invokespecial #54 // Method abortedTransactions:()Lscala/collection/mutable/PriorityQueue; 4: invokevirtual #175 // Method scala/collection/mutable/PriorityQueue.headOption:()Lscala/Option; 7: dup 8: ifnonnull 13 11: aconst_null 12: athrow 13: astore 4 15: aload 4 17: invokevirtual #145 // Method scala/Option.isEmpty:()Z 20: ifne 48 23: aload 4 25: invokevirtual #148 // Method scala/Option.get:()Ljava/lang/Object; 28: checkcast #177 // class kafka/log/AbortedTxn ``` The increased inlining causes some spurious spotBugs warnings, I added a few suppressions and fixed one warning by avoiding unnecessary boxing. Reviewers: Guozhang Wang <wangguoz@gmail.com>
5 years ago
<!-- Suppression for the equals() for extension methods. -->
<Class name="kafka.api.package$ElectLeadersRequestOps"/>
<Bug pattern="EQ_UNUSUAL"/>
</Match>
<Match>
<!-- Add a suppression for auto-generated calls to instanceof in kafka.utils.Json -->
<Source name="Json.scala"/>
<Package name="kafka.utils"/>
<Bug pattern="BC_VACUOUS_INSTANCEOF"/>
</Match>
KAFKA-8991: Enable scalac optimizer (#7452) The scalac optimizer is able to inline methods to avoid lambda allocations, eliminating the runtime cost of higher order functions in many cases. The compilation parameters we are using here were introduced in 2.12.x, so we don't enable them for Scala 2.11. Also, we enable a more aggressive inlining policy for the `core` project since it's not meant to be used as a library. See https://www.lightbend.com/blog/scala-inliner-optimizer for more information about the optimizer. I verified that the lambda allocation in the code below (from LogCleaner.scala) went away after this change with Scala 2.12 and 2.13. ```scala private def consumeAbortedTxnsUpTo(offset: Long): Unit = { while (abortedTransactions.headOption.exists(_.firstOffset <= offset)) { val abortedTxn = abortedTransactions.dequeue() ongoingAbortedTxns.getOrElseUpdate(abortedTxn.producerId, new AbortedTransactionMetadata(abortedTxn)) } } ``` The relevant part of the bytecode when compiled with Scala 2.13 looks like: ```text private void consumeAbortedTxnsUpTo(long); Code: 0: aload_0 1: invokespecial #54 // Method abortedTransactions:()Lscala/collection/mutable/PriorityQueue; 4: invokevirtual #175 // Method scala/collection/mutable/PriorityQueue.headOption:()Lscala/Option; 7: dup 8: ifnonnull 13 11: aconst_null 12: athrow 13: astore 4 15: aload 4 17: invokevirtual #145 // Method scala/Option.isEmpty:()Z 20: ifne 48 23: aload 4 25: invokevirtual #148 // Method scala/Option.get:()Ljava/lang/Object; 28: checkcast #177 // class kafka/log/AbortedTxn ``` The increased inlining causes some spurious spotBugs warnings, I added a few suppressions and fixed one warning by avoiding unnecessary boxing. Reviewers: Guozhang Wang <wangguoz@gmail.com>
5 years ago
<Match>
<!-- A spurious null check after inlining by the scalac optimizer confuses spotBugs -->
<Class name="kafka.log.Log"/>
<Bug pattern="NP_NULL_ON_SOME_PATH_EXCEPTION"/>
</Match>
<Match>
<!-- A spurious null check after inlining by the scalac optimizer confuses spotBugs -->
<Class name="kafka.tools.StateChangeLogMerger$"/>
<Bug pattern="NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE"/>
</Match>
<Match>
<!-- Unboxing to Int to make scalac happy makes spotBugs unhappy -->
<Class name="kafka.tools.ConsoleConsumer$ConsumerConfig"/>
<Bug pattern="BX_UNBOXING_IMMEDIATELY_REBOXED"/>
</Match>
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="AdminUtils.scala"/>
<Package name="kafka.admin"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="ControllerContext.scala"/>
<Package name="kafka.controller"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
<Match>
<!-- offsets is a lazy val and it confuses spotBugs with its locking scheme -->
<Class name="kafka.server.checkpoints.LazyOffsetCheckpointMap"/>
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
</Match>
KAFKA-8991: Enable scalac optimizer (#7452) The scalac optimizer is able to inline methods to avoid lambda allocations, eliminating the runtime cost of higher order functions in many cases. The compilation parameters we are using here were introduced in 2.12.x, so we don't enable them for Scala 2.11. Also, we enable a more aggressive inlining policy for the `core` project since it's not meant to be used as a library. See https://www.lightbend.com/blog/scala-inliner-optimizer for more information about the optimizer. I verified that the lambda allocation in the code below (from LogCleaner.scala) went away after this change with Scala 2.12 and 2.13. ```scala private def consumeAbortedTxnsUpTo(offset: Long): Unit = { while (abortedTransactions.headOption.exists(_.firstOffset <= offset)) { val abortedTxn = abortedTransactions.dequeue() ongoingAbortedTxns.getOrElseUpdate(abortedTxn.producerId, new AbortedTransactionMetadata(abortedTxn)) } } ``` The relevant part of the bytecode when compiled with Scala 2.13 looks like: ```text private void consumeAbortedTxnsUpTo(long); Code: 0: aload_0 1: invokespecial #54 // Method abortedTransactions:()Lscala/collection/mutable/PriorityQueue; 4: invokevirtual #175 // Method scala/collection/mutable/PriorityQueue.headOption:()Lscala/Option; 7: dup 8: ifnonnull 13 11: aconst_null 12: athrow 13: astore 4 15: aload 4 17: invokevirtual #145 // Method scala/Option.isEmpty:()Z 20: ifne 48 23: aload 4 25: invokevirtual #148 // Method scala/Option.get:()Ljava/lang/Object; 28: checkcast #177 // class kafka/log/AbortedTxn ``` The increased inlining causes some spurious spotBugs warnings, I added a few suppressions and fixed one warning by avoiding unnecessary boxing. Reviewers: Guozhang Wang <wangguoz@gmail.com>
5 years ago
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="ReplicaManager.scala"/>
<Package name="kafka.server"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
KAFKA-8991: Enable scalac optimizer (#7452) The scalac optimizer is able to inline methods to avoid lambda allocations, eliminating the runtime cost of higher order functions in many cases. The compilation parameters we are using here were introduced in 2.12.x, so we don't enable them for Scala 2.11. Also, we enable a more aggressive inlining policy for the `core` project since it's not meant to be used as a library. See https://www.lightbend.com/blog/scala-inliner-optimizer for more information about the optimizer. I verified that the lambda allocation in the code below (from LogCleaner.scala) went away after this change with Scala 2.12 and 2.13. ```scala private def consumeAbortedTxnsUpTo(offset: Long): Unit = { while (abortedTransactions.headOption.exists(_.firstOffset <= offset)) { val abortedTxn = abortedTransactions.dequeue() ongoingAbortedTxns.getOrElseUpdate(abortedTxn.producerId, new AbortedTransactionMetadata(abortedTxn)) } } ``` The relevant part of the bytecode when compiled with Scala 2.13 looks like: ```text private void consumeAbortedTxnsUpTo(long); Code: 0: aload_0 1: invokespecial #54 // Method abortedTransactions:()Lscala/collection/mutable/PriorityQueue; 4: invokevirtual #175 // Method scala/collection/mutable/PriorityQueue.headOption:()Lscala/Option; 7: dup 8: ifnonnull 13 11: aconst_null 12: athrow 13: astore 4 15: aload 4 17: invokevirtual #145 // Method scala/Option.isEmpty:()Z 20: ifne 48 23: aload 4 25: invokevirtual #148 // Method scala/Option.get:()Ljava/lang/Object; 28: checkcast #177 // class kafka/log/AbortedTxn ``` The increased inlining causes some spurious spotBugs warnings, I added a few suppressions and fixed one warning by avoiding unnecessary boxing. Reviewers: Guozhang Wang <wangguoz@gmail.com>
5 years ago
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="LogManager.scala"/>
<Package name="kafka.log"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="DelayedElectLeader.scala"/>
<Package name="kafka.server"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="AdminZkClient.scala"/>
<Package name="kafka.zk"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
MINOR: Fix Scala 2.13 compiler warnings (#8390) Once Scala 2.13.2 is officially released, I will submit a follow up PR that enables `-Xfatal-warnings` with the necessary warning exclusions. Compiler warning exclusions were only introduced in 2.13.2 and hence why we have to wait for that. I used a snapshot build to test it in the meantime. Changes: * Remove Deprecated annotation from internal request classes * Class.newInstance is deprecated in favor of Class.getConstructor().newInstance * Replace deprecated JavaConversions with CollectionConverters * Remove unused kafka.cluster.Cluster * Don't use Map and Set methods deprecated in 2.13: - collection.Map +, ++, -, --, mapValues, filterKeys, retain - collection.Set +, ++, -, -- * Add scala-collection-compat dependency to streams-scala and update version to 2.1.4. * Replace usages of deprecated Either.get and Either.right * Replace usage of deprecated Integer(String) constructor * `import scala.language.implicitConversions` is not needed in Scala 2.13 * Replace usage of deprecated `toIterator`, `Traversable`, `seq`, `reverseMap`, `hasDefiniteSize` * Replace usage of deprecated alterConfigs with incrementalAlterConfigs where possible * Fix implicit widening conversions from Long/Int to Double/Float * Avoid implicit conversions to String * Eliminate usage of deprecated procedure syntax * Remove `println`in `LogValidatorTest` instead of fixing the compiler warning since tests should not `println`. * Eliminate implicit conversion from Array to Seq * Remove unnecessary usage of 3 argument assertEquals * Replace `toStream` with `iterator` * Do not use deprecated SaslConfigs.DEFAULT_SASL_ENABLED_MECHANISMS * Replace StringBuilder.newBuilder with new StringBuilder * Rename AclBuffers to AclSeqs and remove usage of `filterKeys` * More consistent usage of Set/Map in Controller classes: this also fixes deprecated warnings with Scala 2.13 * Add spotBugs exclusion for inliner artifact in KafkaApis with Scala 2.12. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
5 years ago
<Match>
<!-- Uncallable anonymous methods are left behind after inlining by scalac 2.12, fixed in 2.13 -->
<Source name="KafkaApis.scala"/>
<Package name="kafka.server"/>
<Bug pattern="UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS"/>
</Match>
<Match>
<!-- Keeping this class for compatibility. It's deprecated and will be removed in the next major release -->
<Source name="MessageFormatter.scala"/>
<Package name="kafka.common"/>
<Bug pattern="NM_SAME_SIMPLE_NAME_AS_INTERFACE"/>
</Match>
<Match>
<!-- Suppress a warning about some static initializers in Schema using instances of a
subclass. -->
<Or>
<Class name="org.apache.kafka.connect.data.Schema"/>
<Class name="org.apache.kafka.connect.data.SchemaBuilder"/>
</Or>
<Bug pattern="IC_SUPERCLASS_USES_SUBCLASS_DURING_INITIALIZATION"/>
</Match>
<Match>
<!-- Suppress warnings about unread protected fields in some public classes.
Although these are not read in Kafka Connect code, they are part of the API. -->
<Or>
<Class name="org.apache.kafka.connect.connector.Connector"/>
<Class name="org.apache.kafka.connect.sink.SinkTask"/>
<Class name="org.apache.kafka.connect.source.SourceTask"/>
</Or>
<Bug pattern="URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD"/>
</Match>
<Match>
<!-- Suppress warnings about converting an integer number of
milliseconds to a java.util.Date object. We do this intentionally in
org.apache.kafka.connect.data.Time. -->
<Class name="org.apache.kafka.connect.data.Time"/>
<Method name="toLogical"/>
<Bug pattern="ICAST_INT_2_LONG_AS_INSTANT"/>
</Match>
<Match>
<!-- Suppress some minor warnings about machine-generated code for
benchmarking. -->
<Or>
<Package name="org.apache.kafka.jmh.cache.generated"/>
<Package name="org.apache.kafka.jmh.common.generated"/>
<Package name="org.apache.kafka.jmh.record.generated"/>
KAFKA-8841; Reduce overhead of ReplicaManager.updateFollowerFetchState (#7324) This PR makes two changes to code in the ReplicaManager.updateFollowerFetchState path, which is in the hot path for follower fetches. Although calling ReplicaManager.updateFollowerFetch state is inexpensive on its own, it is called once for each partition every time a follower fetch occurs. 1. updateFollowerFetchState no longer calls maybeExpandIsr when the follower is already in the ISR. This avoid repeated expansion checks. 2. Partition.maybeIncrementLeaderHW is also in the hot path for ReplicaManager.updateFollowerFetchState. Partition.maybeIncrementLeaderHW calls Partition.remoteReplicas four times each iteration, and it performs a toSet conversion. maybeIncrementLeaderHW now avoids generating any intermediate collections when updating the HWM. **Benchmark results for Partition.updateFollowerFetchState on a r5.xlarge:** Old: ``` 1288.633 ±(99.9%) 1.170 ns/op [Average] (min, avg, max) = (1287.343, 1288.633, 1290.398), stdev = 1.037 CI (99.9%): [1287.463, 1289.802] (assumes normal distribution) ``` New (when follower fetch offset is updated): ``` 261.727 ±(99.9%) 0.122 ns/op [Average] (min, avg, max) = (261.565, 261.727, 261.937), stdev = 0.114 CI (99.9%): [261.605, 261.848] (assumes normal distribution) ``` New (when follower fetch offset is the same): ``` 68.484 ±(99.9%) 0.025 ns/op [Average] (min, avg, max) = (68.446, 68.484, 68.520), stdev = 0.023 CI (99.9%): [68.460, 68.509] (assumes normal distribution) ``` Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
5 years ago
<Package name="org.apache.kafka.jmh.partition.generated"/>
<Package name="org.apache.kafka.jmh.producer.generated"/>
KAFKA-9039: Optimize ReplicaFetcher fetch path (#7443) Improves the performance of the replica fetcher for high partition count fetch requests, where a majority of the partitions did not update between fetch requests. All benchmarks were run on an r5x.large. Vanilla Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 15 26491.825 ± 438.463 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 15 153941.952 ± 4337.073 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 15 339868.602 ± 4201.462 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 15 2588878.448 ± 22172.482 ns/op From 100 to 5000 partitions the latency increase is 2588878.448 / 26491.825 = 97. Avoid gettimeofdaycalls in steady state fetch states 8545888 Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 15 22685.381 ± 267.727 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 15 113622.521 ± 1854.254 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 15 273698.740 ± 9269.554 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 15 2189223.207 ± 1706.945 ns/op From 100 to 5000 partitions the latency increase is 2189223.207 / 22685.381 = 97X Avoid copying partition states to maintain fetch offsets 29fdd60 Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 15 17039.989 ± 609.355 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 15 99371.086 ± 1833.256 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 15 216071.333 ± 3714.147 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 15 2035678.223 ± 5195.232 ns/op From 100 to 5000 partitions the latency increase is 2035678.223 / 17039.989 = 119X Keep lag alongside PartitionFetchState to avoid expensive isReplicaInSync check 0e57e3e Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 15 15131.684 ± 382.088 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 15 86813.843 ± 3346.385 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 15 193050.381 ± 3281.833 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 15 1801488.513 ± 2756.355 ns/op From 100 to 5000 partitions the latency increase is 1801488.513 / 15131.684 = 119X Fetch session optimizations (mostly presizing the next hashmap, and avoiding making a copy of sessionPartitions, as a deep copy is not required for the ReplicaFetcher) 2614b24 Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 15 11386.203 ± 416.701 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 15 60820.292 ± 3163.001 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 15 146242.158 ± 1937.254 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 15 1366768.926 ± 3305.712 ns/op From 100 to 5000 partitions the latency increase is 1366768.926 / 11386.203 = 120 Reviewers: Jun Rao <junrao@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
5 years ago
<Package name="org.apache.kafka.jmh.fetchsession.generated"/>
<Package name="org.apache.kafka.jmh.fetcher.generated"/>
<Package name="org.apache.kafka.jmh.server.generated"/>
<Package name="org.apache.kafka.jmh.consumer.generated"/>
</Or>
</Match>
KAFKA-7609; Add Protocol Generator for Kafka (#5893) This patch adds a framework to automatically generate the request/response classes for Kafka's protocol. The code will be updated to use the generated classes in follow-up patches. Below is a brief summary of the included components: **buildSrc/src** The message generator code is here. This code is automatically re-run by gradle when one of the schema files changes. The entire directory is processed at once to minimize the number of times we have to start a new JVM. We use Jackson to translate the JSON files into Java objects. **clients/src/main/java/org/apache/kafka/common/protocol/Message.java** This is the interface implemented by all automatically generated messages. **clients/src/main/java/org/apache/kafka/common/protocol/MessageUtil.java** Some utility functions used by the generated message code. **clients/src/main/java/org/apache/kafka/common/protocol/Readable.java, Writable.java, ByteBufferAccessor.java** The generated message code uses these classes for writing to a buffer. **clients/src/main/message/README.md** This README file explains how the JSON schemas work. **clients/src/main/message/\*.json** The JSON files in this directory implement every supported version of every Kafka API. The unit tests automatically validate that the generated schemas match the hand-written schemas in our code. Additionally, there are some things like request and response headers that have schemas here. **clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java** I added an optimization here for empty sets. This is useful here because I want all messages to start with empty sets by default prior to being loaded with data. This is similar to the "empty list" optimizations in the `java.util.ArrayList` class. Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Ismael Juma <ismael@juma.me.uk>, Bob Barrett <bob.barrett@outlook.com>, Jason Gustafson <jason@confluent.io>
6 years ago
<Match>
<!-- Suppress warnings about generated schema arrays. -->
<Or>
<Package name="org.apache.kafka.common.message"/>
<Package name="org.apache.kafka.streams.internals.generated"/>
</Or>
KAFKA-7609; Add Protocol Generator for Kafka (#5893) This patch adds a framework to automatically generate the request/response classes for Kafka's protocol. The code will be updated to use the generated classes in follow-up patches. Below is a brief summary of the included components: **buildSrc/src** The message generator code is here. This code is automatically re-run by gradle when one of the schema files changes. The entire directory is processed at once to minimize the number of times we have to start a new JVM. We use Jackson to translate the JSON files into Java objects. **clients/src/main/java/org/apache/kafka/common/protocol/Message.java** This is the interface implemented by all automatically generated messages. **clients/src/main/java/org/apache/kafka/common/protocol/MessageUtil.java** Some utility functions used by the generated message code. **clients/src/main/java/org/apache/kafka/common/protocol/Readable.java, Writable.java, ByteBufferAccessor.java** The generated message code uses these classes for writing to a buffer. **clients/src/main/message/README.md** This README file explains how the JSON schemas work. **clients/src/main/message/\*.json** The JSON files in this directory implement every supported version of every Kafka API. The unit tests automatically validate that the generated schemas match the hand-written schemas in our code. Additionally, there are some things like request and response headers that have schemas here. **clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java** I added an optimization here for empty sets. This is useful here because I want all messages to start with empty sets by default prior to being loaded with data. This is similar to the "empty list" optimizations in the `java.util.ArrayList` class. Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Ismael Juma <ismael@juma.me.uk>, Bob Barrett <bob.barrett@outlook.com>, Jason Gustafson <jason@confluent.io>
6 years ago
<Bug pattern="MS_MUTABLE_ARRAY"/>
</Match>
<Match>
<!-- Suppress warnings about ignoring the return value of await.
This is done intentionally because we use other clues to determine
if the wait was cut short. -->
<Class name="org.apache.kafka.connect.runtime.WorkerSourceTask"/>
<Method name="execute"/>
<Bug pattern="RV_RETURN_VALUE_IGNORED"/>
</Match>
<Match>
<!-- Suppress a warning about ignoring the return value of await.
This is done intentionally because we use other clues to determine
if the wait was cut short. -->
<Package name="kafka.log"/>
<Source name="LogCleanerManager.scala"/>
<Bug pattern="RV_RETURN_VALUE_IGNORED,RV_RETURN_VALUE_IGNORED_BAD_PRACTICE"/>
</Match>
<Match>
<!-- Suppress some warnings about intentional switch statement fallthrough. -->
<Class name="org.apache.kafka.connect.runtime.WorkerConnector"/>
<Or>
<Method name="doStart"/>
<Method name="pause"/>
</Or>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
</Match>
<Match>
<!-- Suppress a warning about intentional switch statement fallthrough. -->
<Class name="org.apache.kafka.common.security.authenticator.SaslClientAuthenticator"/>
<Method name="authenticate"/>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
</Match>
<Match>
<!-- Suppress a spurious warning about a missing default case. -->
<Or>
<Class name="org.apache.kafka.common.utils.Crc32"/>
<Class name="org.apache.kafka.common.utils.PureJavaCrc32C"/>
</Or>
<Method name="update"/>
<Bug pattern="SF_SWITCH_NO_DEFAULT"/>
</Match>
<Match>
<!-- Suppress a warning about intentional missing default cases and fallthroughs. -->
<Class name="org.apache.kafka.common.utils.Utils"/>
<Method name="murmur2"/>
<Or>
<Bug pattern="SF_SWITCH_NO_DEFAULT"/>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
</Or>
</Match>
<Match>
<!-- Suppress a warning about intentional missing default cases and fallthroughs. -->
<Class name="org.apache.kafka.streams.state.internals.Murmur3"/>
<Or>
<Bug pattern="SF_SWITCH_NO_DEFAULT"/>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
</Or>
</Match>
<Match>
<!-- Suppress a warning about intentional missing default cases and fallthroughs. -->
<Class name="org.apache.kafka.streams.state.internals.Murmur3$IncrementalHash32"/>
<Or>
<Bug pattern="SF_SWITCH_NO_DEFAULT"/>
<Bug pattern="SF_SWITCH_FALLTHROUGH"/>
</Or>
</Match>
<Match>
<!-- Suppress a spurious warning about locks not being released on all paths.
This happens because there is an 'if' statement that checks if we have the lock before
releasing it. -->
<Class name="org.apache.kafka.clients.producer.internals.BufferPool"/>
<Method name="allocate"/>
<Bug pattern="UL_UNRELEASED_LOCK"/>
</Match>
<Match>
<!-- Suppress warnings about synchronizing on the UnsentRequests
ConcurrentHashMap. This is done deliberately. -->
<Package name="org.apache.kafka.clients.consumer.internals"/>
<Source name="ConsumerNetworkClient.java"/>
<Bug pattern="JLM_JSR166_UTILCONCURRENT_MONITORENTER"/>
</Match>
<Match>
<!-- Suppress inconsistent synchronization warnings about
AbstractCoordinator#coordinator. See KAFKA-4992 for details.-->
<Class name="org.apache.kafka.clients.consumer.internals.AbstractCoordinator"/>
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
</Match>
<Match>
<!-- Suppress warnings about implementing compareTo but not equals. -->
<Class name="org.apache.kafka.streams.processor.internals.Stamped"/>
<Bug pattern="EQ_COMPARETO_USE_OBJECT_EQUALS"/>
</Match>
<Match>
<!-- Suppress warning about applicationID that gets initialized before
any other threads are created, but is used in synchronized and
unsynchronized methods because it comes from the configs,
passed through rewriteTopology. -->
<Package name="org.apache.kafka.streams.processor.internals"/>
<Source name="InternalTopologyBuilder.java"/>
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
</Match>
<Match>
<!-- Ignore a warning about synchronizing on an AtomicBoolean -->
<Package name="kafka.metrics"/>
<Source name="KafkaMetricsReporter.scala"/>
<Method name="startReporters"/>
<Bug pattern="JLM_JSR166_UTILCONCURRENT_MONITORENTER"/>
</Match>
<Match>
<!-- Ignore spurious warning about imbalanced synchronization on a cached Exception. Some accesses of the
variable must occur in synchronized blocks, while some need not, because of the nature of the producer.
This seems to throw the static checker off. -->
<Package name="org.apache.kafka.clients.producer" />
<Source name="TransactionState.java" />
<Bug pattern="IS2_INCONSISTENT_SYNC" />
</Match>
<Match>
<!-- Suppress a spurious warning about an unreleased lock. -->
<Class name="kafka.utils.timer.SystemTimer"/>
<Method name="add"/>
<Bug pattern="UL_UNRELEASED_LOCK_EXCEPTION_PATH"/>
</Match>
<Match>
<!-- Suppress a warning about an intentional infinite loop. -->
<Package name="kafka.utils"/>
<Source name="Throttler.scala"/>
<Method name="main"/>
<Bug pattern="IL_INFINITE_LOOP"/>
</Match>
<Match>
<!-- Suppress a spurious warning about calling notify without modifying
other state under the monitor. -->
<Package name="org.apache.kafka.trogdor.workload"/>
<Source name="RoundTripWorker.java"/>
<Bug pattern="NN_NAKED_NOTIFY"/>
</Match>
<Match>
<Package name="org.apache.kafka.streams.scala"/>
<Source name="FunctionConversions.scala"/>
<Bug pattern="EQ_UNUSUAL"/>
</Match>
<Match>
<Package name="org.apache.kafka.streams.scala"/>
<Source name="FunctionsCompatConversions.scala"/>
<Bug pattern="EQ_UNUSUAL"/>
</Match>
- <Match>
<Package name="org.apache.kafka.streams.scala"/>
<Or>
<Class name="org.apache.kafka.streams.scala.Serializer" />
<Class name="org.apache.kafka.streams.scala.Deserializer" />
</Or>
<Bug pattern="NM_SAME_SIMPLE_NAME_AS_INTERFACE"/>
</Match>
<!-- Suppress warnings for unused members that are undetectably used by Jackson -->
<Match>
<Package name="org.apache.kafka.streams.examples.pageview"/>
<Bug pattern="NP_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD"/>
</Match>
<Match>
<Package name="org.apache.kafka.streams.examples.pageview"/>
<Bug pattern="URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD"/>
</Match>
<Match>
<Package name="org.apache.kafka.streams.examples.pageview"/>
<Bug pattern="UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD"/>
</Match>
<!-- END Suppress warnings for unused members that are undetectably used by Jackson -->
<Match>
<!-- Suppress a warning about ignoring return value because this is intentional. -->
<Class name="org.apache.kafka.common.config.AbstractConfig$ResolvingMap"/>
<Method name="get"/>
<Bug pattern="RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT"/>
</Match>
Kafka 9626: Improve ACLAuthorizer.acls() performance This PR avoids creation of unnecessary sets in AclAuthorizer.acls() method implementation. Perf results: **Old** ``` Benchmark (aclCount) (resourceCount) Mode Cnt Score Error Units AclAuthorizerBenchmark.testAclsIterator 5 5000 avgt 15 5.821 ? 0.309 ms/op AclAuthorizerBenchmark.testAclsIterator 5 10000 avgt 15 15.303 ? 0.107 ms/op AclAuthorizerBenchmark.testAclsIterator 5 50000 avgt 15 74.976 ? 0.543 ms/op AclAuthorizerBenchmark.testAclsIterator 10 5000 avgt 15 15.366 ? 0.184 ms/op AclAuthorizerBenchmark.testAclsIterator 10 10000 avgt 15 29.899 ? 0.129 ms/op AclAuthorizerBenchmark.testAclsIterator 10 50000 avgt 15 167.301 ? 1.723 ms/op AclAuthorizerBenchmark.testAclsIterator 15 5000 avgt 15 21.980 ? 0.114 ms/op AclAuthorizerBenchmark.testAclsIterator 15 10000 avgt 15 44.385 ? 0.255 ms/op AclAuthorizerBenchmark.testAclsIterator 15 50000 avgt 15 241.919 ? 3.955 ms/op ``` **New** ``` Benchmark (aclCount) (resourceCount) Mode Cnt Score Error Units AclAuthorizerBenchmark.testAclsIterator 5 5000 avgt 15 0.666 ? 0.004 ms/op AclAuthorizerBenchmark.testAclsIterator 5 10000 avgt 15 1.427 ? 0.015 ms/op AclAuthorizerBenchmark.testAclsIterator 5 50000 avgt 15 21.410 ? 0.225 ms/op AclAuthorizerBenchmark.testAclsIterator 10 5000 avgt 15 1.230 ? 0.018 ms/op AclAuthorizerBenchmark.testAclsIterator 10 10000 avgt 15 4.303 ? 0.744 ms/op AclAuthorizerBenchmark.testAclsIterator 10 50000 avgt 15 36.724 ? 0.409 ms/op AclAuthorizerBenchmark.testAclsIterator 15 5000 avgt 15 2.433 ? 0.379 ms/op AclAuthorizerBenchmark.testAclsIterator 15 10000 avgt 15 9.818 ? 0.214 ms/op AclAuthorizerBenchmark.testAclsIterator 15 50000 avgt 15 52.886 ? 0.525 ms/op ``` Author: Manikumar Reddy <manikumar.reddy@gmail.com> Author: Lucas Bradstreet <lucas@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Lucas Bradstreet <lucas@confluent.io> Closes #8199 from omkreddy/KAFKA-9626
5 years ago
<Match>
<!-- Suppress warnings related to jmh generated code -->
<Package name="org.apache.kafka.jmh.acl.generated"/>
</Match>
<Match>
<!-- Suppress warnings related to jmh generated code -->
<Package name="org.apache.kafka.jmh.metadata.generated"/>
</Match>
</FindBugsFilter>