Mirror of Apache Kafka
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

192 lines
0 B

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
ext {
versions = [:]
libs = [:]
// Available if -PscalaVersion is used. This is useful when we want to support a Scala version that has
// a higher minimum Java requirement than Kafka. This was previously the case for Scala 2.12 and Java 7.
availableScalaVersions = [ '2.12', '2.13' ]
}
// Add Scala version
def defaultScala212Version = '2.12.12'
def defaultScala213Version = '2.13.3'
if (hasProperty('scalaVersion')) {
if (scalaVersion == '2.12') {
versions["scala"] = defaultScala212Version
} else if (scalaVersion == '2.13') {
versions["scala"] = defaultScala213Version
} else {
versions["scala"] = scalaVersion
}
} else {
versions["scala"] = defaultScala212Version
}
/* Resolve base Scala version according to these patterns:
1. generally available Scala versions (such as: 2.12.y and 2.13.z) corresponding base versions will be: 2.12 and 2.13 (respectively)
2. pre-release Scala versions (i.e. milestone/rc, such as: 2.13.0-M5, 2.13.0-RC1, 2.14.0-M1, etc.) will have identical base versions;
rationale: pre-release Scala versions are not binary compatible with each other and that's the reason why libraries include the full
Scala release string in their name for pre-releases (see dependencies below with an artifact name suffix '_$versions.baseScala')
*/
if ( !versions.scala.contains('-') ) {
versions["baseScala"] = versions.scala.substring(0, versions.scala.lastIndexOf("."))
} else {
versions["baseScala"] = versions.scala
}
versions += [
activation: "1.1.1",
apacheda: "1.0.2",
apacheds: "2.0.0-M24",
argparse4j: "0.7.0",
bcpkix: "1.66",
checkstyle: "8.20",
commonsCli: "1.4",
MINOR: Update to Gradle 6.5 and tweak build jvm config (#8751) Gradle 6.5 includes a fix for https://github.com/gradle/gradle/pull/12866, which affects the performance of Scala compilation. I profiled the scalac build with async profiler and 54% of the time was on GC even after the Gradle upgrade (it was more than 60% before), so I switched to the throughput GC (GC latency is less important for batch builds) and it was reduced to 38%. I also centralized the jvm configuration in `build.gradle` and simplified it a bit by removing the minHeapSize configuration from the test tasks. On my desktop, the time to execute clean builds with no cached Gradle daemon was reduced from 127 seconds to 97 seconds. With a cached daemon, it was reduced from 120 seconds to 88 seconds. The performance regression when we upgraded to Gradle 6.x was 27 seconds with a cached daemon (https://github.com/apache/kafka/pull/7677#issuecomment-616271179), so it should be fixed now. Gradle 6.4 with no cached daemon: ``` BUILD SUCCESSFUL in 2m 7s 115 actionable tasks: 112 executed, 3 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.15s user 0.12s system 0% cpu 2:08.06 total ``` Gradle 6.4 with cached daemon: ``` BUILD SUCCESSFUL in 2m 0s 115 actionable tasks: 111 executed, 4 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 0.95s user 0.10s system 0% cpu 2:01.42 total ``` Gradle 6.5 with no cached daemon: ``` BUILD SUCCESSFUL in 1m 46s 115 actionable tasks: 111 executed, 4 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.27s user 0.12s system 1% cpu 1:47.71 total ``` Gradle 6.5 with cached daemon: ``` BUILD SUCCESSFUL in 1m 37s 115 actionable tasks: 111 executed, 4 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.02s user 0.10s system 1% cpu 1:38.31 total ``` This PR with no cached Gradle daemon: ``` BUILD SUCCESSFUL in 1m 37s 115 actionable tasks: 81 executed, 34 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.27s user 0.10s system 1% cpu 1:38.70 total ``` This PR with cached Gradle daemon: ``` BUILD SUCCESSFUL in 1m 28s 115 actionable tasks: 111 executed, 4 up-to-date ./gradlew clean compileScala compileJava compileTestScala compileTestJava 1.02s user 0.10s system 1% cpu 1:29.35 total ``` Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
5 years ago
gradle: "6.5",
gradleVersionsPlugin: "0.29.0",
grgit: "4.0.2",
httpclient: "4.5.12",
easymock: "4.2",
jackson: "2.10.2",
jacoco: "0.8.5",
jetty: "9.4.30.v20200611",
jersey: "2.31",
jmh: "1.23",
hamcrest: "2.2",
log4j: "1.2.17",
scalaLogging: "3.9.2",
jaxb: "2.3.0",
jaxrs: "2.1.1",
jfreechart: "1.0.0",
jopt: "5.0.4",
junit: "4.13",
kafka_0100: "0.10.0.1",
kafka_0101: "0.10.1.1",
kafka_0102: "0.10.2.2",
kafka_0110: "0.11.0.3",
kafka_10: "1.0.2",
kafka_11: "1.1.1",
kafka_20: "2.0.1",
kafka_21: "2.1.1",
kafka_22: "2.2.2",
kafka_23: "2.3.1",
kafka_24: "2.4.1",
kafka_25: "2.5.0",
lz4: "1.7.1",
mavenArtifact: "3.6.3",
metrics: "2.2.0",
mockito: "3.4.4",
netty: "4.1.51.Final",
owaspDepCheckPlugin: "5.3.2.1",
powermock: "2.0.7",
reflections: "0.9.12",
rocksDB: "5.18.4",
scalaCollectionCompat: "2.1.6",
scalafmt: "1.5.1",
scalaJava8Compat : "0.9.1",
MINOR: Make the build compile with Scala 2.13 (#6989) Scala 2.13 support was added to build via #5454. This PR adjusts the code so that it compiles with 2.11, 2.12 and 2.13. Changes: * Add `scala-collection-compat` dependency. * Import `scala.collection.Seq` in a number of places for consistent behavior between Scala 2.11, 2.12 and 2.13. * Remove wildcard imports that were causing the Java classes to have priority over the Scala ones, related Scala issue: https://github.com/scala/scala/pull/6589. * Replace parallel collection usage with `Future`. The former is no longer included by default in the standard library. * Replace val _: Unit workaround with one that is more concise and works with Scala 2.13 * Replace `filterKeys` with `filter` when we expect a `Map`. `filterKeys` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `mapValues` with `map` or add a `toMap` as an additional transformation when we expect a `Map`. `mapValues` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `breakOut` with `iterator` and `to`, `breakOut` was removed in Scala 2.13. * Replace to() with toMap, toIndexedSeq and toSet * Replace `mutable.Buffer.--` with `filterNot`. * ControlException is an abstract class in Scala 2.13. * Variable arguments can only receive arrays or immutable.Seq in Scala 2.13. * Use `Factory` instead of `CanBuildFrom` in DecodeJson. `CanBuildFrom` behaves a bit differently in Scala 2.13 and it's been deprecated. `Factory` has the behavior we need and it's available via the compat library. * Fix failing tests due to behavior change in Scala 2.13, "Map.values.map is not strict in Scala 2.13" (https://github.com/scala/bug/issues/11589). * Use Java collections instead of Scala ones in StreamResetter (a Java class). * Adjust CheckpointFile.write to take an `Iterable` instead of `Seq` to avoid unnecessary collection copies. * Fix DelayedElectLeader to use a Map instead of Set and avoid `to` call that doesn't work in Scala 2.13. * Use unordered map for mapping in SimpleAclAuthorizer, mapping of ordered maps require an `Ordering` in Scala 2.13 for safety reasons. * Adapt `ConsumerGroupCommand` to compile with Scala 2.13. * CoreUtils.min takes an `Iterable` instead of `TraversableOnce`, the latter does not exist in Scala 2.13. * Replace `Unit` with `()` in a couple places. Scala 2.13 is stricter when it expects a value instead of a type. * Fix bug in CustomQuotaCallbackTest where we did not necessarily set `partitionRatio` correctly, `forall` can terminate early. * Add a couple of spotbugs exclusions that are needed by code generated by Scala 2.13 * Remove unused variables, simplify some code and remove procedure syntax in a few places. * Remove unused `CoreUtils.JSONEscapeString`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, José Armando García Sancio <jsancio@users.noreply.github.com>
5 years ago
scalatest: "3.0.8",
scoverage: "1.4.1",
scoveragePlugin: "4.0.2",
shadowPlugin: "6.0.0",
slf4j: "1.7.30",
snappy: "1.1.7.6",
spotbugs: "4.0.6",
spotbugsPlugin: "4.4.4",
spotlessPlugin: "5.1.0",
testRetryPlugin: "1.1.6",
zinc: "1.3.5",
zookeeper: "3.5.8",
zstd: "1.4.5-6"
]
libs += [
activation: "javax.activation:activation:$versions.activation",
apacheda: "org.apache.directory.api:api-all:$versions.apacheda",
apachedsCoreApi: "org.apache.directory.server:apacheds-core-api:$versions.apacheds",
apachedsInterceptorKerberos: "org.apache.directory.server:apacheds-interceptor-kerberos:$versions.apacheds",
apachedsProtocolShared: "org.apache.directory.server:apacheds-protocol-shared:$versions.apacheds",
apachedsProtocolKerberos: "org.apache.directory.server:apacheds-protocol-kerberos:$versions.apacheds",
apachedsProtocolLdap: "org.apache.directory.server:apacheds-protocol-ldap:$versions.apacheds",
apachedsLdifPartition: "org.apache.directory.server:apacheds-ldif-partition:$versions.apacheds",
apachedsMavibotPartition: "org.apache.directory.server:apacheds-mavibot-partition:$versions.apacheds",
apachedsJdbmPartition: "org.apache.directory.server:apacheds-jdbm-partition:$versions.apacheds",
argparse4j: "net.sourceforge.argparse4j:argparse4j:$versions.argparse4j",
bcpkix: "org.bouncycastle:bcpkix-jdk15on:$versions.bcpkix",
commonsCli: "commons-cli:commons-cli:$versions.commonsCli",
easymock: "org.easymock:easymock:$versions.easymock",
jacksonAnnotations: "com.fasterxml.jackson.core:jackson-annotations:$versions.jackson",
jacksonDatabind: "com.fasterxml.jackson.core:jackson-databind:$versions.jackson",
jacksonDataformatCsv: "com.fasterxml.jackson.dataformat:jackson-dataformat-csv:$versions.jackson",
jacksonModuleScala: "com.fasterxml.jackson.module:jackson-module-scala_$versions.baseScala:$versions.jackson",
jacksonJDK8Datatypes: "com.fasterxml.jackson.datatype:jackson-datatype-jdk8:$versions.jackson",
jacksonJaxrsJsonProvider: "com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:$versions.jackson",
jaxbApi: "javax.xml.bind:jaxb-api:$versions.jaxb",
jaxrsApi: "javax.ws.rs:javax.ws.rs-api:$versions.jaxrs",
jettyServer: "org.eclipse.jetty:jetty-server:$versions.jetty",
jettyClient: "org.eclipse.jetty:jetty-client:$versions.jetty",
jettyServlet: "org.eclipse.jetty:jetty-servlet:$versions.jetty",
jettyServlets: "org.eclipse.jetty:jetty-servlets:$versions.jetty",
jerseyContainerServlet: "org.glassfish.jersey.containers:jersey-container-servlet:$versions.jersey",
jerseyHk2: "org.glassfish.jersey.inject:jersey-hk2:$versions.jersey",
jmhCore: "org.openjdk.jmh:jmh-core:$versions.jmh",
jmhCoreBenchmarks: "org.openjdk.jmh:jmh-core-benchmarks:$versions.jmh",
jmhGeneratorAnnProcess: "org.openjdk.jmh:jmh-generator-annprocess:$versions.jmh",
joptSimple: "net.sf.jopt-simple:jopt-simple:$versions.jopt",
junit: "junit:junit:$versions.junit",
hamcrest: "org.hamcrest:hamcrest:$versions.hamcrest",
kafkaStreams_0100: "org.apache.kafka:kafka-streams:$versions.kafka_0100",
kafkaStreams_0101: "org.apache.kafka:kafka-streams:$versions.kafka_0101",
kafkaStreams_0102: "org.apache.kafka:kafka-streams:$versions.kafka_0102",
kafkaStreams_0110: "org.apache.kafka:kafka-streams:$versions.kafka_0110",
kafkaStreams_10: "org.apache.kafka:kafka-streams:$versions.kafka_10",
kafkaStreams_11: "org.apache.kafka:kafka-streams:$versions.kafka_11",
kafkaStreams_20: "org.apache.kafka:kafka-streams:$versions.kafka_20",
kafkaStreams_21: "org.apache.kafka:kafka-streams:$versions.kafka_21",
kafkaStreams_22: "org.apache.kafka:kafka-streams:$versions.kafka_22",
kafkaStreams_23: "org.apache.kafka:kafka-streams:$versions.kafka_23",
kafkaStreams_24: "org.apache.kafka:kafka-streams:$versions.kafka_24",
kafkaStreams_25: "org.apache.kafka:kafka-streams:$versions.kafka_25",
log4j: "log4j:log4j:$versions.log4j",
lz4: "org.lz4:lz4-java:$versions.lz4",
metrics: "com.yammer.metrics:metrics-core:$versions.metrics",
mockitoCore: "org.mockito:mockito-core:$versions.mockito",
nettyHandler: "io.netty:netty-handler:$versions.netty",
nettyTransportNativeEpoll: "io.netty:netty-transport-native-epoll:$versions.netty",
powermockJunit4: "org.powermock:powermock-module-junit4:$versions.powermock",
powermockEasymock: "org.powermock:powermock-api-easymock:$versions.powermock",
reflections: "org.reflections:reflections:$versions.reflections",
rocksDBJni: "org.rocksdb:rocksdbjni:$versions.rocksDB",
MINOR: Make the build compile with Scala 2.13 (#6989) Scala 2.13 support was added to build via #5454. This PR adjusts the code so that it compiles with 2.11, 2.12 and 2.13. Changes: * Add `scala-collection-compat` dependency. * Import `scala.collection.Seq` in a number of places for consistent behavior between Scala 2.11, 2.12 and 2.13. * Remove wildcard imports that were causing the Java classes to have priority over the Scala ones, related Scala issue: https://github.com/scala/scala/pull/6589. * Replace parallel collection usage with `Future`. The former is no longer included by default in the standard library. * Replace val _: Unit workaround with one that is more concise and works with Scala 2.13 * Replace `filterKeys` with `filter` when we expect a `Map`. `filterKeys` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `mapValues` with `map` or add a `toMap` as an additional transformation when we expect a `Map`. `mapValues` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `breakOut` with `iterator` and `to`, `breakOut` was removed in Scala 2.13. * Replace to() with toMap, toIndexedSeq and toSet * Replace `mutable.Buffer.--` with `filterNot`. * ControlException is an abstract class in Scala 2.13. * Variable arguments can only receive arrays or immutable.Seq in Scala 2.13. * Use `Factory` instead of `CanBuildFrom` in DecodeJson. `CanBuildFrom` behaves a bit differently in Scala 2.13 and it's been deprecated. `Factory` has the behavior we need and it's available via the compat library. * Fix failing tests due to behavior change in Scala 2.13, "Map.values.map is not strict in Scala 2.13" (https://github.com/scala/bug/issues/11589). * Use Java collections instead of Scala ones in StreamResetter (a Java class). * Adjust CheckpointFile.write to take an `Iterable` instead of `Seq` to avoid unnecessary collection copies. * Fix DelayedElectLeader to use a Map instead of Set and avoid `to` call that doesn't work in Scala 2.13. * Use unordered map for mapping in SimpleAclAuthorizer, mapping of ordered maps require an `Ordering` in Scala 2.13 for safety reasons. * Adapt `ConsumerGroupCommand` to compile with Scala 2.13. * CoreUtils.min takes an `Iterable` instead of `TraversableOnce`, the latter does not exist in Scala 2.13. * Replace `Unit` with `()` in a couple places. Scala 2.13 is stricter when it expects a value instead of a type. * Fix bug in CustomQuotaCallbackTest where we did not necessarily set `partitionRatio` correctly, `forall` can terminate early. * Add a couple of spotbugs exclusions that are needed by code generated by Scala 2.13 * Remove unused variables, simplify some code and remove procedure syntax in a few places. * Remove unused `CoreUtils.JSONEscapeString`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, José Armando García Sancio <jsancio@users.noreply.github.com>
5 years ago
scalaCollectionCompat: "org.scala-lang.modules:scala-collection-compat_$versions.baseScala:$versions.scalaCollectionCompat",
scalaJava8Compat: "org.scala-lang.modules:scala-java8-compat_$versions.baseScala:$versions.scalaJava8Compat",
scalaLibrary: "org.scala-lang:scala-library:$versions.scala",
scalaLogging: "com.typesafe.scala-logging:scala-logging_$versions.baseScala:$versions.scalaLogging",
scalaReflect: "org.scala-lang:scala-reflect:$versions.scala",
scalatest: "org.scalatest:scalatest_$versions.baseScala:$versions.scalatest",
slf4jApi: "org.slf4j:slf4j-api:$versions.slf4j",
slf4jlog4j: "org.slf4j:slf4j-log4j12:$versions.slf4j",
snappy: "org.xerial.snappy:snappy-java:$versions.snappy",
zookeeper: "org.apache.zookeeper:zookeeper:$versions.zookeeper",
jfreechart: "jfreechart:jfreechart:$versions.jfreechart",
mavenArtifact: "org.apache.maven:maven-artifact:$versions.mavenArtifact",
zstd: "com.github.luben:zstd-jni:$versions.zstd",
httpclient: "org.apache.httpcomponents:httpclient:$versions.httpclient"
]