KAFKA-7524: Recommend Scala 2.12 and use it for development (#5530)

Scala 2.12 has better support for newer Java versions and includes additional
compiler warnings that are helpful during development. In addition, Scala 2.11
hasn't been supported by the Scala community for a long time, the soon to be
released Spark 2.4.0 will finally support Scala 2.12 (this was the main reason
preventing many from upgrading to Scala 2.12) and Scala 2.13 is at the RC stage.
It's time to start recommending the Scala 2.12 build as we prepare support for
Scala 2.13 and start thinking about removing support for Scala 2.11.

In the meantime, Jenkins will continue to build all supported Scala versions (including
Scala 2.11) so the PR and trunk jobs will fail if people accidentally use methods
introduced in Scala 2.12.

Reviewers: Ewen Cheslack-Postava <me@ewencp.org>

README.md

@@ -6,7 +6,9 @@ You need to have [Gradle](http://www.gradle.org/installation) and [Java](http://
 
 Kafka requires Gradle 4.7 or higher.
 
-Java 8 should be used for building in order to support both Java 8 and Java 10 at runtime.
+Java 8 should be used for building in order to support both Java 8 and Java 11 at runtime.
+
+Scala 2.12 is used by default, see below for how to use a different Scala version or all of the supported Scala versions.
 
 ### First bootstrap and download the wrapper ###
     cd kafka_source_dir
@@ -64,26 +66,33 @@ Generate coverage for a single module, i.e.:
     ./gradlew clients:reportCoverage
     
 ### Building a binary release gzipped tar ball ###
-    ./gradlew clean
-    ./gradlew releaseTarGz
+    ./gradlew clean releaseTarGz
 
 The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run:
 
-    ./gradlew releaseTarGz -x signArchives
+    ./gradlew clean releaseTarGz -x signArchives
 
 The release file can be found inside `./core/build/distributions/`.
 
 ### Cleaning the build ###
     ./gradlew clean
 
-### Running a task on a particular version of Scala (either 2.11.x or 2.12.x) ###
-*Note that if building the jars with a version other than 2.11.12, you need to set the `SCALA_VERSION` variable or change it in `bin/kafka-run-class.sh` to run the quick start.*
+### Running a task with a particular version of Scala (either 2.11.x or 2.12.x) ###
+*Note that if building the jars with a version other than 2.12.x, you need to set the `SCALA_VERSION` variable or change it in `bin/kafka-run-class.sh` to run the quick start.*
+
+You can pass either the major version (eg 2.12) or the full version (eg 2.12.7):
+
+    ./gradlew -PscalaVersion=2.12 jar
+    ./gradlew -PscalaVersion=2.12 test
+    ./gradlew -PscalaVersion=2.12 releaseTarGz
 
-You can pass either the major version (eg 2.11) or the full version (eg 2.11.12):
+### Running a task with all scala versions ###
 
-    ./gradlew -PscalaVersion=2.11 jar
-    ./gradlew -PscalaVersion=2.11 test
-    ./gradlew -PscalaVersion=2.11 releaseTarGz
+Append `All` to the task name:
+
+    ./gradlew testAll
+    ./gradlew jarAll
+    ./gradlew releaseTarGzAll
 
 ### Running a task for a specific project ###
 This is for `core`, `examples` and `clients`
@@ -104,15 +113,6 @@ The `eclipse` task has been configured to use `${project_dir}/build_eclipse` as
 build directory (`${project_dir}/bin`) clashes with Kafka's scripts directory and we don't use Gradle's build directory
 to avoid known issues with this configuration.
 
-### Building the jar for all scala versions and for all projects ###
-    ./gradlew jarAll
-
-### Running unit/integration tests for all scala versions and for all projects ###
-    ./gradlew testAll
-
-### Building a binary release gzipped tar ball for all scala versions ###
-    ./gradlew releaseTarGzAll
-
 ### Publishing the jar for all version of Scala and for all projects to maven ###
     ./gradlew uploadArchivesAll
 

bin/kafka-run-class.sh

@@ -48,7 +48,7 @@ should_include_file() {
 base_dir=$(dirname $0)/..
 
 if [ -z "$SCALA_VERSION" ]; then
-  SCALA_VERSION=2.11.12
+  SCALA_VERSION=2.12.7
 fi
 
 if [ -z "$SCALA_BINARY_VERSION" ]; then

bin/windows/kafka-run-class.bat

@@ -27,7 +27,7 @@ set BASE_DIR=%CD%
 popd
 
 IF ["%SCALA_VERSION%"] EQU [""] (
-  set SCALA_VERSION=2.11.12
+  set SCALA_VERSION=2.12.7
 )
 
 IF ["%SCALA_BINARY_VERSION%"] EQU [""] (

docs/api.html

@@ -81,12 +81,12 @@
 	<p>
 	When using Scala you may optionally include the <code>kafka-streams-scala</code> library.  Additional documentation on using the Kafka Streams DSL for Scala is available <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html#scala-dsl">in the developer guide</a>.
 	<p>
-	To use Kafka Streams DSL for Scala for Scala 2.11 you can use the following maven dependency:
+	To use Kafka Streams DSL for Scala for Scala {{scalaVersion}} you can use the following maven dependency:
 
 	<pre class="brush: xml;">
 		&lt;dependency&gt;
 			&lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
-			&lt;artifactId&gt;kafka-streams-scala_2.11&lt;/artifactId&gt;
+			&lt;artifactId&gt;kafka-streams-scala_{{scalaVersion}}&lt;/artifactId&gt;
 			&lt;version&gt;{{fullDotVersion}}&lt;/version&gt;
 		&lt;/dependency&gt;
 	</pre>

docs/js/templateData.js

@@ -20,5 +20,5 @@ var context={
     "version": "21",
     "dotVersion": "2.1",
     "fullDotVersion": "2.1.0",
-    "scalaVersion": "2.11"
+    "scalaVersion": "2.12"
 };

docs/quickstart.html

@@ -25,11 +25,11 @@ Since Kafka console scripts are different for Unix-based and Windows platforms,
 
 <h4><a id="quickstart_download" href="#quickstart_download">Step 1: Download the code</a></h4>
 
-<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/{{fullDotVersion}}/kafka_2.11-{{fullDotVersion}}.tgz" title="Kafka downloads">Download</a> the {{fullDotVersion}} release and un-tar it.
+<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/{{fullDotVersion}}/kafka_{{scalaVersion}}-{{fullDotVersion}}.tgz" title="Kafka downloads">Download</a> the {{fullDotVersion}} release and un-tar it.
 
 <pre class="brush: bash;">
-&gt; tar -xzf kafka_2.11-{{fullDotVersion}}.tgz
-&gt; cd kafka_2.11-{{fullDotVersion}}
+&gt; tar -xzf kafka_{{scalaVersion}}-{{fullDotVersion}}.tgz
+&gt; cd kafka_{{scalaVersion}}-{{fullDotVersion}}
 </pre>
 
 <h4><a id="quickstart_startserver" href="#quickstart_startserver">Step 2: Start the server</a></h4>

docs/streams/developer-guide/dsl-api.html

@@ -3414,11 +3414,11 @@ groupedTable
               <li><code class="docutils literal"><span class="pre">org.apache.kafka.streams.scala.ImplicitConversions</span></code>: Module that brings into scope the implicit conversions between the Scala and Java classes.</li>
               <li><code class="docutils literal"><span class="pre">org.apache.kafka.streams.scala.Serdes</span></code>: Module that contains all primitive SerDes that can be imported as implicits and a helper to create custom SerDes.</li>
             </ul>
-            <p>The library is cross-built with Scala 2.11 and 2.12.  To reference the library compiled against Scala 2.11 include the following in your maven <code>pom.xml</code> add the following:</p>
+            <p>The library is cross-built with Scala 2.11 and 2.12.  To reference the library compiled against Scala {{scalaVersion}} include the following in your maven <code>pom.xml</code> add the following:</p>
             <pre class="brush: xml;">
               &lt;dependency&gt;
                 &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
-                &lt;artifactId&gt;kafka-streams-scala_2.11&lt;/artifactId&gt;
+                &lt;artifactId&gt;kafka-streams-scala_{{scalaVersion}}&lt;/artifactId&gt;
                 &lt;version&gt;{{fullDotVersion}}&lt;/version&gt;
               &lt;/dependency&gt;
             </pre>

docs/streams/developer-guide/write-streams.html

@@ -101,10 +101,10 @@
     <artifactId>kafka-clients</artifactId>
     <version>{{fullDotVersion}}</version>
 </dependency>
-<!-- Optionally include Kafka Streams DSL for Scala for Scala 2.11 -->
+<!-- Optionally include Kafka Streams DSL for Scala for Scala {{scalaVersion}} -->
 <dependency>
     <groupId>org.apache.kafka</groupId>
-    <artifactId>kafka-streams-scala_2.11</artifactId>
+    <artifactId>kafka-streams-scala_{{scalaVersion}}</artifactId>
     <version>{{fullDotVersion}}</version>
 </dependency>
           </pre>

gradle.properties

@@ -20,6 +20,6 @@ group=org.apache.kafka
 #  - tests/kafkatest/version.py (variable DEV_VERSION)
 #  - kafka-merge-pr.py.
 version=2.2.0-SNAPSHOT
-scalaVersion=2.11.12
+scalaVersion=2.12.7
 task=build
 org.gradle.jvmargs=-Xmx1024m -Xss2m

release.py

@@ -245,7 +245,7 @@ def command_stage_docs():
 
     cmd("Building docs", "./gradlew -Pversion=%s clean releaseTarGzAll aggregatedJavadoc" % gradle_version_override, cwd=REPO_HOME, env=jdk8_env)
 
-    docs_tar = os.path.join(REPO_HOME, 'core', 'build', 'distributions', 'kafka_2.11-%s-site-docs.tgz' % gradle_version_override)
+    docs_tar = os.path.join(REPO_HOME, 'core', 'build', 'distributions', 'kafka_2.12-%s-site-docs.tgz' % gradle_version_override)
 
     versioned_docs_path = os.path.join(kafka_site_repo_path, docs_version(version))
     if not os.path.exists(versioned_docs_path):
@@ -641,8 +641,8 @@ Now you should sanity check it before proceeding. All subsequent steps start mak
 Some suggested steps:
 
  * Grab the source archive and make sure it compiles: http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka-%(release_version)s-src.tgz
- * Grab one of the binary distros and run the quickstarts against them: http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka_2.11-%(release_version)s.tgz
- * Extract and verify one of the site docs jars: http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka_2.11-%(release_version)s-site-docs.tgz
+ * Grab one of the binary distros and run the quickstarts against them: http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka_2.12-%(release_version)s.tgz
+ * Extract and verify one of the site docs jars: http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka_2.12-%(release_version)s-site-docs.tgz
  * Build a sample against jars in the staging repo: (TODO: Can we get a temporary URL before "closing" the staged artifacts?)
  * Validate GPG signatures on at least one file:
       wget http://home.apache.org/~%(apache_id)s/kafka-%(rc_tag)s/kafka-%(release_version)s-src.tgz &&