From 6a3382c9d397619a7f27766fd28aff41d85ed4ff Mon Sep 17 00:00:00 2001 From: Guozhang Wang Date: Fri, 12 Oct 2018 14:48:28 -0700 Subject: [PATCH] MINOR: Streams Update for KIP-330 / KIP-356 (#5794) Reviewers: Matthias J. Sax , Bill Bejeck --- docs/js/templateData.js | 6 +-- docs/streams/upgrade-guide.html | 65 +++++++++++++++++---------------- 2 files changed, 37 insertions(+), 34 deletions(-) diff --git a/docs/js/templateData.js b/docs/js/templateData.js index 2bf62ac0e7d..eae0b30a067 100644 --- a/docs/js/templateData.js +++ b/docs/js/templateData.js @@ -17,8 +17,8 @@ limitations under the License. // Define variables for doc templates var context={ - "version": "20", - "dotVersion": "2.0", - "fullDotVersion": "2.0.0", + "version": "21", + "dotVersion": "2.1", + "fullDotVersion": "2.1.0", "scalaVersion": "2.11" }; diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index 4bece4ae5be..ddf6e4a1ec9 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -34,23 +34,22 @@

- Upgrading from any older version to 2.0.0 is possible: (1) you need to make sure to update you code and config accordingly, because there are some minor non-compatible API changes since older - releases (the code changes are expected to be minimal, please see below for the details), - (2) upgrading to 2.0.0 in the online mode requires two rolling bounces. - For (2), in the first rolling bounce phase users need to set config upgrade.from="older version" (possible values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", and "1.1") + Upgrading from any older version to {{fullDotVersion}} is possible: (1) if you are upgrading from 2.0.x to {{fullDotVersion}} then a single rolling bounce is needed to swap in the new jar, + (2) if you are upgrading from older versions than 2.0.x in the online mode, you would need two rolling bounces where + the first rolling bounce phase you need to set config upgrade.from="older version" (possible values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", and "1.1") (cf. KIP-268):

    -
  • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to the version from which it is being upgrade to new version 2.0.0
    • +
  • prepare your application instances for a rolling bounce and make sure that config upgrade.from is set to the version from which it is being upgrade.
  • bounce each instance of your application once
    • -
  • prepare your newly deployed 2.0.0 application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.mode
    • +
  • prepare your newly deployed {{fullDotVersion}} application instances for a second round of rolling bounces; make sure to remove the value for config upgrade.mode
  • bounce each instance of your application once more to complete the upgrade
-

As an alternative, an offline upgrade is also possible. Upgrading from any versions as old as 0.10.0.x to 2.0.0 in offline mode require the following steps:

+

As an alternative, an offline upgrade is also possible. Upgrading from any versions as old as 0.10.0.x to {{fullDotVersion}} in offline mode require the following steps:

  • stop all old (e.g., 0.10.0.x) application instances
  • update your code and swap old code and jar file with new code and new jar file
    • -
  • restart all new (2.0.0) application instances
    • +
  • restart all new ({{fullDotVersion}}) application instances

@@ -65,32 +64,10 @@ accidentally: we still reuse the source topic as the changelog topic for restoring, but will also create a separate changelog topic to append the update records from source topic to. In the 2.0 release, we have fixed this issue and now users can choose whether or not to reuse the source topic based on the StreamsConfig#TOPOLOGY_OPTIMIZATION: if you are upgrading from the old KStreamBuilder class and hence you need to change your code to use the new StreamsBuilder, you should set this config value to StreamsConfig#OPTIMIZE to continue reusing the source topic; if you are upgrading from 1.0 or 1.1 where you are already using StreamsBuilder and hence have already - created a separate changelog topic, you should set this config value to StreamsConfig#NO_OPTIMIZATION when upgrading to 2.0.0 in order to use that changelog topic for restoring the state store. + created a separate changelog topic, you should set this config value to StreamsConfig#NO_OPTIMIZATION when upgrading to {{fullDotVersion}} in order to use that changelog topic for restoring the state store. More details about the new config StreamsConfig#TOPOLOGY_OPTIMIZATION can be found in KIP-295.

-

- In 2.0.0 we have added a few new APIs on the ReadOnlyWindowStore interface (for details please read Streams API changes below). - If you have customized window store implementations that extends the ReadOnlyWindowStore interface you need to make code changes. -

- -

- In addition, if you using Java 8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguities. - Hot-swapping the jar-file only might not work for this case. - See below a complete list of 2.0.0 - API and semantic changes that allow you to advance your application and/or simplify your code base. -

- -

- We moved Consumed interface from org.apache.kafka.streams to org.apache.kafka.streams.kstream - as it was mistakenly placed in the previous release. If your code has already used it there is a simple one-liner change needed in your import statement. -

- -

- We have also removed some public APIs that are deprecated prior to 1.0.x in 2.0.0. - See below for a detailed list of removed APIs. -

-

Streams API changes in 2.1.0

We updated TopologyDescription API to allow for better runtime checking. @@ -108,11 +85,37 @@ they can override this config to a larger value.

+

+ We've added the missing SessionBytesStoreSupplier#retentionPeriod() to be consistent with the WindowBytesStoreSupplier which allows users to get the specified retention period for session-windowed stores. + We've also added the missing StoreBuilder#withCachingDisabled() to allow users to turn off caching for their customized stores. +

+

We added a new serde for UUIDs (Serdes.UUIDSerde) that you can use via Serdes.UUID() (cf. KIP-206).

Streams API changes in 2.0.0

+

+ In 2.0.0 we have added a few new APIs on the ReadOnlyWindowStore interface (for details please read Streams API changes below). + If you have customized window store implementations that extends the ReadOnlyWindowStore interface you need to make code changes. +

+ +

+ In addition, if you using Java 8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguities. + Hot-swapping the jar-file only might not work for this case. + See below a complete list of 2.0.0 + API and semantic changes that allow you to advance your application and/or simplify your code base. +

+ +

+ We moved Consumed interface from org.apache.kafka.streams to org.apache.kafka.streams.kstream + as it was mistakenly placed in the previous release. If your code has already used it there is a simple one-liner change needed in your import statement. +

+ +

+ We have also removed some public APIs that are deprecated prior to 1.0.x in 2.0.0. + See below for a detailed list of removed APIs. +

We have removed the skippedDueToDeserializationError-rate and skippedDueToDeserializationError-total metrics. Deserialization errors, and all other causes of record skipping, are now accounted for in the pre-existing metrics