From 7df79e1508496e1b9350866c390de692434b1549 Mon Sep 17 00:00:00 2001 From: "Matthias J. Sax" Date: Thu, 23 May 2019 23:19:12 -0700 Subject: [PATCH] MINOR: add Kafka Streams upgrade notes for 2.3 release (#6758) Reviewers: A. Sophie Blee-Goldman , John Roesler , Bill Bejeck --- docs/streams/upgrade-guide.html | 56 +++++++++++++++++++++++++++++++-- docs/upgrade.html | 10 ++++++ 2 files changed, 63 insertions(+), 3 deletions(-) diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index e07b5a9acd8..f4c5d48c4cd 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -71,14 +71,64 @@

Streams API changes in 2.3.0

+

Version 2.3.0 adds the Suppress operator to the kafka-streams-scala Ktable API.

+ +

+ As of 2.3.0 Streams now offers an in-memory version of the window (KIP-428) + and the session (KIP-445) store, in addition to the persistent ones based on RocksDB. + The new public interfaces inMemoryWindowStore() and inMemorySessionStore() are added to Stores and provide the built-in in-memory window or session store. +

+ +

+ In 2.3.0 we have added default implementation to close() and configure() for Serializer, + Deserializer and Serde so that they can be implemented by lambda expression. + For more details please read KIP-331. +

+ +

+ To improve operator semantics, new store types are added that allow storing an additional timestamp per key-value pair or window. + Some DSL operators (for example KTables) are using those new stores. + Hence, you can now retrieve the last update timestamp via Interactive Queries if you specify + TimestampedKeyValueStoreType or TimestampedWindowStoreType as your QueryableStoreType. + While this change is mainly transparent, there are some corner cases that may require code changes: + Caution: If you receive an untyped store and use a cast, you might need to update your code to cast to the correct type. + Otherwise, you might get an exception similar to + java.lang.ClassCastException: class org.apache.kafka.streams.state.ValueAndTimestamp cannot be cast to class YOUR-VALUE-TYPE + upon getting a value from the store. + Additionally, TopologyTestDriver#getStateStore() only returns non-built-in stores and throws an exception if a built-in store is accessed. + For more details please read KIP-258. +

+ +

+ To improve type safety, a new operator KStream#flatTransformValues is added. + For more details please read KIP-313. +

+ +

+ Kafka Streams used to set the configuration parameter max.poll.interval.ms to Integer.MAX_VALUE. + This default value is removed and Kafka Streams uses the consumer default value now. + For more details please read KIP-442. +

+ +

+ Default configuration for repartition topic was changed: + The segment size for index files (segment.index.bytes) is no longer 50MB, but uses the cluster default. + Similarly, the configuration segment.ms in no longer 10 minutes, but uses the cluster default configuration. + Lastly, the retention period (retention.ms) is changed from Long.MAX_VALUE to -1 (infinite). + For more details please read KIP-443. +

+

- As of 2.3.0 Streams now offers an in-memory version of the window and the session store, in addition to the persistent ones based on RocksDB. The new public interfaces inMemoryWindowStore() and inMemorySessionStore() are added to Stores and provide the built-in in-memory window or session store. + To avoid memory leaks, RocksDBConfigSetter has a new close() method that is called on shutdown. + Users should implement this method to release any memory used by RocksDB config objects, by closing those objects. + For more details please read KIP-453.

- In 2.3.0 we have added default implementation to close() and configure() for Serializer, Deserializer and Serde so that they can be - implemented by lambda expression. For more details please read KIP-331. + RocksDB dependency was updated to version 5.18.3. + The new version allows to specify more RocksDB configurations, including WriteBufferManager that help to limit RocksDB off-head memory usage. + For more details please read KAFKA-8215.

Streams API changes in 2.2.0

diff --git a/docs/upgrade.html b/docs/upgrade.html index 1828ab78139..be9886cb990 100644 --- a/docs/upgrade.html +++ b/docs/upgrade.html @@ -19,6 +19,16 @@