Pass through the root StateStore in the init method so the inner StateStore can register that object.
Author: tomdearman <tom.dearman@gmail.com>
Reviewers: Yasuhiro Matsuda
Closes#904 from tomdearman/KAFKA-3229
* We need to poll periodically even when all partitions are paused in order to respond to a possible rebalance promptly.
* There is a race condition when two (or more) threads try to clean up the same state directory. One of the thread fails with FileNotFoundException. Thus the new code simply catches it and ignore.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Gwen Shapira
Closes#893 from ymatsuda/hotfix
There are multi-byte characters In quickstart.html and security.html.
This PR will fix it.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Grant Henke
Closes#897 from sasakitoa/remove_multi_byte_character
This is the most of the KIP-42: Producer and consumer interceptor. (Except exposing CRC and record sizes to the interceptor, which is coming as a separate PR; tracked by KAFKA-3196).
This PR includes:
1. Add ProducerInterceptor interface and call its callbacks from appropriate places in Kafka Producer.
2. Add ConsumerInterceptor interface and call its callbacks from appropriate places in Kafka Consumer.
3. Add unit tests for interceptor changes
4. Add integration test for both mutable consumer and producer interceptors.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Jason Gustavson, Ismael Juma, Gwen Shapira
Closes#854 from apovzner/kip42
* During window store initialization, we have to open segments in the segment id order and update ```currentSegmentId```, otherwise cleanup won't work.
* ```getSegment()``` should not create a segment and clean up old segments if the segment id is greater than ```currentSegmentId```. Segment maintenance should be driven not by query but only by data insertion.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#891 from ymatsuda/hotfix2
Correct the text that said that you can't change the replication factor of a topic.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma
Closes#881 from wushujames/KAFKA-3216
Buffered records of change logs must be cleared upon reassignment of standby tasks.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#889 from ymatsuda/hotfix
Producers that are not closed auto-create topics in subsequent tests when Kafka server port is reused. Added missing close().
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#882 from rajinisivaram/KAFKA-3217
Inference sometimes fails for this case.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#885 from ijuma/use-explicit-type-in-acl-command
Added an example clarifying the correct way to use explicit offsets with commitSync().
Author: Adam Kunicki <adam@streamsets.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#850 from kunickiaj/KAFKA-3191
guozhangwang
A window store should open all existing segments. This is important for segment cleanup, and it also ensures that the first fetch() call returns the hits, the values in the search range. (previously, it missed the hits in fetch() immediately after initialization).
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#886 from ymatsuda/hotfix3
The range is inclusive according to KeyValueStore's java doc.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#883 from ymatsuda/minor
* RocksDBStore.putInternal should bypass logging.
* StoreChangeLogger should not call context.recordCollector() when nothing to log
* This is for standby tasks. In standby task, recordCollector() throws an exception. There should be nothing to log anyway.
* fixed ConcurrentModificationException in StreamThread
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#877 from ymatsuda/hotfix2
Provides a more actionable and descriptive error message.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ashish Singh <asingh@cloudera.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#847 from granthenke/broker-id-error
… is not specified correctly
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#876 from granthenke/rebalance-doc
Mirror maker doesn't commit offset with new consumer enabled when data volume is low. This is caused by infinite loop in ```receive()``` which would never jump out of loop if no data coming
Author: Tao Xiao <xiaotao183@gmail.com>
Reviewers: Ismael Juma, Jason Gustafson
Closes#821 from xiaotao183/KAFKA-3157
The fix itself is simple.
Some explanation on unit tests. Currently we the vast majority of unit test is running with uncompressed messages. I was initially thinking about run all the tests using compressed messages. But it seems uncompressed messages are necessary in a many test cases because we need the bytes sent and appended to the log to be predictable. In most of other cases, it does not matter whether the message is compressed or not, and compression will slow down the unit test. So I just added one method in the BaseConsumerTest to send compressed messages whenever we need it.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>
Closes#842 from becketqin/KAFKA-3179
workround partition ordering not preserved by the consumer group management.
guozhangwang
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#868 from ymatsuda/partitionOrder
The hack here is no longer necessary with up-to-date versions of Vagrant, vagrant-hostmanager, and vagrant-aws. What's more, the change in c8b60b63 caused a chain of infinite recursion on OSX, preventing bringup of VMs on a typical laptop.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#867 from granders/remove-vagrantfile-hack and squashes the following commits:
14f4395 [Geoff Anderson] Removed uneccessary references to version 1.5.0 of vagrant-hostmanager
8799afe [Geoff Anderson] Removed Vagrantfile hack which is no longer necessary with up-to-date versions of Vagrant, vagrant-hostmanager, and vagrant-aws
guozhangwang
Temporarily disabled state store access checking.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#864 from ymatsuda/fix_table_lookup
KAFKA-3198: Ticket Renewal Thread exits prematurely due to inverted comparison
The >= should be < since we are actually able to renew if the renewTill time is later than the current ticket expiration.
Author: Adam Kunicki <adam@streamsets.com>
Reviewers: Ismael Juma, Gwen Shapira
Closes#858 from kunickiaj/KAFKA-3198
Set default to 1 instead of 1024, this matches the existing doc and feels like a better default value. Have run the unit tests with the change.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson, Ismael Juma, Grant Henke, Guozhang Wang
Closes#832 from rajinisivaram/KAFKA-3170
ewencp ijuma if this looks good please merge when you can. Thanks.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#823 from enothereska/kafka-3068-alt
We need to close producer first before closing tasks to make sure all messages are acked and hence checkpoint offsets are updated before closing tasks and their state. It was re-ordered mistakenly before.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Yasuhiro Matsuda <yasuhiro@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#845 from guozhangwang/KStreamState
Changes include:
1) Move logging logic from MeteredXXXStore to internal stores, and leave WindowedStore API clean by removed all internalPut/Get functions.
2) Wrap common logging behavior of InMemory and LRUCache stores into one class.
3) Fix a bug for StoreChangeLogger where byte arrays are not comparable in HashSet by using a specified RawStoreChangeLogger.
4) Add a caching layer on top of RocksDBStore with object caching, it relies on the object's equals and hashCode function to be consistent with serdes.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Yasuhiro Matsuda <yasuhiro@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#826 from guozhangwang/K3060
When the disk (raid with caches dir) dies on a Kafka broker, typically the filesystem gets mounted into read-only mode, and hence when Kafka tries to read the disk, they'll get a FileNotFoundException with the read-only errno set (EROFS). However, as long as there is no produce request received, hence no writes attempted on the disks, Kafka will not exit on such FATAL error and keep on throwing exception : java.io.FileNotFoundException
In this case, the JVM should stop if the underlying file system goes in to Read only mode.
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Lin Dong, Gwen Shapira, Ismael Juma, Guozhang Wang
Closes#698 from MayureshGharat/KAFKA-1860
tomdearman
Yasuhiro Matsuda
Jason Gustafson
Sasaki Toru
Anna Povzner
Ashish Singh
James Cheng
Grant Henke
Rajini Sivaram
Ismael Juma
Adam Kunicki
Tao Xiao
Jiangjie Qin
Guozhang Wang
Gwen Shapira
Geoff Anderson
Eno Thereska
Mayuresh Gharat