Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>, Ashish Singh <asingh@cloudera.com>, Jun Rao <junrao@gmail.com>
Closes#528 from ijuma/kafka-2831-consumer-group-command-zookeeper-new-consumer
guozhangwang
An optimization which may reduce unnecessary poll for standby tasks.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#535 from ymatsuda/remove_empty_standby_task
guozhangwang
* added a new config param "num.standby.replicas" (the default value is 0).
* added a new abstract class AbstractTask
* added StandbyTask as a subclass of AbstractTask
* modified StreamTask to a subclass of AbstractTask
* StreamThread
* standby tasks are created by calling StreamThread.addStandbyTask() from onPartitionsAssigned()
* standby tasks are destroyed by calling StreamThread.removeStandbyTasks() from onPartitionRevoked()
* In addStandbyTasks(), change log partitions are assigned to restoreConsumer.
* In removeStandByTasks(), change log partitions are removed from restoreConsumer.
* StreamThread polls change log records using restoreConsumer in the runLoop with timeout=0.
* If records are returned, StreamThread calls StandbyTask.update and pass records to each standby tasks.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#526 from ymatsuda/standby_task
Log warning message before truncating log in order to
display right offset value for the truncated log.
Author: Francois Visconte <f.visconte@criteo.com>
Reviewers: Dong Lin <lindong28@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#287 from dopuskh3/KAFKA-2624
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#519 from hachikuji/KAFKA-2821
Removed support for BLOCK_ON_BUFFER_FULL_CONFIG (block.on.buffer.full)
Removed support for METADATA_FETCH_TIMEOUT_CONFIG
Removed support for TIMEOUT_CONFIG (aka timeout.ms)
Added support for MAX_BLOCK_MS_CONFIG
Added support for REQUEST_TIMEOUT_MS_CONFIG
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#525 from benstopford/KAFKA-2791
This avoids spurious log warning messages. Also tweak log message
if wrapResult.getStatus != CLOSED.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jun Rao <junrao@gmail.com>
Closes#511 from ijuma/kafka-2817-unconnected-ssl-transport-layer-close
Increased timeout in downstream consumer doing validation step. This addresses a transient failure case in mirror maker tests with mirror maker failover.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Gwen Shapira
Closes#521 from granders/minor-mm-transient-failure
…tent group, it used to throw IllegalArgumentException
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#515 from SinghAsDev/KAFKA-2822
Gradle does not handle subprojects with the same name (top-level tools vs
connect/tools) properly, making the dependency impossible to express correctly
since we need to move the ThroughputThrottler class into the top level tools
project. Moving the current set of tools into the runtime jar works fine since
they are only used for system tests at the moment.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira
Closes#512 from ewencp/kafka-2807-redux
Added PASSWORD_STRING in ConfigDef that returns "[hidden]" when method toString is invoked.
Author: Jakub Nowak <jakub.nowak94@interia.pl>
Reviewers: Ismael Juma, Gwen Shapira, Jun Rao
Closes#371 from Mszak/ssl-password-protection
Fails when order of elements is incorrect
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Yasuhiro Matsuda
Closes#510 from granthenke/streams-test
`Selector.poll` no longer throws it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#506 from ijuma/poll-no-longer-throws-invalid-receive-exception
Patched Selector.poll() to close the connection on any exception.
Author: Jun Rao <junrao@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Gwen Shapira <cshapi@gmail.com>
Closes#501 from junrao/KAFKA-2813
In system tests zookeeper service, it is overkill and space-intensive to collect zookeeper data logs by default. This minor patch turns off default collection.
Author: Geoff Anderson <geoff@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#504 from granders/minor-zk-change-log-collect
Removed the check for expiring only those batches whose metadata is unavailable. Now the batches will be expired irrespective of whether the leader is available or not, as soon as it reaches the requestimeout threshold.
Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#503 from MayureshGharat/kafka-2805
guozhangwang
When the rebalance happens each consumer reports the following information to the coordinator.
* Client UUID (a unique id assigned to an instance of KafkaStreaming)
* Task ids of previously running tasks
* Task ids of valid local states on the client's state directory
TaskAssignor does the following
* Assign a task to a client which was running it previously. If there is no such client, assign a task to a client which has its valid local state.
* Try to balance the load among stream threads.
* A client may have more than one stream threads. The assignor tries to assign tasks to a client proportionally to the number of threads.
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Reviewers: Guozhang Wang
Closes#497 from ymatsuda/task_assignment
Process any remaining data in the network read buffer in `SslTransportLayer` when `read()` is invoked. On handshake completion, there could be application data ready to be processed that was read into `netReadBuffer` during handshake processing. `read()` is already invoked from `Selector` after handshake completion, but data already read into the `netReadBuffer` was not being processed. This PR adds a check for remaining data and continues with processing data if data is available.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#493 from rajinisivaram/KAFKA-2801
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ben Stopford, Geoff Anderson, Guozhang Wang
Closes#432 from ewencp/kafka-2752-copycat-clean-bounce-test
-- Refactored multi-consumer integration group assignment validation tests for round-robin assignment
-- Added multi-consumer integration tests for session timeout expiration:
1. When a consumer stops polling
2. When a consumer calls close()
-- Fixes to issues found with session timeout expiration tests woth help from Jason Gustafson: Try to avoid SendFailedException exception by cancelling the scheduled tasks and ensuring metadata update before sending group leave requests + send leave group request with retries.
Author: Anna Povzner <anna@confluent.io>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#472 from apovzner/cpkafka-81
Ismael Juma
Yasuhiro Matsuda
Francois Visconte
Jun Rao
Parth Brahmbhatt
Guozhang Wang
Ashish Singh
Jason Gustafson
Ben Stopford
Sriharsha Chintalapani
Ewen Cheslack-Postava
Geoff Anderson
Jakub Nowak
Grant Henke
Mayuresh Gharat
Gwen Shapira
Rajini Sivaram
Anna Povzner