From f9cdadfdcf64b95afe21590884594eb3b11043f8 Mon Sep 17 00:00:00 2001 From: Jon Lee Date: Sat, 21 Jul 2018 00:04:16 -0700 Subject: [PATCH] KAFKA-7177; Update 2.0 documentation to reflect changed quota behaviors by KIP-219 Updated the 2.0 document for changed quota behaviors. Author: Jon Lee Reviewers: Ismael Juma , Dong Lin Closes #5384 from jonlee2/KAFKA-7177 --- docs/design.html | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/design.html b/docs/design.html index 69d1941effd..bdc7e637ea9 100644 --- a/docs/design.html +++ b/docs/design.html @@ -610,10 +610,12 @@ having a fixed cluster wide bandwidth per client because that would require a mechanism to share client quota usage among all the brokers. This can be harder to get right than the quota implementation itself!

- How does a broker react when it detects a quota violation? In our solution, the broker does not return an error rather it attempts to slow down a client exceeding its quota. - It computes the amount of delay needed to bring a guilty client under its quota and delays the response for that time. This approach keeps the quota violation transparent to clients - (outside of client-side metrics). This also keeps them from having to implement any special backoff and retry behavior which can get tricky. In fact, bad client behavior (retry without backoff) - can exacerbate the very problem quotas are trying to solve. + How does a broker react when it detects a quota violation? In our solution, the broker first computes the amount of delay needed to bring the violating client under its quota + and returns a response with the delay immediately. In case of a fetch request, the response will not contain any data. Then, the broker mutes the channel to the client, + not to process requests from the client anymore, until the delay is over. Upon receiving a response with a non-zero delay duration, the Kafka client will also refrain from + sending further requests to the broker during the delay. Therefore, requests from a throttled client are effectively blocked from both sides. + Even with older client implementations that do not respect the delay response from the broker, the back pressure applied by the broker via muting its socket channel + can still handle the throttling of badly behaving clients. Those clients who sent further requests to the throttled channel will receive responses only after the delay is over.

Byte-rate and thread utilization are measured over multiple small windows (e.g. 30 windows of 1 second each) in order to detect and correct quota violations quickly. Typically, having large measurement windows