You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
196 lines
13 KiB
196 lines
13 KiB
<!-- |
|
Licensed to the Apache Software Foundation (ASF) under one or more |
|
contributor license agreements. See the NOTICE file distributed with |
|
this work for additional information regarding copyright ownership. |
|
The ASF licenses this file to You under the Apache License, Version 2.0 |
|
(the "License"); you may not use this file except in compliance with |
|
the License. You may obtain a copy of the License at |
|
|
|
http://www.apache.org/licenses/LICENSE-2.0 |
|
|
|
Unless required by applicable law or agreed to in writing, software |
|
distributed under the License is distributed on an "AS IS" BASIS, |
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|
See the License for the specific language governing permissions and |
|
limitations under the License. |
|
--> |
|
|
|
<script><!--#include virtual="../../js/templateData.js" --></script> |
|
|
|
<script id="content-template" type="text/x-handlebars-template"> |
|
<!-- h1>Developer Guide for Kafka Streams</h1 --> |
|
<div class="sub-nav-sticky"> |
|
<div class="sticky-top"> |
|
<!-- div style="height:35px"> |
|
<a href="/{{version}}/documentation/streams/">Introduction</a> |
|
<a class="active-menu-item" href="/{{version}}/documentation/streams/developer-guide">Developer Guide</a> |
|
<a href="/{{version}}/documentation/streams/core-concepts">Concepts</a> |
|
<a href="/{{version}}/documentation/streams/quickstart">Run Demo App</a> |
|
<a href="/{{version}}/documentation/streams/tutorial">Tutorial: Write App</a> |
|
</div --> |
|
</div> |
|
</div> |
|
|
|
<div class="section" id="application-reset-tool"> |
|
<span id="streams-developer-guide-app-reset"></span><h1>Application Reset Tool<a class="headerlink" href="#application-reset-tool" title="Permalink to this headline"></a></h1> |
|
<p>You can reset an application and force it to reprocess its data from scratch by using the application reset tool. |
|
This can be useful for development and testing, or when fixing bugs.</p> |
|
<p>The application reset tool handles the Kafka Streams <a class="reference internal" href="manage-topics.html#streams-developer-guide-topics-user"><span class="std std-ref">user topics</span></a> (input, |
|
output, and intermediate topics) and <a class="reference internal" href="manage-topics.html#streams-developer-guide-topics-internal"><span class="std std-ref">internal topics</span></a> differently |
|
when resetting the application.</p> |
|
<p>Here’s what the application reset tool does for each topic type:</p> |
|
<ul class="simple"> |
|
<li>Input topics: Reset offsets to specified position (by default to the beginning of the topic).</li> |
|
<li>Intermediate topics: Skip to the end of the topic, i.e., set the application’s committed consumer offsets for all partitions to each partition’s <code class="docutils literal"><span class="pre">logSize</span></code> (for consumer group <code class="docutils literal"><span class="pre">application.id</span></code>).</li> |
|
<li>Internal topics: Delete the internal topic (this automatically deletes any committed offsets).</li> |
|
</ul> |
|
<p>The application reset tool does not:</p> |
|
<ul class="simple"> |
|
<li>Reset output topics of an application. If any output (or intermediate) topics are consumed by downstream |
|
applications, it is your responsibility to adjust those downstream applications as appropriate when you reset the |
|
upstream application.</li> |
|
<li>Reset the local environment of your application instances. It is your responsibility to delete the local |
|
state on any machine on which an application instance was run. See the instructions in section |
|
<a class="reference internal" href="#streams-developer-guide-reset-local-environment"><span class="std std-ref">Step 2: Reset the local environments of your application instances</span></a> on how to do this.</li> |
|
</ul> |
|
<dl class="docutils"> |
|
<dt>Prerequisites</dt> |
|
<dd><ul class="first last"> |
|
<li><p class="first">All instances of your application must be stopped. Otherwise, the application may enter an invalid state, crash, or produce incorrect results. You can verify whether the consumer group with ID <code class="docutils literal"><span class="pre">application.id</span></code> is still active by using <code class="docutils literal"><span class="pre">bin/kafka-consumer-groups</span></code>.</p> |
|
</li> |
|
<li><p class="first">Use this tool with care and double-check its parameters: If you provide wrong parameter values (e.g., typos in <code class="docutils literal"><span class="pre">application.id</span></code>) or specify parameters inconsistently (e.g., specify the wrong input topics for the application), this tool might invalidate the application’s state or even impact other applications, consumer groups, or your Kafka topics.</p> |
|
</li> |
|
<li><p class="first">You should manually delete and re-create any intermediate topics before running the application reset tool. This will free up disk space in Kafka brokers.</p> |
|
</li> |
|
<li><p class="first">You should delete and recreate intermediate topics before running the application reset tool, unless the following applies:</p> |
|
<blockquote> |
|
<div><ul class="simple"> |
|
<li>You have external downstream consumers for the application’s intermediate topics.</li> |
|
<li>You are in a development environment where manually deleting and re-creating intermediate topics is unnecessary.</li> |
|
</ul> |
|
</div></blockquote> |
|
</li> |
|
</ul> |
|
</dd> |
|
</dl> |
|
<div class="section" id="step-1-run-the-application-reset-tool"> |
|
<h2>Step 1: Run the application reset tool<a class="headerlink" href="#step-1-run-the-application-reset-tool" title="Permalink to this headline"></a></h2> |
|
<p>Invoke the application reset tool from the command line</p> |
|
<div class="highlight-bash"><div class="highlight"><pre><span></span><path-to-kafka>/bin/kafka-streams-application-reset |
|
</pre></div> |
|
</div> |
|
<p>The tool accepts the following parameters:</p> |
|
<div class="highlight-bash"><div class="highlight"><pre><span></span>Option <span class="o">(</span>* <span class="o">=</span> required<span class="o">)</span> Description |
|
--------------------- ----------- |
|
* --application-id <String: id> The Kafka Streams application ID |
|
<span class="o">(</span>application.id<span class="o">)</span>. |
|
--bootstrap-servers <String: urls> Comma-separated list of broker urls with |
|
format: HOST1:PORT1,HOST2:PORT2 |
|
<span class="o">(</span>default: localhost:9092<span class="o">)</span> |
|
--by-duration <String: urls> Reset offsets to offset by duration from |
|
current timestamp. Format: '<span>PnDTnHnMnS</span>' |
|
--config-file <String: file name> Property file containing configs to be |
|
passed to admin clients and embedded |
|
consumer. |
|
--dry-run Display the actions that would be |
|
performed without executing the reset |
|
commands. |
|
--from-file <String: urls> Reset offsets to values defined in CSV |
|
file. |
|
--input-topics <String: list> Comma-separated list of user input |
|
topics. For these topics, the tool will |
|
reset the offset to the earliest |
|
available offset. |
|
--intermediate-topics <String: list> Comma-separated list of intermediate user |
|
topics <span class="o">(</span>topics used in the through<span class="o">()</span> |
|
method<span class="o">)</span>. For these topics, the tool |
|
will skip to the end. |
|
--shift-by <Long: number-of-offsets> Reset offsets shifting current offset by |
|
'n', where 'n' can be positive or |
|
negative |
|
--to-datetime <String> Reset offsets to offset from datetime. |
|
Format: 'YYYY-MM-DDTHH:mm:SS.sss' |
|
--to-earliest Reset offsets to earliest offset. |
|
--to-latest Reset offsets to latest offset. |
|
--to-offset <Long> Reset offsets to a specific offset. |
|
--zookeeper Zookeeper option is deprecated by |
|
bootstrap.servers, as the reset tool |
|
would no longer access Zookeeper |
|
directly. |
|
</pre></div> |
|
</div> |
|
<p>Consider the following as reset-offset scenarios for <code>input-topics</code>:</p> |
|
<ul> |
|
<li> by-duration</li> |
|
<li> from-file</li> |
|
<li> shift-by</li> |
|
<li> to-datetime</li> |
|
<li> to-earliest</li> |
|
<li> to-latest</li> |
|
<li> to-offset</li> |
|
</ul> |
|
<p>Only one of these scenarios can be defined. If not, <code>to-earliest</code> will be executed by default</p> |
|
<p>All the other parameters can be combined as needed. For example, if you want to restart an application from an |
|
empty internal state, but not reprocess previous data, simply omit the parameters <code class="docutils literal"><span class="pre">--input-topics</span></code> and |
|
<code class="docutils literal"><span class="pre">--intermediate-topics</span></code>.</p> |
|
</div> |
|
<div class="section" id="step-2-reset-the-local-environments-of-your-application-instances"> |
|
<span id="streams-developer-guide-reset-local-environment"></span><h2>Step 2: Reset the local environments of your application instances<a class="headerlink" href="#step-2-reset-the-local-environments-of-your-application-instances" title="Permalink to this headline"></a></h2> |
|
<p>For a complete application reset, you must delete the application’s local state directory on any machines where the |
|
application instance was run. You must do this before restarting an application instance on the same machine. You can |
|
use either of these methods:</p> |
|
<ul class="simple"> |
|
<li>The API method <code class="docutils literal"><span class="pre">KafkaStreams#cleanUp()</span></code> in your application code.</li> |
|
<li>Manually delete the corresponding local state directory (default location: <code class="docutils literal"><span class="pre">/var/lib/kafka-streams/<application.id></span></code>). For more information, see <a class="reference internal" href="../javadocs.html#streams-javadocs"><span class="std std-ref">state.dir</span></a> StreamsConfig class.</li> |
|
</ul> |
|
</div> |
|
</div> |
|
|
|
|
|
</div> |
|
</div> |
|
<div class="pagination"> |
|
<a href="/{{version}}/documentation/streams/developer-guide/security" class="pagination__btn pagination__btn__prev">Previous</a> |
|
<a href="/{{version}}/documentation/streams/upgrade-guide" class="pagination__btn pagination__btn__next">Next</a> |
|
</div> |
|
</script> |
|
|
|
<!--#include virtual="../../../includes/_header.htm" --> |
|
<!--#include virtual="../../../includes/_top.htm" --> |
|
<div class="content documentation documentation--current"> |
|
<!--#include virtual="../../../includes/_nav.htm" --> |
|
<div class="right"> |
|
<!--#include virtual="../../../includes/_docs_banner.htm" --> |
|
<ul class="breadcrumbs"> |
|
<li><a href="/documentation">Documentation</a></li> |
|
<li><a href="/documentation/streams">Kafka Streams</a></li> |
|
<li><a href="/documentation/streams/developer-guide/">Developer Guide</a></li> |
|
</ul> |
|
<div class="p-content"></div> |
|
</div> |
|
</div> |
|
<!--#include virtual="../../../includes/_footer.htm" --> |
|
<script> |
|
$(function() { |
|
// Show selected style on nav item |
|
$('.b-nav__streams').addClass('selected'); |
|
|
|
//sticky secondary nav |
|
var $navbar = $(".sub-nav-sticky"), |
|
y_pos = $navbar.offset().top, |
|
height = $navbar.height(); |
|
|
|
$(window).scroll(function() { |
|
var scrollTop = $(window).scrollTop(); |
|
|
|
if (scrollTop > y_pos - height) { |
|
$navbar.addClass("navbar-fixed") |
|
} else if (scrollTop <= y_pos) { |
|
$navbar.removeClass("navbar-fixed") |
|
} |
|
}); |
|
|
|
// Display docs subnav items |
|
$('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); |
|
}); |
|
</script> |