From bbfecaef725456f648f03530d26a5395042966fa Mon Sep 17 00:00:00 2001 From: Konstantine Karantasis Date: Fri, 21 Feb 2020 12:25:35 -0800 Subject: [PATCH] MINOR: Document endpoints for connector topic tracking (KIP-558) Update the site documentation to include the endpoints introduced with KIP-558 and a short paragraph on how this feature is used in Connect. Author: Konstantine Karantasis Reviewers: Toby Drake , Ewen Cheslack-Postava Closes #8148 from kkonstantine/kip-558-docs --- docs/connect.html | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/connect.html b/docs/connect.html index a92bb04849f..473569cdf68 100644 --- a/docs/connect.html +++ b/docs/connect.html @@ -239,6 +239,8 @@
  • POST /connectors/{name}/restart - restart a connector (typically because it has failed)
  • POST /connectors/{name}/tasks/{taskId}/restart - restart an individual task (typically because it has failed)
  • DELETE /connectors/{name} - delete a connector, halting all tasks and deleting its configuration
    • +
  • GET /connectors/{name}/topics - get the set of topics that a specific connector is using since the connector was created or since a request to reset its set of active topics was issued
    • +
  • PUT /connectors/{name}/topics/reset - send a request to empty the set of active topics of a connector

    • Kafka Connect also provides a REST API for getting information about connector plugins:

    @@ -576,6 +578,10 @@ In most cases, connector and task states will match, though they may be different for short periods of time when changes are occurring or if tasks have failed. For example, when a connector is first started, there may be a noticeable delay before the connector and its tasks have all transitioned to the RUNNING state. States will also diverge when tasks fail since Connect does not automatically restart failed tasks. To restart a connector/task manually, you can use the restart APIs listed above. Note that if you try to restart a task while a rebalance is taking place, Connect will return a 409 (Conflict) status code. You can retry after the rebalance completes, but it might not be necessary since rebalances effectively restart all the connectors and tasks in the cluster.

    +

    + Starting with 2.5.0, Kafka Connect uses the status.storage.topic to also store information related to the topics that each connector is using. Connect Workers use these per-connector topic status updates to respond to requests to the REST endpoint GET /connectors/{name}/topics by returning the set of topic names that a connector is using. A request to the REST endpoint PUT /connectors/{name}/topics/reset resets the set of active topics for a connector and allows a new set to be populated, based on the connector's latest pattern of topic usage. Upon connector deletion, the set of the connector's active topics is also deleted. Topic tracking is enabled by default but can be disabled by setting topic.tracking.enable=false. If you want to disallow requests to reset the active topics of connectors during runtime, set the Worker property topic.tracking.allow.reset=false. +

    +

    It's sometimes useful to temporarily stop the message processing of a connector. For example, if the remote system is undergoing maintenance, it would be preferable for source connectors to stop polling it for new data instead of filling logs with exception spam. For this use case, Connect offers a pause/resume API. While a source connector is paused, Connect will stop polling it for additional records. While a sink connector is paused, Connect will stop pushing new messages to it. The pause state is persistent, so even if you restart the cluster, the connector will not begin message processing again until the task has been resumed. Note that there may be a delay before all of a connector's tasks have transitioned to the PAUSED state since it may take time for them to finish whatever processing they were in the middle of when being paused. Additionally, failed tasks will not transition to the PAUSED state until they have been restarted.