KAFKA-7500: MirrorMaker 2.0 (KIP-382)
Implementation of [KIP-382 "MirrorMaker 2.0"](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0)
Author: Ryanne Dolan <ryannedolan@gmail.com>
Author: Arun Mathew <arunmathew88@gmail.com>
Author: In Park <inpark@cloudera.com>
Author: Andre Price <obsoleted@users.noreply.github.com>
Author: christian.hagel@rio.cloud <christian.hagel@rio.cloud>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, William Hammond <william.t.hammond@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>, Jakub Korzeniowski, Tim Carey-Smith, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Arun Mathew, Jeremy-l-ford, vpernin, Oleg Kasian <oleg.kasian@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Qihong Chen, Sriharsha Chintalapani <sriharsha@apache.org>, Jun Rao <junrao@gmail.com>, Randall Hauch <rhauch@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes #6295 from ryannedolan/KIP-382
5 years ago
|
|
|
# Licensed to the Apache Software Foundation (ASF) under A or more
|
|
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
|
|
# this work for additional information regarding copyright ownership.
|
|
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
# (the "License"); you may not use this file except in compliance with
|
|
|
|
# the License. You may obtain a copy of the License at
|
|
|
|
#
|
|
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
#
|
|
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
# See the License for the specific language governing permissions and
|
|
|
|
# limitations under the License.
|
|
|
|
# see org.apache.kafka.clients.consumer.ConsumerConfig for more details
|
|
|
|
|
|
|
|
# Sample MirrorMaker 2.0 top-level configuration file
|
|
|
|
# Run with ./bin/connect-mirror-maker.sh connect-mirror-maker.properties
|
|
|
|
|
|
|
|
# specify any number of cluster aliases
|
|
|
|
clusters = A, B
|
KAFKA-7500: MirrorMaker 2.0 (KIP-382)
Implementation of [KIP-382 "MirrorMaker 2.0"](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0)
Author: Ryanne Dolan <ryannedolan@gmail.com>
Author: Arun Mathew <arunmathew88@gmail.com>
Author: In Park <inpark@cloudera.com>
Author: Andre Price <obsoleted@users.noreply.github.com>
Author: christian.hagel@rio.cloud <christian.hagel@rio.cloud>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, William Hammond <william.t.hammond@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>, Jakub Korzeniowski, Tim Carey-Smith, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Arun Mathew, Jeremy-l-ford, vpernin, Oleg Kasian <oleg.kasian@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Qihong Chen, Sriharsha Chintalapani <sriharsha@apache.org>, Jun Rao <junrao@gmail.com>, Randall Hauch <rhauch@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes #6295 from ryannedolan/KIP-382
5 years ago
|
|
|
|
|
|
|
# connection information for each cluster
|
|
|
|
# This is a comma separated host:port pairs for each cluster
|
|
|
|
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
|
KAFKA-7500: MirrorMaker 2.0 (KIP-382)
Implementation of [KIP-382 "MirrorMaker 2.0"](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0)
Author: Ryanne Dolan <ryannedolan@gmail.com>
Author: Arun Mathew <arunmathew88@gmail.com>
Author: In Park <inpark@cloudera.com>
Author: Andre Price <obsoleted@users.noreply.github.com>
Author: christian.hagel@rio.cloud <christian.hagel@rio.cloud>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, William Hammond <william.t.hammond@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>, Jakub Korzeniowski, Tim Carey-Smith, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Arun Mathew, Jeremy-l-ford, vpernin, Oleg Kasian <oleg.kasian@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Qihong Chen, Sriharsha Chintalapani <sriharsha@apache.org>, Jun Rao <junrao@gmail.com>, Randall Hauch <rhauch@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes #6295 from ryannedolan/KIP-382
5 years ago
|
|
|
A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092
|
|
|
|
B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092
|
|
|
|
|
|
|
|
# enable and configure individual replication flows
|
|
|
|
A->B.enabled = true
|
|
|
|
|
|
|
|
# regex which defines which topics gets replicated. For eg "foo-.*"
|
|
|
|
A->B.topics = .*
|
|
|
|
|
|
|
|
B->A.enabled = true
|
|
|
|
B->A.topics = .*
|
|
|
|
|
|
|
|
# Setting replication factor of newly created remote topics
|
|
|
|
replication.factor=1
|
|
|
|
|
|
|
|
############################# Internal Topic Settings #############################
|
|
|
|
# The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and
|
|
|
|
# "mm2-offset-syncs.B.internal"
|
|
|
|
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
|
|
|
|
checkpoints.topic.replication.factor=1
|
|
|
|
heartbeats.topic.replication.factor=1
|
|
|
|
offset-syncs.topic.replication.factor=1
|
|
|
|
|
|
|
|
# The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and
|
|
|
|
# "mm2-status.B.internal"
|
|
|
|
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
|
|
|
|
offset.storage.replication.factor=1
|
|
|
|
status.storage.replication.factor=1
|
|
|
|
config.storage.replication.factor=1
|
KAFKA-7500: MirrorMaker 2.0 (KIP-382)
Implementation of [KIP-382 "MirrorMaker 2.0"](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0)
Author: Ryanne Dolan <ryannedolan@gmail.com>
Author: Arun Mathew <arunmathew88@gmail.com>
Author: In Park <inpark@cloudera.com>
Author: Andre Price <obsoleted@users.noreply.github.com>
Author: christian.hagel@rio.cloud <christian.hagel@rio.cloud>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, William Hammond <william.t.hammond@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>, Jakub Korzeniowski, Tim Carey-Smith, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Arun Mathew, Jeremy-l-ford, vpernin, Oleg Kasian <oleg.kasian@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Qihong Chen, Sriharsha Chintalapani <sriharsha@apache.org>, Jun Rao <junrao@gmail.com>, Randall Hauch <rhauch@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes #6295 from ryannedolan/KIP-382
5 years ago
|
|
|
|
|
|
|
# customize as needed
|
|
|
|
# replication.policy.separator = _
|
|
|
|
# sync.topic.acls.enabled = false
|
|
|
|
# emit.heartbeats.interval.seconds = 5
|