You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
74 lines
2.7 KiB
74 lines
2.7 KiB
4 years ago
|
Kafka Raft
|
||
|
|
=================
|
||
|
|
Kafka Raft is a sub module of Apache Kafka which features a tailored version of
|
||
|
|
[Raft Consensus Protocol](https://www.usenix.org/system/files/conference/atc14/atc14-paper-ongaro.pdf).
|
||
|
|
<p>
|
||
|
|
|
||
|
|
Eventually this module will be integrated into the Kafka server. For now,
|
||
|
|
we have a standalone test server which can be used for performance testing.
|
||
|
|
Below we describe the details to set this up.
|
||
|
|
|
||
|
|
### Run Single Quorum ###
|
||
|
|
bin/test-raft-server-start.sh config/raft.properties
|
||
|
|
|
||
|
|
### Run Multi Node Quorum ###
|
||
|
|
Create 3 separate raft quorum properties as the following:
|
||
|
|
|
||
|
|
`cat << EOF >> config/raft-quorum-1.properties`
|
||
|
|
|
||
|
|
broker.id=1
|
||
|
|
listeners=PLAINTEXT://localhost:9092
|
||
|
|
quorum.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094
|
||
|
|
quorum.bootstrap.voters=1,2,3
|
||
|
|
log.dirs=/tmp/raft-logs-1
|
||
|
|
verbose=true
|
||
|
|
|
||
|
|
zookeeper.connect=localhost:2181
|
||
|
|
EOF
|
||
|
|
|
||
|
|
`cat << EOF >> config/raft-quorum-2.properties`
|
||
|
|
|
||
|
|
broker.id=2
|
||
|
|
listeners=PLAINTEXT://localhost:9093
|
||
|
|
quorum.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094
|
||
|
|
quorum.bootstrap.voters=1,2,3
|
||
|
|
log.dirs=/tmp/raft-logs-2
|
||
|
|
verbose=true
|
||
|
|
|
||
|
|
zookeeper.connect=localhost:2181
|
||
|
|
EOF
|
||
|
|
|
||
|
|
`cat << EOF >> config/raft-quorum-3.properties`
|
||
|
|
|
||
|
|
broker.id=3
|
||
|
|
listeners=PLAINTEXT://localhost:9094
|
||
|
|
quorum.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094
|
||
|
|
quorum.bootstrap.voters=1,2,3
|
||
|
|
log.dirs=/tmp/raft-logs-3
|
||
|
|
verbose=true
|
||
|
|
|
||
|
|
zookeeper.connect=localhost:2181
|
||
|
|
EOF
|
||
|
|
|
||
|
|
Open up 3 separate terminals, and run individual commands:
|
||
|
|
|
||
|
|
bin/test-raft-server-start.sh config/raft-quorum-1.properties
|
||
|
|
bin/test-raft-server-start.sh config/raft-quorum-2.properties
|
||
|
|
bin/test-raft-server-start.sh config/raft-quorum-3.properties
|
||
|
|
|
||
|
|
This would setup a three node Raft quorum with node id 1,2,3 using different endpoints and log dirs.
|
||
|
|
|
||
|
|
### Simulate a distributed state machine ###
|
||
|
|
You need to use a `VerifiableProducer` to produce monolithic increasing records to the replicated state machine.
|
||
|
|
|
||
|
|
./bin/kafka-run-class.sh org.apache.kafka.tools.VerifiableProducer --bootstrap-server http://localhost:9092 \
|
||
|
|
--topic __cluster_metadata --max-messages 2000 --throughput 1 --producer.config config/producer.properties
|
||
|
|
### Run Performance Test ###
|
||
|
|
Make sure to turn off the printing by setting `verbose=false` in the property
|
||
|
|
files `config/raft-*.properties`, to ensure minimum affection to the performance.
|
||
|
|
|
||
|
|
Then run the `ProducerPerformance` module using this example command:
|
||
|
|
|
||
|
|
./bin/kafka-producer-perf-test.sh --topic __cluster_metadata --num-records 2000 --throughput -1 --record-size 10 --producer.config config/producer.properties
|
||
|
|
Collect the print out throughput to compare with Kafka performance.
|