src-kafka/raft

Kafka Raft

Kafka Raft is a sub module of Apache Kafka which features a tailored version of Raft Consensus Protocol.

Eventually this module will be integrated into the Kafka server. For now, we have a standalone test server which can be used for performance testing. Below we describe the details to set this up.

Run Single Quorum

bin/test-raft-server-start.sh config/raft.properties

Run Multi Node Quorum

Create 3 separate raft quorum properties as the following (note that the zookeeper.connect config is required, but unused):

cat << EOF >> config/raft-quorum-1.properties

broker.id=1
listeners=PLAINTEXT://localhost:9092
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-1

zookeeper.connect=localhost:2181
EOF

cat << EOF >> config/raft-quorum-2.properties

broker.id=2
listeners=PLAINTEXT://localhost:9093
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-2

zookeeper.connect=localhost:2181
EOF

cat << EOF >> config/raft-quorum-3.properties

broker.id=3
listeners=PLAINTEXT://localhost:9094
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-3

zookeeper.connect=localhost:2181
EOF

Open up 3 separate terminals, and run individual commands:

bin/test-raft-server-start.sh --config config/raft-quorum-1.properties
bin/test-raft-server-start.sh --config config/raft-quorum-2.properties
bin/test-raft-server-start.sh --config config/raft-quorum-3.properties

Once a leader is elected, it will begin writing to an internal __cluster_metadata topic with a steady workload of random data. You can control the workload using the --throughput and --record-size arguments passed to test-raft-server-start.sh.