|
|
|
|
dist
|
|
|
|
|
*classes
|
|
|
|
|
*.class
|
|
|
|
|
target/
|
|
|
|
|
build/
|
|
|
|
|
build_eclipse/
|
|
|
|
|
out/
|
|
|
|
|
.gradle/
|
|
|
|
|
.vscode/
|
|
|
|
|
lib_managed/
|
|
|
|
|
src_managed/
|
|
|
|
|
project/boot/
|
|
|
|
|
project/plugins/project/
|
|
|
|
|
patch-process/*
|
|
|
|
|
.idea
|
|
|
|
|
.svn
|
|
|
|
|
.classpath
|
|
|
|
|
/.metadata
|
|
|
|
|
/.recommenders
|
|
|
|
|
*~
|
|
|
|
|
*#
|
|
|
|
|
.#*
|
|
|
|
|
rat.out
|
|
|
|
|
TAGS
|
|
|
|
|
*.iml
|
|
|
|
|
.project
|
|
|
|
|
.settings
|
|
|
|
|
*.ipr
|
|
|
|
|
*.iws
|
|
|
|
|
.vagrant
|
|
|
|
|
Vagrantfile.local
|
|
|
|
|
/logs
|
|
|
|
|
.DS_Store
|
|
|
|
|
|
|
|
|
|
config/server-*
|
|
|
|
|
config/zookeeper-*
|
|
|
|
|
core/data/*
|
|
|
|
|
gradle/wrapper/*.jar
|
|
|
|
|
gradlew.bat
|
|
|
|
|
|
|
|
|
|
results
|
|
|
|
|
tests/results
|
|
|
|
|
.ducktape
|
|
|
|
|
tests/.ducktape
|
|
|
|
|
tests/venv
|
|
|
|
|
.cache
|
|
|
|
|
|
|
|
|
|
docs/generated/
|
|
|
|
|
|
|
|
|
|
.release-settings.json
|
|
|
|
|
|
|
|
|
|
kafkatest.egg-info/
|
|
|
|
|
systest/
|
|
|
|
|
*.swp
|
|
|
|
|
jmh-benchmarks/generated
|
|
|
|
|
jmh-benchmarks/src/main/generated
|
|
|
|
|
**/.jqwik-database
|
|
|
|
|
**/src/generated
|
|
|
|
|
**/src/generated-test
|
[KAFKA-13369] Follower fetch protocol changes for tiered storage. (#11390)
This PR implements the follower fetch protocol as mentioned in KIP-405.
Added a new version for ListOffsets protocol to receive local log start offset on the leader replica. This is used by follower replicas to find the local log star offset on the leader.
Added a new version for FetchRequest protocol to receive OffsetMovedToTieredStorageException error. This is part of the enhanced fetch protocol as described in KIP-405.
We introduced a new field locaLogStartOffset to maintain the log start offset in the local logs. Existing logStartOffset will continue to be the log start offset of the effective log that includes the segments in remote storage.
When a follower receives OffsetMovedToTieredStorage, then it tries to build the required state from the leader and remote storage so that it can be ready to move to fetch state.
Introduced RemoteLogManager which is responsible for
initializing RemoteStorageManager and RemoteLogMetadataManager instances.
receives any leader and follower replica events and partition stop events and act on them
also provides APIs to fetch indexes, metadata about remote log segments.
Followup PRs will add more functionality like copying segments to tiered storage, retention checks to clean local and remote log segments. This will change the local log start offset and make sure the follower fetch protocol works fine for several cases.
You can look at the detailed protocol changes in KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-FollowerReplication
Co-authors: satishd@apache.org, kamal.chandraprakash@gmail.com, yingz@uber.com
Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Cong Ding <cong@ccding.com>, Tirtha Chatterjee <tirtha.p.chatterjee@gmail.com>, Yaodong Yang <yangyaodong88@gmail.com>, Divij Vaidya <diviv@amazon.com>, Luke Chen <showuon@gmail.com>, Jun Rao <junrao@gmail.com>
2 years ago
|
|
|
|
|
|
|
|
storage/kafka-tiered-storage/
|
