Repository navigation
Rebuild a missing Raft snapshot or stop reporting a live voter - #20
Conversation
…ter. The segmented raft log is the write-ahead log until it is purged. After purge, a missing snapshot file is rebuilt from the LSM before the read returns. If the LSM cannot cover that purged prefix, cluster status returns snapshot unrecoverable instead of the cached voter set. Co-authored-by: RustFS <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
Forwarded writes bypass snapshot-repair safeguards, while recovery can accept or publish snapshots that do not cover the required index, leaving critical storage-safety issues unresolved.
3 open issues
What changed in this PR
This PR adds recovery for missing published Raft snapshots using the LSM and propagates unrecoverable storage failures to cluster status instead of reporting cached live voters.
Changes:
- Adds snapshot repair checks to proposal and read paths, plus storage-health reporting.
- Adds regressions for snapshot rebuilding, unpurged-log recovery, and unrecoverable status.
- Documents recovery limits and corrects memtable configuration descriptions.
| File | Description |
|---|---|
| docs/zh/operations.md | Corrects memtable behavior notes. |
| docs/zh/configuration.md | Updates flush-threshold descriptions. |
| docs/zh/architecture.md | Clarifies memtable flushing and backpressure. |
| docs/ja/operations.md | Corrects memtable behavior notes. |
| docs/ja/configuration.md | Updates flush-threshold descriptions. |
| docs/ja/architecture.md | Clarifies memtable flushing and backpressure. |
| docs/fr/operations.md | Corrects memtable behavior notes. |
| docs/fr/configuration.md | Updates flush-threshold descriptions. |
| docs/fr/architecture.md | Clarifies memtable flushing and backpressure. |
| docs/en/architecture.md | Explains snapshot rebuilding and recovery limits. |
| docs/designs/correctness_hardening.md | Updates snapshot recovery guarantees. |
| docs/de/operations.md | Corrects memtable behavior notes. |
| docs/de/configuration.md | Updates flush-threshold descriptions. |
| docs/de/architecture.md | Clarifies memtable flushing and backpressure. |
| crates/metadb-server/tests/snapshot_status_tests.rs | Tests status rejection after unrecoverable storage loss. |
| crates/metadb-server/src/grpc_services.rs | Uses health-aware routes in GetRoutes. |
| crates/metadb-raft/tests/snapshot_read_path_tests.rs | Tests rebuilding, log replay, and unrecoverable reads. |
| crates/metadb-raft/src/state_machine.rs | Rebuilds missing snapshots and records recovery failures. |
| crates/metadb-raft/src/group_host.rs | Adds repair checks before proposals and reads. |
| crates/metadb-cluster/src/cell_manager.rs | Rejects live-status routes for unrecoverable local shards. |
Review effort: Balanced
| payload: Vec<u8>, | ||
| deadline: tokio::time::Instant, | ||
| ) -> Result<MetaDbRaftResponse> { | ||
| self.repair_published_snapshot().await?; |
| Ok(snapshot) => { | ||
| self.clear_unrecoverable(); | ||
| return Ok(Some(snapshot)); |
| let dir = self | ||
| .shard_dir | ||
| .join(format!("raft/snapshots/snap_{index:020}")); | ||
| !snapshot_dir_is_complete(&dir) |
ClientWrite enters propose_local, so that path now repairs or rejects an unrecoverable snapshot. A loaded or rebuilt snapshot is kept only when its index covers the purged log, and healthy requests stat SNAPSHOT_META instead of decoding every SSTable. Co-authored-by: RustFS <[email protected]>
loverustfs
left a comment
There was a problem hiding this comment.
Storage review of 46636fac30560a1e0ed4e463f3118d1de6a26afd on cursor/snapshot-wal-rebuild-ded3. These must-fix items still block merge. GitHub will not accept REQUEST_CHANGES from the connected account because it is also the PR author, so this is recorded as a comment review.
This is not an independent post-purge Raft WAL: after purge, a rebuild still depends on the flushed LSM, and there is no fourth copy. The PR body already states that correctly. Do not describe this change as adding an independent WAL.
The second commit correctly routes forwarded ClientWrite through propose_local repair, rejects a loaded or rebuilt snapshot that does not cover the purged or published index, and caches a healthy SNAPSHOT_META stat. Those changes do not close the live-but-unrepaired window.
Must fix before merge
-
Recompute repair or unrecoverable in
ShardRaftGroup::startandShardStateMachine::open.openstill setsunrecoverable: Mutex::new(None), so a restart after the purged log, snapshot, and LSM are all gone can look healthy until a later snapshot read, and may accept writes as an empty new shard. -
routes_for_live_statusmust probepublished_snapshot_missingor attempt repair. It currently only reads the in-memorystorage_unrecoverable()latch.GetRoutesmust not advertise this replica as a live voter when the published snapshot is missing and no priorget_current_snapshothas run. -
Do not treat a green Format, Clippy & Test run as a first-ship or scale pass. Trillion-scale figures in the Baidu paper are not results of this repository. The historical
Snapshot(None)假活 evidence from thef325f6apack 063136 must not be written as a result of this tree.
Should fix
- Do not latch unrecoverable on every transient
build_snapshot_syncI/O error. - Maintenance or
startshould rewrite a missing snapshot from the LSM without waiting for a client read. - Keep product wording in
docs/enfirst. Do not paste Englishmemtable_size_bytesnotes intodocs/zh,de,ja, orfr.
Must-land tests still missing on this PR
Keep these on the same #20 line; do not open a new PR.
restart_after_purged_log_missing_lsm_and_snapshot_is_unrecoverable_before_servingroutes_for_live_status_detects_missing_snapshot_without_prior_get_current_snapshotrestart_rebuilds_snapshot_from_lsm_without_a_prior_live_readlearner_install_succeeds_after_leader_snapshot_file_deletedlearner_install_fails_closed_when_leader_is_unrecoverable
Existing coverage that should stay: live rewrite after SNAPSHOT_META is deleted; unpurged log reconstructs after restart; live triple-loss returns snapshot unrecoverable; propose_local rewrite; an older snapshot must not satisfy a purged prefix. The status test still calls get_current_snapshot before routes_for_live_status, so it does not cover item 2.
2PC process-restart-after-flush-and-purge tests belong on #18 or a follow-up, not in #20's files.
| last_applied_hlc: RwLock::new(cp.flushed_max_hlc), | ||
| last_membership: RwLock::new(initial_membership), | ||
| frontier_tx, | ||
| unrecoverable: Mutex::new(None), |
There was a problem hiding this comment.
Must fix: this latch is reset to None on every open. After a restart where the purged log, snapshot, and LSM are all gone, the process can look healthy until a later snapshot read, and may accept writes as an empty new shard. Recompute repair or unrecoverable here and in ShardRaftGroup::start before serving.
| let routes = self.list_routes(); | ||
| for route in &routes { | ||
| if let Some(group) = self.host.get_group(route.shard_id) { | ||
| if let Some(reason) = group.storage_unrecoverable() { |
There was a problem hiding this comment.
Must fix: this only reads the in-memory storage_unrecoverable() latch. It does not probe published_snapshot_missing and does not attempt repair. GetRoutes must not advertise this replica as a live voter when the published snapshot is missing, even if no prior get_current_snapshot has set the latch.
Open recomputes the snapshot before the group can serve, and cluster status repairs or rejects a missing snapshot without waiting for get_current_snapshot. After purge, reconstruction still depends on the LSM. A transient publish error does not latch the shard unrecoverable. Co-authored-by: RustFS <[email protected]>
Keep the fail-closed snapshot checks together with the restored-node identity binding, the shutdown tick cancel and 10s flush bound, and proposal-receipt retry rules. A linearizable read repairs a missing snapshot before the read-quorum test hook can fire. Co-authored-by: RustFS <[email protected]>


Summary
Closes the remaining fake-alive window after a missing Raft snapshot. This is the independent RustFS implementation. It does not claim the Baidu paper's production figures, and a green CI run is not a first-ship pass.
The segmented raft log is the write-ahead log only until it is purged. After purge, the LSM is the remaining copy. There is no fourth copy of the command log.
Startup recomputes that state before the group can serve. A missing snapshot is rewritten from the LSM. If the log was purged and the LSM cannot be opened or cannot cover the purged index, open returns
snapshot unrecoverableand does not accept writes onto an empty LSM. The 250 ms maintenance tick performs the same rewrite without waiting for a client read. Cluster status probes the snapshot directory itself, soGetRoutescannot advertise a live voter while the file is already gone and the in-memory latch is still empty.A readable older snapshot does not satisfy a higher purged index. A rebuild that cannot cover that index is not published. A transient snapshot-publish I/O error does not latch the shard unrecoverable. Forwarded
ClientWriteRPCs enterpropose_localand use the same check.Tests
cargo test -p metadb-raft --tests --lockedcargo test -p metadb-server --test snapshot_status_tests --lockedcargo test -p metadb-raft --test snapshot_read_path_tests --lockedcargo clippy -p metadb-raft -p metadb-cluster -p metadb-server --all-targets --locked -- -D warningscargo fmt --checkRegressions kept:
SNAPSHOT_METAfrom the LSM, and Lookup/List of the committed keys succeed.snapshot unrecoverable, androutes_for_live_statusfails.Added:
restart_after_purged_log_missing_lsm_and_snapshot_is_unrecoverable_before_servingroutes_for_live_status_detects_missing_snapshot_without_prior_get_current_snapshotrestart_rebuilds_snapshot_from_lsm_without_a_prior_live_readlearner_install_succeeds_after_leader_snapshot_file_deletedlearner_install_fails_closed_when_leader_is_unrecoverableRemaining gaps
SNAPSHOT_METAis checked when the snapshot is loaded or its meta stat changes, not by decoding every SSTable on every request.