Skip to content

Rebuild a missing Raft snapshot or stop reporting a live voter - #20

Merged
loverustfs merged 4 commits into
mainfrom
cursor/snapshot-wal-rebuild-ded3
Nov 19, 2026
Merged

loverustfs merged 4 commits into
mainfrom
cursor/snapshot-wal-rebuild-ded3

Conversation

@loverustfs

@loverustfs loverustfs commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes the remaining fake-alive window after a missing Raft snapshot. This is the independent RustFS implementation. It does not claim the Baidu paper's production figures, and a green CI run is not a first-ship pass.

The segmented raft log is the write-ahead log only until it is purged. After purge, the LSM is the remaining copy. There is no fourth copy of the command log.

Startup recomputes that state before the group can serve. A missing snapshot is rewritten from the LSM. If the log was purged and the LSM cannot be opened or cannot cover the purged index, open returns snapshot unrecoverable and does not accept writes onto an empty LSM. The 250 ms maintenance tick performs the same rewrite without waiting for a client read. Cluster status probes the snapshot directory itself, so GetRoutes cannot advertise a live voter while the file is already gone and the in-memory latch is still empty.

A readable older snapshot does not satisfy a higher purged index. A rebuild that cannot cover that index is not published. A transient snapshot-publish I/O error does not latch the shard unrecoverable. Forwarded ClientWrite RPCs enter propose_local and use the same check.

Tests

  • cargo test -p metadb-raft --tests --locked
  • cargo test -p metadb-server --test snapshot_status_tests --locked
  • cargo test -p metadb-raft --test snapshot_read_path_tests --locked
  • cargo clippy -p metadb-raft -p metadb-cluster -p metadb-server --all-targets --locked -- -D warnings
  • cargo fmt --check

Regressions kept:

  • A live process rewrites a missing SNAPSHOT_META from the LSM, and Lookup/List of the committed keys succeed.
  • Restart with an unpurged log reconstructs those keys after the snapshot directory is removed.
  • After purge, deleting the SSTables and the snapshot on a live process returns snapshot unrecoverable, and routes_for_live_status fails.

Added:

  • restart_after_purged_log_missing_lsm_and_snapshot_is_unrecoverable_before_serving
  • routes_for_live_status_detects_missing_snapshot_without_prior_get_current_snapshot
  • restart_rebuilds_snapshot_from_lsm_without_a_prior_live_read
  • learner_install_succeeds_after_leader_snapshot_file_deleted
  • learner_install_fails_closed_when_leader_is_unrecoverable

Remaining gaps

  • After purge, reconstruction still depends on the LSM. This PR does not add another command WAL.
  • A torn SSTable behind an unchanged SNAPSHOT_META is checked when the snapshot is loaded or its meta stat changes, not by decoding every SSTable on every request.
  • The 165 leadership and write-quorum application failures from the older overwrite retest are not in this PR. The later 11 Gi three-process OOM is not a snapshot bug and is not addressed here.
  • Jumbo membership, multi-cell operation, cluster receipts, backup leftovers, the stress harness, and 2PC restart coverage are not part of this change.
Open in Web Open in Cursor 

…ter.

The segmented raft log is the write-ahead log until it is purged. After purge, a missing snapshot file is rebuilt from the LSM before the read returns. If the LSM cannot cover that purged prefix, cluster status returns snapshot unrecoverable instead of the cached voter set.

Co-authored-by: RustFS <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 11:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Forwarded writes bypass snapshot-repair safeguards, while recovery can accept or publish snapshots that do not cover the required index, leaving critical storage-safety issues unresolved.

3 open issues
What changed in this PR

This PR adds recovery for missing published Raft snapshots using the LSM and propagates unrecoverable storage failures to cluster status instead of reporting cached live voters.

Changes:

  • Adds snapshot repair checks to proposal and read paths, plus storage-health reporting.
  • Adds regressions for snapshot rebuilding, unpurged-log recovery, and unrecoverable status.
  • Documents recovery limits and corrects memtable configuration descriptions.
File Description
docs/​zh/​operations.md Corrects memtable behavior notes.
docs/​zh/​configuration.md Updates flush-threshold descriptions.
docs/​zh/​architecture.md Clarifies memtable flushing and backpressure.
docs/​ja/​operations.md Corrects memtable behavior notes.
docs/​ja/​configuration.md Updates flush-threshold descriptions.
docs/​ja/​architecture.md Clarifies memtable flushing and backpressure.
docs/​fr/​operations.md Corrects memtable behavior notes.
docs/​fr/​configuration.md Updates flush-threshold descriptions.
docs/​fr/​architecture.md Clarifies memtable flushing and backpressure.
docs/​en/​architecture.md Explains snapshot rebuilding and recovery limits.
docs/​designs/​correctness_hardening.md Updates snapshot recovery guarantees.
docs/​de/​operations.md Corrects memtable behavior notes.
docs/​de/​configuration.md Updates flush-threshold descriptions.
docs/​de/​architecture.md Clarifies memtable flushing and backpressure.
crates/​metadb-server/​tests/​snapshot_status_tests.rs Tests status rejection after unrecoverable storage loss.
crates/​metadb-server/​src/​grpc_services.rs Uses health-aware routes in GetRoutes.
crates/​metadb-raft/​tests/​snapshot_read_path_tests.rs Tests rebuilding, log replay, and unrecoverable reads.
crates/​metadb-raft/​src/​state_machine.rs Rebuilds missing snapshots and records recovery failures.
crates/​metadb-raft/​src/​group_host.rs Adds repair checks before proposals and reads.
crates/​metadb-cluster/​src/​cell_manager.rs Rejects live-status routes for unrecoverable local shards.

Review effort: Balanced

Comment thread crates/metadb-raft/src/group_host.rs Outdated
payload: Vec<u8>,
deadline: tokio::time::Instant,
) -> Result<MetaDbRaftResponse> {
self.repair_published_snapshot().await?;
Comment on lines +467 to +469
Ok(snapshot) => {
self.clear_unrecoverable();
return Ok(Some(snapshot));
Comment thread crates/metadb-raft/src/state_machine.rs Outdated
let dir = self
.shard_dir
.join(format!("raft/snapshots/snap_{index:020}"));
!snapshot_dir_is_complete(&dir)
ClientWrite enters propose_local, so that path now repairs or rejects an unrecoverable snapshot. A loaded or rebuilt snapshot is kept only when its index covers the purged log, and healthy requests stat SNAPSHOT_META instead of decoding every SSTable.

Co-authored-by: RustFS <[email protected]>

@loverustfs loverustfs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storage review of 46636fac30560a1e0ed4e463f3118d1de6a26afd on cursor/snapshot-wal-rebuild-ded3. These must-fix items still block merge. GitHub will not accept REQUEST_CHANGES from the connected account because it is also the PR author, so this is recorded as a comment review.

This is not an independent post-purge Raft WAL: after purge, a rebuild still depends on the flushed LSM, and there is no fourth copy. The PR body already states that correctly. Do not describe this change as adding an independent WAL.

The second commit correctly routes forwarded ClientWrite through propose_local repair, rejects a loaded or rebuilt snapshot that does not cover the purged or published index, and caches a healthy SNAPSHOT_META stat. Those changes do not close the live-but-unrepaired window.

Must fix before merge

  1. Recompute repair or unrecoverable in ShardRaftGroup::start and ShardStateMachine::open. open still sets unrecoverable: Mutex::new(None), so a restart after the purged log, snapshot, and LSM are all gone can look healthy until a later snapshot read, and may accept writes as an empty new shard.

  2. routes_for_live_status must probe published_snapshot_missing or attempt repair. It currently only reads the in-memory storage_unrecoverable() latch. GetRoutes must not advertise this replica as a live voter when the published snapshot is missing and no prior get_current_snapshot has run.

  3. Do not treat a green Format, Clippy & Test run as a first-ship or scale pass. Trillion-scale figures in the Baidu paper are not results of this repository. The historical Snapshot(None) 假活 evidence from the f325f6a pack 063136 must not be written as a result of this tree.

Should fix

  • Do not latch unrecoverable on every transient build_snapshot_sync I/O error.
  • Maintenance or start should rewrite a missing snapshot from the LSM without waiting for a client read.
  • Keep product wording in docs/en first. Do not paste English memtable_size_bytes notes into docs/zh, de, ja, or fr.

Must-land tests still missing on this PR

Keep these on the same #20 line; do not open a new PR.

  • restart_after_purged_log_missing_lsm_and_snapshot_is_unrecoverable_before_serving
  • routes_for_live_status_detects_missing_snapshot_without_prior_get_current_snapshot
  • restart_rebuilds_snapshot_from_lsm_without_a_prior_live_read
  • learner_install_succeeds_after_leader_snapshot_file_deleted
  • learner_install_fails_closed_when_leader_is_unrecoverable

Existing coverage that should stay: live rewrite after SNAPSHOT_META is deleted; unpurged log reconstructs after restart; live triple-loss returns snapshot unrecoverable; propose_local rewrite; an older snapshot must not satisfy a purged prefix. The status test still calls get_current_snapshot before routes_for_live_status, so it does not cover item 2.

2PC process-restart-after-flush-and-purge tests belong on #18 or a follow-up, not in #20's files.

last_applied_hlc: RwLock::new(cp.flushed_max_hlc),
last_membership: RwLock::new(initial_membership),
frontier_tx,
unrecoverable: Mutex::new(None),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must fix: this latch is reset to None on every open. After a restart where the purged log, snapshot, and LSM are all gone, the process can look healthy until a later snapshot read, and may accept writes as an empty new shard. Recompute repair or unrecoverable here and in ShardRaftGroup::start before serving.

let routes = self.list_routes();
for route in &routes {
if let Some(group) = self.host.get_group(route.shard_id) {
if let Some(reason) = group.storage_unrecoverable() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must fix: this only reads the in-memory storage_unrecoverable() latch. It does not probe published_snapshot_missing and does not attempt repair. GetRoutes must not advertise this replica as a live voter when the published snapshot is missing, even if no prior get_current_snapshot has set the latch.

cursoragent and others added 2 commits November 19, 2026 12:01
Open recomputes the snapshot before the group can serve, and cluster status repairs or rejects a missing snapshot without waiting for get_current_snapshot. After purge, reconstruction still depends on the LSM. A transient publish error does not latch the shard unrecoverable.

Co-authored-by: RustFS <[email protected]>
Keep the fail-closed snapshot checks together with the restored-node identity binding, the shutdown tick cancel and 10s flush bound, and proposal-receipt retry rules. A linearizable read repairs a missing snapshot before the read-quorum test hook can fire.

Co-authored-by: RustFS <[email protected]>
@loverustfs
loverustfs merged commit 0d967f2 into main Nov 19, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants