Skip to content

Close backup publish, manifest, and scratch leftovers - #19

Merged
loverustfs merged 2 commits into
mainfrom
cursor/backup-restore-leftovers-c11b
Nov 19, 2026
Merged

loverustfs merged 2 commits into
mainfrom
cursor/backup-restore-leftovers-c11b

Conversation

@loverustfs

@loverustfs loverustfs commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes three leftovers from the operator backup path that is already on main. This branch does not change the hold / create / release / restore control plane. backup create still fails with install a hold before copying so purge cannot remove the image when no active hold is installed.

Rebased onto fe9c94ea (current main, including #25, #18, #21, #22, and #23). The only conflict was crates/metadb-raft/src/backup.rs, in copy_node_metadata. Resolution keeps both sides:

  • From Finish CI tests stuck on shutdown or a restored node identity #25: backup copies CLUSTER_IDENTITY and refuses a cell directory that lacks it. Restore refuses a tree whose manifest lists CELL_MANAGER_STATE without the matching CLUSTER_IDENTITY.
  • From this branch: the publish-window lock, offline readonly MANIFEST reads, and crash-scratch cleanup. The scratch lock is held through the byte copy.

The optional copy that this branch had before the rebase (copy CLUSTER_IDENTITY only when the file already exists, and allow restore without it) was not kept. #25's checks are unchanged.

A green CI run is not a first-ship pass. This is the independent RustFS implementation. Paper figures in arXiv:2610.32247v1 are Baidu production numbers, not results from this repository.

What changed

  1. Publish window. be/SHARD_SET.lock is held from the final live shard-directory and registry check through the artifact rename. Shard directory creation, split publication, registry updates, and cell_* state writes take the same lock. A shard that shows up before that locked check fails the backup. A creator that takes the lock cannot create the directory between the check and the rename. Covered by a_shard_created_before_rename_rejects_publication and shard_directory_cannot_appear_while_the_publish_lock_is_held.

  2. Offline MANIFEST reads. ManifestLog::read_checkpoint_readonly reads CURRENT and MANIFEST without opening the manifest for append and without truncating a torn tail. Offline backup uses it for the source LSM. offline_backup_does_not_mutate_source_manifests makes the source manifest files unwritable, checks that their bytes are unchanged, and still produces an artifact. The hold requirement is unchanged (offline_backup_without_a_hold_is_rejected).

  3. Crash scratch. Before an online checkpoint, a plain directory named exactly .backup-ckpt- plus a 20-digit shard id is removed. A symlink, a non-directory, or any other name is left in place. Covered by crash_checkpoint_scratch_is_removed_without_touching_other_names. Cleanup, create_checkpoint, the byte copy, and removal share .backup-ckpt-<shard>.lock. BACKUP_JOB.lock only serializes the maintenance-loop job file, so a direct second create_backup could otherwise delete the directory the first call was still copying. Covered by checkpoint_scratch_is_not_cleared_while_another_backup_holds_it.

CLUSTER_IDENTITY stays mandatory, as landed in #25. Backup fails when a cell directory with CELL_MANAGER_STATE has no CLUSTER_IDENTITY. Restore fails when the manifest lists that state file without the identity file. That is separate from a probe that starts the restored node while the original process is still bound to node 1.

Tests

Locally passed after the rebase:

  • cargo fmt --check --all
  • cargo test --locked -p metadb-raft --lib backup:: (11 passed)
  • cargo test --locked -p metadb-server --test backup_restore (5 passed, including the existing hold, release, expiry, restore Lookup/List, truncated-artifact, and read-only source manifest cases)
  • cargo test --locked -p metadb-server --test restore_safety (2 passed)

These passes are not a first-ship pass. CI was not waited on.

Leftovers

  • Still one data directory. Not a multi-Cell or Jumbo backup, and not a consistent cut across a multi-process replica set.
  • No independent Raft WAL. A missing log, snapshot, and LSM together are not reconstructed.
  • A process that creates a shard directory or rewrites the registry without taking SHARD_SET.lock can still race the rename. The in-tree creators take the lock.
  • Each file is still read into memory while it is copied.
  • A symlink left at .backup-ckpt-<shard> is not deleted; the next online checkpoint for that shard fails until an operator removes it.
  • Two backups of different shards do not share a scratch lock. Two backups of the same shard do.
  • Starting a restored directory while another live process is still bound to the same node id is still rejected by the cluster identity rule. That rejection is not a statement about whether the backup bytes can be opened on their own.
Open in Web Open in Cursor 

Copilot AI balanced review requested due to automatic review settings November 19, 2026 11:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent backup calls can delete each other’s active checkpoint because locking does not cover scratch cleanup through copying and removal, causing missing-file failures.

1 open issue
What changed in this PR

This PR hardens the existing single-data-directory backup path without changing the hold/create/release/restore control plane.

Changes:

  • Coordinates backup publication with shard creation, registry updates, and cell-state writes.
  • Adds read-only offline MANIFEST reads and copies cluster identity for restore.
  • Cleans crash checkpoint scratch directories and adds regression tests and operational guidance.
File Description
docs/​en/​operations.md Documents publication locking, identity preservation, and scratch cleanup.
crates/​metadb-storage/​src/​manifest.rs Adds read-only checkpoint reads and shares manifest replay logic.
crates/​metadb-server/​tests/​backup_restore.rs Tests offline backups with unwritable source manifests.
crates/​metadb-raft/​src/​state_machine.rs Locks shard-directory creation.
crates/​metadb-raft/​src/​group_host.rs Locks registry updates.
crates/​metadb-raft/​src/​backup.rs Adds locked publication, scratch cleanup, identity copying, and regression tests.
crates/​metadb-cluster/​src/​split_worker.rs Locks split-child directory publication.
crates/​metadb-cluster/​src/​cell_manager.rs Coordinates cell-state writes with backup publication.

Review effort: Balanced

Comment thread crates/metadb-raft/src/backup.rs Outdated
)));
}
let temp = data_dir.join(checkpoint_scratch_name(shard.shard_id));
clear_checkpoint_scratch(data_dir, shard.shard_id)?;
cursoragent and others added 2 commits November 19, 2026 14:20
Shard creation, registry updates, and cell-state writes share a lock with
the final backup check and rename, so a shard that appears in that window
fails the backup instead of being omitted. Offline backup reads MANIFEST
without append access. A plain .backup-ckpt-<shard> directory left by a
crash is removed before the next online checkpoint. CLUSTER_IDENTITY is
copied so a restored directory keeps its node instance id.

Co-authored-by: RustFS <[email protected]>
Concurrent online backups of the same shard share .backup-ckpt-<shard>.lock
from crash-residue cleanup through checkpoint creation, copying, and removal,
so one call cannot delete the directory the other is still reading.

Co-authored-by: RustFS <[email protected]>
@cursor
cursor Bot force-pushed the cursor/backup-restore-leftovers-c11b branch from 3b206b1 to 8a00dc3 Compare November 19, 2026 14:29
@loverustfs
loverustfs merged commit 3ed51ee into main Nov 19, 2026
11 of 13 checks passed
loverustfs added a commit that referenced this pull request Nov 19, 2026
Rebuild a missing published snapshot from the LSM, or fail closed with snapshot unrecoverable so a purged shard is not advertised as a live voter and does not accept writes onto an empty LSM. Start checks the unrecoverable latch before register_group. routes_for_live_status probes the snapshot directory through ensure_published_snapshot_sync. There is no fourth copy of the command log.

This branch is one commit behind main after #19. A local merge into 3ed51ee has no text conflicts and keeps both #19's SHARD_SET publish lock and #25's identity and shutdown bounds.

A merge is not a first-ship pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants