Repository navigation
Close backup publish, manifest, and scratch leftovers - #19
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent backup calls can delete each other’s active checkpoint because locking does not cover scratch cleanup through copying and removal, causing missing-file failures.
1 open issue
What changed in this PR
This PR hardens the existing single-data-directory backup path without changing the hold/create/release/restore control plane.
Changes:
- Coordinates backup publication with shard creation, registry updates, and cell-state writes.
- Adds read-only offline MANIFEST reads and copies cluster identity for restore.
- Cleans crash checkpoint scratch directories and adds regression tests and operational guidance.
| File | Description |
|---|---|
docs/en/operations.md |
Documents publication locking, identity preservation, and scratch cleanup. |
crates/metadb-storage/src/manifest.rs |
Adds read-only checkpoint reads and shares manifest replay logic. |
crates/metadb-server/tests/backup_restore.rs |
Tests offline backups with unwritable source manifests. |
crates/metadb-raft/src/state_machine.rs |
Locks shard-directory creation. |
crates/metadb-raft/src/group_host.rs |
Locks registry updates. |
crates/metadb-raft/src/backup.rs |
Adds locked publication, scratch cleanup, identity copying, and regression tests. |
crates/metadb-cluster/src/split_worker.rs |
Locks split-child directory publication. |
crates/metadb-cluster/src/cell_manager.rs |
Coordinates cell-state writes with backup publication. |
Review effort: Balanced
| ))); | ||
| } | ||
| let temp = data_dir.join(checkpoint_scratch_name(shard.shard_id)); | ||
| clear_checkpoint_scratch(data_dir, shard.shard_id)?; |
This was referenced Nov 19, 2026
Shard creation, registry updates, and cell-state writes share a lock with the final backup check and rename, so a shard that appears in that window fails the backup instead of being omitted. Offline backup reads MANIFEST without append access. A plain .backup-ckpt-<shard> directory left by a crash is removed before the next online checkpoint. CLUSTER_IDENTITY is copied so a restored directory keeps its node instance id. Co-authored-by: RustFS <[email protected]>
Concurrent online backups of the same shard share .backup-ckpt-<shard>.lock from crash-residue cleanup through checkpoint creation, copying, and removal, so one call cannot delete the directory the other is still reading. Co-authored-by: RustFS <[email protected]>
cursor
Bot
force-pushed
the
cursor/backup-restore-leftovers-c11b
branch
from
November 19, 2026 14:29
3b206b1 to
8a00dc3
Compare
loverustfs
added a commit
that referenced
this pull request
Nov 19, 2026
Rebuild a missing published snapshot from the LSM, or fail closed with snapshot unrecoverable so a purged shard is not advertised as a live voter and does not accept writes onto an empty LSM. Start checks the unrecoverable latch before register_group. routes_for_live_status probes the snapshot directory through ensure_published_snapshot_sync. There is no fourth copy of the command log. This branch is one commit behind main after #19. A local merge into 3ed51ee has no text conflicts and keeps both #19's SHARD_SET publish lock and #25's identity and shutdown bounds. A merge is not a first-ship pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Closes three leftovers from the operator backup path that is already on
main. This branch does not change the hold / create / release / restore control plane.backup createstill fails withinstall a hold before copying so purge cannot remove the imagewhen no active hold is installed.Rebased onto
fe9c94ea(currentmain, including #25, #18, #21, #22, and #23). The only conflict wascrates/metadb-raft/src/backup.rs, incopy_node_metadata. Resolution keeps both sides:CLUSTER_IDENTITYand refuses a cell directory that lacks it. Restore refuses a tree whose manifest listsCELL_MANAGER_STATEwithout the matchingCLUSTER_IDENTITY.MANIFESTreads, and crash-scratch cleanup. The scratch lock is held through the byte copy.The optional copy that this branch had before the rebase (copy
CLUSTER_IDENTITYonly when the file already exists, and allow restore without it) was not kept. #25's checks are unchanged.A green CI run is not a first-ship pass. This is the independent RustFS implementation. Paper figures in arXiv:2610.32247v1 are Baidu production numbers, not results from this repository.
What changed
Publish window.
be/SHARD_SET.lockis held from the final live shard-directory and registry check through the artifact rename. Shard directory creation, split publication, registry updates, andcell_*state writes take the same lock. A shard that shows up before that locked check fails the backup. A creator that takes the lock cannot create the directory between the check and the rename. Covered bya_shard_created_before_rename_rejects_publicationandshard_directory_cannot_appear_while_the_publish_lock_is_held.Offline MANIFEST reads.
ManifestLog::read_checkpoint_readonlyreadsCURRENTandMANIFESTwithout opening the manifest for append and without truncating a torn tail. Offline backup uses it for the source LSM.offline_backup_does_not_mutate_source_manifestsmakes the source manifest files unwritable, checks that their bytes are unchanged, and still produces an artifact. The hold requirement is unchanged (offline_backup_without_a_hold_is_rejected).Crash scratch. Before an online checkpoint, a plain directory named exactly
.backup-ckpt-plus a 20-digit shard id is removed. A symlink, a non-directory, or any other name is left in place. Covered bycrash_checkpoint_scratch_is_removed_without_touching_other_names. Cleanup,create_checkpoint, the byte copy, and removal share.backup-ckpt-<shard>.lock.BACKUP_JOB.lockonly serializes the maintenance-loop job file, so a direct secondcreate_backupcould otherwise delete the directory the first call was still copying. Covered bycheckpoint_scratch_is_not_cleared_while_another_backup_holds_it.CLUSTER_IDENTITYstays mandatory, as landed in #25. Backup fails when a cell directory withCELL_MANAGER_STATEhas noCLUSTER_IDENTITY. Restore fails when the manifest lists that state file without the identity file. That is separate from a probe that starts the restored node while the original process is still bound to node 1.Tests
Locally passed after the rebase:
cargo fmt --check --allcargo test --locked -p metadb-raft --lib backup::(11 passed)cargo test --locked -p metadb-server --test backup_restore(5 passed, including the existing hold, release, expiry, restore Lookup/List, truncated-artifact, and read-only source manifest cases)cargo test --locked -p metadb-server --test restore_safety(2 passed)These passes are not a first-ship pass. CI was not waited on.
Leftovers
SHARD_SET.lockcan still race the rename. The in-tree creators take the lock..backup-ckpt-<shard>is not deleted; the next online checkpoint for that shard fails until an operator removes it.