Repository navigation
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The script can attempt reconstruction before peers re-admit the replacement, causing spurious failures, and cleanup timeouts can leave resources and .runtime.env behind.
2 open issues
What changed in this PR
Adds regression coverage for replacing a failed RustFS member with an empty node, extending existing outage tests to verify recovery of writes acknowledged while the member was offline.
Changes:
- Adds a Docker-based four-node, one-drive-per-node scenario with physical shard checks and forced reconstruction through the replacement.
- Adds a manually dispatched workflow, evidence collection, and ownership-scoped cleanup.
- Documents usage and adds regression tests to the existing fault-tolerance checks.
| File | Description |
|---|---|
test_node_replacement.py |
Tests evidence validation, reconstruction controls, topology, and cleanup ownership. |
scripts/node_replacement.py |
Implements the replacement scenario, shard verification, evidence capture, and cleanup. |
README.md |
Documents the scenario, local execution, evidence, and interrupted-run recovery. |
.github/workflows/rustfs-node-replacement-test.yml |
Adds manual integration execution, cleanup, and artifact upload. |
.github/workflows/fault-tolerance-script-tests.yml |
Adds replacement-related triggers and runs the new regression tests. |
Review effort: Balanced
Comment on lines
+361
to
+362
| self.wait_ready(verifier) | ||
| self.read(verifier, "replacement-required-read") |
Comment on lines
+386
to
+404
| result = subprocess.run(["docker", kind, "inspect", name], capture_output=True, text=True, timeout=30) | ||
| if result.returncode: | ||
| # A missing resource is recoverable. Daemon/transport failures remain failures. | ||
| missing = {"container": f"no such container: {name}", | ||
| "volume": f"get {name}: no such volume", | ||
| "network": f"network {name} not found"} | ||
| if missing[kind] in result.stderr.lower(): | ||
| continue | ||
| failures.append({"kind": kind, "name": name, "error": "inspect failed"}) | ||
| continue | ||
| info = json.loads(result.stdout)[0] | ||
| labels = info.get("Labels") if kind != "container" else info.get("Config", {}).get("Labels") | ||
| if (labels or {}).get(LABEL) != resources["runId"]: | ||
| failures.append({"kind": kind, "name": name, "error": "ownership label differs"}) | ||
| continue | ||
| arguments = ["docker", kind, "rm", *(["--force"] if kind == "container" else []), name] | ||
| removed = subprocess.run(arguments, capture_output=True, text=True, timeout=90) | ||
| if removed.returncode: | ||
| failures.append({"kind": kind, "name": name, "error": "removal failed"}) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

The previous directory-backed fixture enabled
RUSTFS_UNSAFE_BYPASS_DISK_CHECK, allowing replacement recovery through the directory fallback. Replace it with mounted XFS filesystems and enforce bothRUSTFS_UNSAFE_BYPASS_DISK_CHECK=falseandMINIO_CI=falsein the actual service processes to cover backlog #3283.The default fixture uses four nodes with four independent XFS loop devices each, one 16-drive pool/set, EC:4, and
mnmddirectories below the mountpoints. The original 4x1 topology is also supported. Rebuilding node 2 archives its stopped endpoints, preserves the filesystem identities, and starts a fresh process with empty data directories and the original configuration.Changes
format.json,pool.bin/xl.meta,pool.bin.identity/xl.meta, object metadata and external shards on every replacement disk. Decide automatic physical repair before recovered-object GETs. Successful survivor-backed reads retain an already failed physical verdict.Live validation
All runs used the official GNU x86_64 RustFS 1.0.5 binary with SHA-256
316894c467167288384c0a8512deba4b7d5be0adc301670d29c79730c5029198, matching #3283. Both bypass flags were observed asfalsein/proc/<pid>/environ.Each failing subdirectory run recorded 54
target mount admission failedoccurrences in node 2's file log and zerodirectory_backed_fallbackoccurrences. The exact-mountpoint control recorded neither signature. Cleanup was rerun successfully and no owned loop devices remained.Local execution used Docker Desktop's Linux kernel on an ARM64 host with the GNU x86_64 binary under architecture emulation. The data filesystems were actual XFS mounts over dedicated loop-backed image files.
Checks
PYTHONPATH=scripts python3 -B -m unittest -q test_node_replacement.py test_rustfs_fault_tolerance.py test_functional_dispatch test_prepare_functional_packagepassed with 62 passing tests and one platform skip.actionlintpassed for both affected workflows.git diff --checkpassed.