Skip to content

test: cover XFS node rebuilds with disk checks enabled - #155

Open
cxymds wants to merge 2 commits into
mainfrom
cxymds/test-node-replacement
Open

cxymds wants to merge 2 commits into
mainfrom
cxymds/test-node-replacement

Conversation

@cxymds

@cxymds cxymds commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The previous directory-backed fixture enabled RUSTFS_UNSAFE_BYPASS_DISK_CHECK, allowing replacement recovery through the directory fallback. Replace it with mounted XFS filesystems and enforce both RUSTFS_UNSAFE_BYPASS_DISK_CHECK=false and MINIO_CI=false in the actual service processes to cover backlog #3283.

The default fixture uses four nodes with four independent XFS loop devices each, one 16-drive pool/set, EC:4, and mnmd directories below the mountpoints. The original 4x1 topology is also supported. Rebuilding node 2 archives its stopped endpoints, preserves the filesystem identities, and starts a fresh process with empty data directories and the original configuration.

Changes

  • Verify the published official GNU archive digest and recheck the runtime binary before each process start. Record XFS UUIDs, device identities, endpoint mount relationships, process identities and observed bypass flags.
  • Seed 100 one-KiB objects and three large objects, then acknowledge three further large PUTs during the outage.
  • Independently assert format.json, pool.bin/xl.meta, pool.bin.identity/xl.meta, object metadata and external shards on every replacement disk. Decide automatic physical repair before recovered-object GETs. Successful survivor-backed reads retain an already failed physical verdict.
  • After repair, disable healing and quarantine survivor external parts and inline metadata so payload verification requires replacement shards. Separately witness cold-start pool metadata failures without the replacement, then require successful startup and reads with its restored metadata.
  • Preserve physical archives, health responses and logs. Recover only owned resources after interruption, with loop-backed images confined to private container mount namespaces.
  • Update the manual workflow, script regression checks and documentation. Add the concurrency setting required by the existing workflow contract.

Live validation

All runs used the official GNU x86_64 RustFS 1.0.5 binary with SHA-256 316894c467167288384c0a8512deba4b7d5be0adc301670d29c79730c5029198, matching #3283. Both bypass flags were observed as false in /proc/<pid>/environ.

Topology Endpoint layout Result Observation
4x4 XFS mountpoint subdirectory FAIL, cleanup PASS All 106 acknowledged objects read back through four endpoints. After the 90-second physical repair budget, the replacement still lacked format and pool metadata and had incomplete shards before GET.
4x1 XFS mountpoint subdirectory FAIL, cleanup PASS The same failure occurred in the originally requested topology.
4x1 Exact endpoint bind mount on XFS PASS, cleanup PASS Pool metadata and shards recovered automatically. All 106 objects passed 424 forced-reconstruction reads. The cold-start metadata negative and recovery controls also passed.

Each failing subdirectory run recorded 54 target mount admission failed occurrences in node 2's file log and zero directory_backed_fallback occurrences. The exact-mountpoint control recorded neither signature. Cleanup was rerun successfully and no owned loop devices remained.

Local execution used Docker Desktop's Linux kernel on an ARM64 host with the GNU x86_64 binary under architecture emulation. The data filesystems were actual XFS mounts over dedicated loop-backed image files.

Checks

  • PYTHONPATH=scripts python3 -B -m unittest -q test_node_replacement.py test_rustfs_fault_tolerance.py test_functional_dispatch test_prepare_functional_package passed with 62 passing tests and one platform skip.
  • actionlint passed for both affected workflows.
  • git diff --check passed.

Copilot AI balanced review requested due to automatic review settings November 19, 2026 08:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The script can attempt reconstruction before peers re-admit the replacement, causing spurious failures, and cleanup timeouts can leave resources and .runtime.env behind.

2 open issues
What changed in this PR

Adds regression coverage for replacing a failed RustFS member with an empty node, extending existing outage tests to verify recovery of writes acknowledged while the member was offline.

Changes:

  • Adds a Docker-based four-node, one-drive-per-node scenario with physical shard checks and forced reconstruction through the replacement.
  • Adds a manually dispatched workflow, evidence collection, and ownership-scoped cleanup.
  • Documents usage and adds regression tests to the existing fault-tolerance checks.
File Description
test_node_replacement.py Tests evidence validation, reconstruction controls, topology, and cleanup ownership.
scripts/​node_replacement.py Implements the replacement scenario, shard verification, evidence capture, and cleanup.
README.md Documents the scenario, local execution, evidence, and interrupted-run recovery.
.github/​workflows/​rustfs-node-replacement-test.yml Adds manual integration execution, cleanup, and artifact upload.
.github/​workflows/​fault-tolerance-script-tests.yml Adds replacement-related triggers and runs the new regression tests.

Review effort: Balanced

Comment thread scripts/node_replacement.py Outdated
Comment on lines +361 to +362
self.wait_ready(verifier)
self.read(verifier, "replacement-required-read")
Comment thread scripts/node_replacement.py Outdated
Comment on lines +386 to +404
result = subprocess.run(["docker", kind, "inspect", name], capture_output=True, text=True, timeout=30)
if result.returncode:
# A missing resource is recoverable. Daemon/transport failures remain failures.
missing = {"container": f"no such container: {name}",
"volume": f"get {name}: no such volume",
"network": f"network {name} not found"}
if missing[kind] in result.stderr.lower():
continue
failures.append({"kind": kind, "name": name, "error": "inspect failed"})
continue
info = json.loads(result.stdout)[0]
labels = info.get("Labels") if kind != "container" else info.get("Config", {}).get("Labels")
if (labels or {}).get(LABEL) != resources["runId"]:
failures.append({"kind": kind, "name": name, "error": "ownership label differs"})
continue
arguments = ["docker", kind, "rm", *(["--force"] if kind == "container" else []), name]
removed = subprocess.run(arguments, capture_output=True, text=True, timeout=90)
if removed.returncode:
failures.append({"kind": kind, "name": name, "error": "removal failed"})
@cxymds cxymds changed the title test: cover empty-node replacement in a four-drive pool test: cover XFS node rebuilds with disk checks enabled Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants