Repository navigation
test(scanner): add usage regression probes and native evidence gates - #153
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical flaws can disclose signed credentials through redirects and persist endpoint secrets in artifacts, alongside probe correctness and reliability defects.
7 open issues
Prevent redirects from forwarding signed credentials · New Validate endpoints before writing run configuration · New Validate usage state type before checking recognized values · New Normalize launch errors while preserving completed records · New Mark cleanup unverified when native execution is interrupted · New Avoid quadratic request journal rewrites · New Seed at least one object through every endpoint · New
What changed in this PR
Adds scanner usage regression tooling alongside the existing smoke test, separating public API evidence from candidate-native assertions and remaining acceptance gaps.
Changes:
- Registers USAGE-01 through USAGE-20 as explicitly selected local cases without changing the 158-case source snapshot.
- Adds S3/Admin probes, native evidence validation, scoped cleanup, and assertion-level reporting.
- Integrates catalog support, regression tests, CI checks, and execution documentation.
| File | Description |
|---|---|
validation-migration-review.json |
Updates the canonical runner’s reviewed hash. |
validation-cases.json |
Registers twenty local scanner usage contracts. |
test_validation_scanner_usage.py |
Tests probes, evidence gates, cleanup, and registration. |
test_validation_migration.py |
Excludes local extensions from source-snapshot comparison. |
scripts/validation_scanner_usage/runner.py |
Implements public scenarios, orchestration, cleanup, and reports. |
scripts/validation_scanner_usage/run_case.sh |
Adds the canonical case entry point. |
scripts/validation_scanner_usage/native.py |
Validates native test identity, execution, assertions, and cleanup. |
scripts/validation_scanner_usage/contracts.py |
Defines case contracts and assertion identifiers. |
scripts/validation_scanner_usage/client.py |
Adds S3/Admin clients and usage metadata validation. |
scripts/validation_scanner_usage/__init__.py |
Introduces the scanner usage package. |
scripts/validation_result.py |
Enables result lookup for local cases. |
scripts/validation_report.py |
Includes local cases in report validation. |
scripts/validation_catalog.py |
Validates and preserves local catalog extensions. |
scripts/select_validation_cases.py |
Allows explicit local-case selection without implicit rotation. |
scripts/run_validation_case.sh |
Dispatches scanner usage cases to their runner. |
rustfs-scanner-usage-test.sh |
Adds the standalone suite launcher. |
README.md |
Introduces the suite and clarifies provenance limits. |
docs/scanner-usage-validation.md |
Documents execution, contracts, evidence, and acceptance gaps. |
.github/workflows/validation-catalog-tests.yml |
Adds suite triggers, unit tests, and shell syntax checks. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


The existing scanner smoke test covers one fixed 20-object snapshot. Add S3/Admin usage probes and candidate-native assertion gates so cross-cycle, publication, watch and compatibility scenarios report their actual evidence and missing coverage separately.
Related to rustfs/backlog#3178. Companion RustFS implementation: rustfs/rustfs#9397.
The new
scanner-usage-regressionsmodule registers USAGE-01 through USAGE-20 as a local catalog extension, preserving the 158-case source snapshot and the original 20-object/20,690-byte smoke test. Public probes retain acknowledged operations, independent paginated inventories, payload readback, service errors, per-bucket metadata and scoped cleanup ownership. Native tests require an exact candidate binary hash, matching source identity fields, one real non-ignored test, complete assertion markers and successful cleanup. Partial coverage cannot pass a case.Execution uses explicitly configured endpoints and creates only invocation-owned buckets. Unacknowledged creates remain visible cleanup failures. The new module enters scheduling through explicit selection.
Validation
python3 -B -m unittest -q test_validation_scanner_usage.py test_validation_migration.py: 63 passed on428e861, covering native evidence failures, diagnostic-write failures, cancellation, request/cleanup ownership, catalog migration and reporting.463fdd02dee43e42fef4f3aacf7506709ce87af535e1a53d91c521bc2e361cad.The 2026-11-13 09:55:29 UTC snapshot confirms both CI checks passed for auto-testing PR #153 on
428e861(MERGEABLE/CLEAN); all seven tooling review threads were resolved.The accompanying product candidate is
31140704a5610e234c2c019bfa23800c74718377, based on main0fde0f14and its upstream #9353 reversion while retaining usage-watch bindings and the Windows review correction. Its four-package Clippy passed (21m 56s), and its four-package no-run build passed (17m 14s) from a clean working tree while reusing the shared target cache; nocargo cleanor full uncached rebuild was performed. Current-candidate targeted nextest passed 151/151, with 9,127 skipped, in 10.295s. Scope: rustfs-protos, usage watches, namespace mutation, the legacy GET producer stack regression and ecstore runtime source/locking tests; this was not a workspace-wide run.The actual adapter used this candidate's hashed libtest binaries and ran 24 unique exact tests, each one passed / zero failed / zero ignored, exit 0 and no timeout. All 27 assertion markers and 24 native cleanup markers passed. The complete report retains 20 cases: 2 PASS / 18 UNSUPPORTED and 59 assertions: 27 PASS / 32 UNSUPPORTED. The actual adapter exit was 1, reflecting 18 unbound native assertions and 14 unexecuted public assertions. All 20 outer cleanup results passed because no public fixture was created; these results do not establish VM cleanup. Historical
78902eb0fcompleted 1,891 selected product regressions and native 24 exact / 27 assertions; that wider scope was not rerun on the current candidate.At 2026-11-13 09:58:03 UTC, product PR #9397's remote HEAD matched
31140704aand remainedMERGEABLE/BLOCKED,REVIEW_REQUIRED; CI had no failure in the snapshot but four checks still reportedIN_PROGRESS, so full product CI success is not established. The old78902eb0fapproval does not approve the new HEAD. No local Windows-target, public API, VM, mixed-version, A/B or production acceptance is claimed.Implementation and acceptance scope
This PR provides 14 public assertions and a strict native adapter. Twenty registered contracts do not establish twenty completed acceptance cases. At the accompanying product source review, 27 of 45 native assertions have reviewed test bindings; the remaining 18 lack complete reviewed bindings and execution evidence. Build-specific manifests and native execution results must use the committed product source and actual executable hashes.
Multi-node C5 coverage, crash recovery, wide-directory scale, mixed-version rollback and matched load measurements remain acceptance gaps. Missing capabilities remain BLOCKED or UNSUPPORTED, with any executed assertions retained individually. This PR does not close the parent acceptance issue.