Skip to content

test(scanner): add usage regression probes and native evidence gates - #153

Merged
cxymds merged 4 commits into
mainfrom
cxymds/test-usage-convergence
Nov 13, 2026
Merged

cxymds merged 4 commits into
mainfrom
cxymds/test-usage-convergence

Conversation

@cxymds

@cxymds cxymds commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

The existing scanner smoke test covers one fixed 20-object snapshot. Add S3/Admin usage probes and candidate-native assertion gates so cross-cycle, publication, watch and compatibility scenarios report their actual evidence and missing coverage separately.

Related to rustfs/backlog#3178. Companion RustFS implementation: rustfs/rustfs#9397.

The new scanner-usage-regressions module registers USAGE-01 through USAGE-20 as a local catalog extension, preserving the 158-case source snapshot and the original 20-object/20,690-byte smoke test. Public probes retain acknowledged operations, independent paginated inventories, payload readback, service errors, per-bucket metadata and scoped cleanup ownership. Native tests require an exact candidate binary hash, matching source identity fields, one real non-ignored test, complete assertion markers and successful cleanup. Partial coverage cannot pass a case.

Execution uses explicitly configured endpoints and creates only invocation-owned buckets. Unacknowledged creates remain visible cleanup failures. The new module enters scheduling through explicit selection.

Validation

  • python3 -B -m unittest -q test_validation_scanner_usage.py test_validation_migration.py: 63 passed on 428e861, covering native evidence failures, diagnostic-write failures, cancellation, request/cleanup ownership, catalog migration and reporting.
  • Existing ILM, pool, replication, Heal, OIDC and webhook suites: 214 passed. Functional-dispatch tests: 16 passed. These earlier checks were not rerun as part of the final 63-test focused run.
  • Catalog validation, complete migration-review gate, workflow actionlint and shell syntax checks passed.
  • The original scanner smoke tool test passed its exact 20-object/20,690-byte oracle and negative cases.
  • The source catalog remains 158 cases with SHA-256 463fdd02dee43e42fef4f3aacf7506709ce87af535e1a53d91c521bc2e361cad.

The 2026-11-13 09:55:29 UTC snapshot confirms both CI checks passed for auto-testing PR #153 on 428e861 (MERGEABLE / CLEAN); all seven tooling review threads were resolved.

The accompanying product candidate is 31140704a5610e234c2c019bfa23800c74718377, based on main 0fde0f14 and its upstream #9353 reversion while retaining usage-watch bindings and the Windows review correction. Its four-package Clippy passed (21m 56s), and its four-package no-run build passed (17m 14s) from a clean working tree while reusing the shared target cache; no cargo clean or full uncached rebuild was performed. Current-candidate targeted nextest passed 151/151, with 9,127 skipped, in 10.295s. Scope: rustfs-protos, usage watches, namespace mutation, the legacy GET producer stack regression and ecstore runtime source/locking tests; this was not a workspace-wide run.

The actual adapter used this candidate's hashed libtest binaries and ran 24 unique exact tests, each one passed / zero failed / zero ignored, exit 0 and no timeout. All 27 assertion markers and 24 native cleanup markers passed. The complete report retains 20 cases: 2 PASS / 18 UNSUPPORTED and 59 assertions: 27 PASS / 32 UNSUPPORTED. The actual adapter exit was 1, reflecting 18 unbound native assertions and 14 unexecuted public assertions. All 20 outer cleanup results passed because no public fixture was created; these results do not establish VM cleanup. Historical 78902eb0f completed 1,891 selected product regressions and native 24 exact / 27 assertions; that wider scope was not rerun on the current candidate.

At 2026-11-13 09:58:03 UTC, product PR #9397's remote HEAD matched 31140704a and remained MERGEABLE / BLOCKED, REVIEW_REQUIRED; CI had no failure in the snapshot but four checks still reported IN_PROGRESS, so full product CI success is not established. The old 78902eb0f approval does not approve the new HEAD. No local Windows-target, public API, VM, mixed-version, A/B or production acceptance is claimed.

Implementation and acceptance scope

This PR provides 14 public assertions and a strict native adapter. Twenty registered contracts do not establish twenty completed acceptance cases. At the accompanying product source review, 27 of 45 native assertions have reviewed test bindings; the remaining 18 lack complete reviewed bindings and execution evidence. Build-specific manifests and native execution results must use the committed product source and actual executable hashes.

Multi-node C5 coverage, crash recovery, wide-directory scale, mixed-version rollback and matched load measurements remain acceptance gaps. Missing capabilities remain BLOCKED or UNSUPPORTED, with any executed assertions retained individually. This PR does not close the parent acceptance issue.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 05:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical flaws can disclose signed credentials through redirects and persist endpoint secrets in artifacts, alongside probe correctness and reliability defects.

7 open issues
What changed in this PR

Adds scanner usage regression tooling alongside the existing smoke test, separating public API evidence from candidate-native assertions and remaining acceptance gaps.

Changes:

  • Registers USAGE-01 through USAGE-20 as explicitly selected local cases without changing the 158-case source snapshot.
  • Adds S3/Admin probes, native evidence validation, scoped cleanup, and assertion-level reporting.
  • Integrates catalog support, regression tests, CI checks, and execution documentation.
File Description
validation-migration-review.json Updates the canonical runner’s reviewed hash.
validation-cases.json Registers twenty local scanner usage contracts.
test_validation_scanner_usage.py Tests probes, evidence gates, cleanup, and registration.
test_validation_migration.py Excludes local extensions from source-snapshot comparison.
scripts/​validation_scanner_usage/​runner.py Implements public scenarios, orchestration, cleanup, and reports.
scripts/​validation_scanner_usage/​run_case.sh Adds the canonical case entry point.
scripts/​validation_scanner_usage/​native.py Validates native test identity, execution, assertions, and cleanup.
scripts/​validation_scanner_usage/​contracts.py Defines case contracts and assertion identifiers.
scripts/​validation_scanner_usage/​client.py Adds S3/Admin clients and usage metadata validation.
scripts/​validation_scanner_usage/​__init__.py Introduces the scanner usage package.
scripts/​validation_result.py Enables result lookup for local cases.
scripts/​validation_report.py Includes local cases in report validation.
scripts/​validation_catalog.py Validates and preserves local catalog extensions.
scripts/​select_validation_cases.py Allows explicit local-case selection without implicit rotation.
scripts/​run_validation_case.sh Dispatches scanner usage cases to their runner.
rustfs-scanner-usage-test.sh Adds the standalone suite launcher.
README.md Introduces the suite and clarifies provenance limits.
docs/​scanner-usage-validation.md Documents execution, contracts, evidence, and acceptance gaps.
.github/​workflows/​validation-catalog-tests.yml Adds suite triggers, unit tests, and shell syntax checks.

Review effort: Balanced

@cxymds
cxymds merged commit f8568ae into main Nov 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants