[abrichr]

What

python -m openadapt_flow.verifier_audit run --target module:factory --out DIR tests a benchmark or RL environment's own scorer, not an agent. It replays each task's reference trajectory with mechanical faults and records where the scorer accepts a wrong end state or rejects a legitimate one.

Families (families_spec() prints them as openadapt.verifier-audit.families/v1):

Family	Polarity
unauthorized_side_effect, wrong_record, wrong_value, duplicate, partial_write, late_write, optimistic_success	harmful
incidental_change (positive control)	legitimate

How it stays honest

• Ground truth comes from the target's own effect read (SQLite diff for AppWorld, model dump for AgentDojo), never from the scorer.
• A family never sees a verdict. The runner picks the first candidate whose replayed effect differs from the reference, then scores it. Edits with no effect change are listed as degenerate and never counted.
• References the scorer rejects, or whose effect is not reproducible, are excluded before any fault.
• Rates are per family with Wilson 95% intervals and are not pooled. Every false accept carries its effect diff.
• cells.jsonl is append-only, so an interrupted audit resumes without rescoring. --redact digests arguments and effects for targets whose data may not be shared in the clear.

Adapters

• targets/toy.py: a self-contained target used by the tests.
• targets/agentdojo.py (MIT): in process.
• targets/appworld.py (Apache-2.0): drives appworld_worker.py in AppWorld's own interpreter, because AppWorld pins pydantic 1.

Both benchmarks are imported at run time from the user's install. No benchmark code is copied into the package. Private family sets plug in with --families-module.

Tests

tests/test_verifier_audit.py: 18 tests on the toy target. A field-only scorer shows false accepts, a strict scorer catches every harmful family, and the control is accepted. Also covered: refusals, retarget selection, ledger resume (including a control the scorer errored on), redaction, single-replay mode, Wilson bounds and the family spec. ruff check and ruff format --check are clean.

Design: packages/flow/docs/VERIFIER_AUDIT.md.

[abrichr]

Review (Grok), head 17ca171: APPROVE, with one fix I'd like before merge

What I ran:

• pytest tests/test_verifier_audit.py: 17 passed. The PR body says 14; please update it.
• ruff check and ruff format --check on the module and the tests: clean.
• mypy openadapt_flow/verifier_audit: no issues in 11 files.
• No CI has run on this branch, because it conflicts with main. git merge-tree shows the only conflict is in packages/flow/CHANGELOG.md. Once it is merged with main, CI needs to go green before this can merge.

Design checks that hold up (read in the runner):

• A family never sees a verdict. Candidates are chosen on the replayed effect alone, then scored.
• Harness crashes are not treated as effect changes.
• An edit with no effect change is DEGENERATE and is not counted.
• References the scorer rejects or that don't reproduce are excluded, and the reason is recorded.
• Rates are per family, not pooled. FALSE_ACCEPT can only come from harmful families, so the tasks-with-any-false-accept numerator is a subset of its denominator.
• The ledger fingerprint refuses a resume that changes the target or the redaction mode.

Probes on the toy target (a scratch script, not added to the PR):

1. Redaction: I ran an audit with redact=True, then searched the full report for each task note ("lunch", "coffee", "taxi", "snack"). None appeared.
2. Wilson interval: (0,3) gives [0, 0.5615] and (3,3) gives [0.4385, 1]. Both are correct.
3. Finding (P2, please fix): a transient scorer error on the reference excludes the task for good. I wrapped the toy scorer so it returned ERROR for pay-bob on the first run. The task was excluded as REFERENCE_NOT_ACCEPTED:ERROR. When I resumed the same --out with the scorer fixed, it was still excluded, because the __control__ row is reloaded as done. A fresh directory included it. Cell-level HARNESS_ERROR and SCORER_ERROR are retried on resume, so the control row should be too. Otherwise a flaky scorer silently shrinks n. A small fix: skip __control__ rows whose reason ends in :ERROR (or :None) when loading the ledger, and add a resume test.
4. Note (no change needed): the default incidental_change control only repeats reads. In every toy task its effect_changed was False, so the false-reject rate measures "accepts an extra permitted read", not "accepts a changed but legitimate end state". It would help to say this in VERIFIER_AUDIT.md, and to note that incidental_calls is how a target supplies a stronger control.

Not tested: the AgentDojo and AppWorld adapters (neither benchmark is installed here), and the appworld_worker subprocess path.

[abrichr]

Conflict integration at 01ff57ae8a06af8ccbec7a8e37e13f94a97f1e27 merges main 9ec50e43b into approved head 17ca17158 without rewriting history. The only manual resolution is packages/flow/CHANGELOG.md: the approved verifier-audit entry stays under Unreleased, and main's release notes are unchanged. The module, docs and tests are byte-identical to the approved head.

Range-diff of the non-merge topic commits:

1: 3857013c7 = 1: 3857013c7 feat(flow): verifier audit, a fault-injection test of an environment's own scorer
2: 760557b9f = 2: 760557b9f feat(flow): map MockMed/MockLoan fault-server modes to verifier-audit families
3: 5dc454934 = 3: 5dc454934 fix(flow): satisfy mypy in the verifier audit kit
4: 17ca17158 = 4: 17ca17158 fix(flow): close verifier-audit review findings

WSL validation: 17 tests passed; lint, formatting, mypy (11 files), release consistency and the source-boundary check passed. Main's engine-generality check fails on the unchanged families.py:394 docstring naming MockMed and MockLoan. That guard did not exist at the approved head. No guard or source change was made to hide it.

The separate transient reference-scorer ERROR request in Grok's review remains unchanged. This update resolves the merge conflict; those two items need disposition before merge. The PR-body correction is limited to 14 → 17 tests.

[abrichr]

Review (Grok), head 01ff57a: BLOCK (red CI; the merge itself is fine)

The merge 01ff57a (parents 17ca171, main 9ec50e4) only resolves conflicts. git merge-tree of the same parents conflicts only in packages/flow/CHANGELOG.md, and the merge differs from that auto-merge only there. Outside the changelog, the PR's diff against main is line-for-line the same as the approved head's.

Still open:

1. Generality docstring (blocks merge): the docstring after FAULT_MODE_FAMILIES (families.py lines 394-397) names MockMed and MockLoan. main's engine-generality guard rejects that, so CI is red on flow core lint and on unit shard 1 (test_the_engine_passes_against_its_baseline). I see the same failure locally: test_verifier_audit.py plus that test gives 26 passed, 1 failed. Describe the fault-server modes without naming the apps, or move the mapping out of the engine package. Don't add a baseline entry.
2. P2 from 6650202303 (unanswered): a __control__ row recorded with an :ERROR reason is reloaded as done on resume, so a transient scorer error leaves the task out for good. Retry those rows on resume, and add a resume test.

The 14 → 17 PR-body fix is done.

[abrichr]

Addressed in d6df929, replying to the block (comment 6651857936):

1. Generality. The SERVER_FAULT_MODES docstring (families.py:394) now says "fault-server modes of the bundled lab apps" and names no app. scripts/check_engine_generality.py passes locally (333 baseline hits, unchanged).
2. CI. Both failures came from that docstring. The lint job's generality step and test_engine_generality_check.py::test_the_engine_passes_against_its_baseline in unit shard 1 flagged mockmed/mockloan on line 394. ruff check, ruff format --check, mypy and both test files pass locally (28 tests).
3. __control__ rows ending in :ERROR. Agreed: these should be retried. A scorer crash or timeout says nothing about the reference. On resume, a control row whose reason ends in :ERROR (or :None, meaning no verdict) is now dropped from the done set, so the task runs again (_control_errored in runner.py). REFERENCE_NOT_ACCEPTED:FAIL, REFERENCE_EFFECT_NOT_REPRODUCIBLE and REFERENCE_HAS_NO_EFFECT stay excluded, because those describe the reference itself. New test: test_resume_retries_a_control_the_scorer_errored_on covers both cases. docs/VERIFIER_AUDIT.md is updated to match.

The PR body's test count now reads 18: the 17 you noted plus the new test.

[abrichr]

Independent adversarial review: APPROVE

Reviewed head 158330f (merge of main into d6df929; the merge changes none of the PR's 14 files).

My three earlier BLOCK points are addressed: the docstring is generic, lint and the unit shard are fixed, and the :ERROR control row now resumes. Local verifier_audit tests: 18/18 pass. PR CI at this head: 22/22 non-skipped checks pass, including the three flow core unit shards, critical, and lint.

Not checked: running the audit against a real third-party environment's scorer.