feat(flow): verifier audit, a fault-injection test of an environment's own scorer

packages/flow/CHANGELOG.md

@@ -25,6 +25,10 @@
 
 - The OpenEMR contracts (`benchmark/lifecycle_proof/verify.yaml`, `benchmark/openemr_local/verify.yaml`) now declare how the New Patient form stores what was typed: street and city in capitals, names with each word capitalized. Without it the exact record check refuted 19 of 20 correct saves on the stand-in suites (#1811). A contract value's `stored` form gains `capitalize_words` (OpenEMR's "C" fields); the set stays closed and the check stays exact.
 
+### Features
+
+- Verifier audit: `python -m openadapt_flow.verifier_audit run --target module:factory --out DIR` tests an environment's own scorer rather than an agent. It replays a reference trajectory with mechanical faults (an unauthorized side effect, the wrong record, a wrong value, a duplicate write, a partial write, a write after the success claim, a claim with no write) plus an incidental-change positive control. It reads the end state independently and reports, per family, where the scorer accepted a wrong end state or rejected a legitimate one, with counterexamples and Wilson 95% intervals. Adapters for AgentDojo and AppWorld ship as examples; private family sets plug in with `--families-module`. Design: `docs/VERIFIER_AUDIT.md`.
+
 ## v2.6.0 (2026-11-04)
 
 Flow adds signed teach authorizations, encrypted local artifacts, durable

packages/flow/docs/VERIFIER_AUDIT.md

@@ -0,0 +1,113 @@
+# Verifier audit
+
+A benchmark or RL environment turns a finished episode into a reward with its
+own scorer. If the scorer accepts a wrong end state, an agent trained or ranked
+on it learns or gets credit for that wrong behavior. The verifier audit tests
+the scorer, not an agent.
+
+The method comes from *Finding Blind Spots in AppWorld and WorkArena Task
+Verifiers* (Abrich, 2026). This module generalizes it to any environment that
+can reset, replay a trajectory, score, and read back its own state.
+
+## How a cell is built
+
+1. The target gives a **reference trajectory** for each task: steps the
+   environment's own scorer accepts. Typical sources are the shipped ground
+   truth or a recorded solution.
+2. The kit replays the reference twice from a fresh reset. A task is excluded
+   if the scorer rejects the reference, if the reference effect differs
+   between the two replays, or if it has no effect at all (an answer-only
+   task).
+3. Each **fault family** edits the reference mechanically. A family never sees
+   a scorer verdict.
+4. For each edit, the kit first runs a **census replay without scoring**. It
+   keeps the edit only if the environment accepted the edited call and the end
+   state differs from the reference. An edit whose effect matches the
+   reference (an idempotent duplicate, a field the app ignores) is
+   **degenerate** and is listed, never counted.
+5. The kept edit is replayed again from a fresh reset and scored once.
+
+Ground truth comes from the target's **effect read**: a diff of the end state
+against the start state, taken through a path that does not run the scorer
+(direct SQLite reads for AppWorld, a model dump for AgentDojo). A harmful-family
+cell whose effect differs from the reference and that the scorer accepts is a
+**false accept**. An incidental-change cell that the scorer rejects is a
+**false reject**.
+
+## Families
+
+| Family | Polarity | Edit |
+|---|---|---|
+| `unauthorized_side_effect` | harmful | Keep every task write, and add a copy of one aimed at a record the task never touches |
+| `wrong_record` | harmful | One task write lands on a different record of the same type |
+| `wrong_value` | harmful | One task write carries a changed non-identity value |
+| `duplicate` | harmful | One task write is sent twice |
+| `partial_write` | harmful | A multi-write task sends only its first write, or drops its last |
+| `late_write` | harmful | The last write is sent after the success claim, so the claim was false when made |
+| `optimistic_success` | harmful | Reads and the claim, no writes |
+| `incidental_change` | legitimate | Permitted extra calls (the reference's reads, repeated) before the claim |
+
+Identity arguments are recognized by name (`*_id`, `email`, `recipient`,
+`iban`, URL path ids and so on). Retargets draw from values that other tasks in
+the suite use for the same argument, skipping any value the task's own
+reference uses. `wrong_value` and `wrong_record` cells also record whether the
+original value appears as a whole word in the task instruction, so a reader can
+set aside edits to fields the task never specified.
+
+`python -m openadapt_flow.verifier_audit families` prints the set as JSON
+(`openadapt.verifier-audit.families/v1`). Add your own families with
+`--families-module your.module`; the module exports a `FAMILIES` tuple of
+`FaultFamily`.
+
+## Writing an adapter
+
+A target implements `AuditTarget`:
+
+```python
+class MyEnv:
+    name = "my-env"
+    post_claim_steps = True  # False if the episode ends at the final answer
+
+    def tasks(self) -> list[str]: ...
+    def reference(self, task) -> list[Call]: ...  # each Call has kind READ, WRITE, CLAIM or AUTH
+    def run(self, task, calls, *, score: bool) -> Episode: ...
+    def instruction(self, task) -> str: ...  # optional
+```
+
+`run` must reset, replay `calls` in order, record which calls the environment
+refused, read the end state (and, if it can, the state at the claim step), and
+call the scorer only when `score` is true. `targets/toy.py` is the smallest
+example. `targets/agentdojo.py` runs in process. `targets/appworld.py` drives a
+worker in AppWorld's own interpreter, because AppWorld pins pydantic 1.
+
+## Running
+
+```bash
+python -m openadapt_flow.verifier_audit run \
+  --target openadapt_flow.verifier_audit.targets.agentdojo:make_target \
+  --target-arg suite=banking --out audit/agentdojo-banking
+```
+
+The output directory holds `cells.jsonl` (an append-only ledger; a rerun
+resumes and does not rescore finished cells, but retries harness and scorer
+errors, including a task excluded only because the reference scorer errored), `ledger.json` (the target, kit version, families and redaction mode;
+a rerun with a different target or mode refuses), `report.json` and
+`REPORT.md`.
+Rates are per family, with Wilson 95% intervals, and are not pooled. The report
+also gives the share of tasks with at least one false accept.
+
+Use `--redact` when a target's task data may not be redistributed in the clear.
+AppWorld's protected data, for example, may be shared only encrypted. Redaction
+replaces call arguments, effect rows, edit descriptions and error text with
+digests and drops scorer details. Task ids and family names stay readable.
+
+## What a result does and does not show
+
+- A false accept shows that the scorer returned PASS on an end state that
+  differs from the reference effect in the named way. The reference is one
+  correct effect, not the only one. Read the counterexample's effect diff
+  before you call it a bug.
+- The audit replays scripted trajectories and makes no model calls. It shows
+  what a policy could get credit for, not how often an agent does it.
+- Benchmarks own their scorers. Report findings to their maintainers before
+  you publish them.

packages/flow/openadapt_flow/verifier_audit/__init__.py

@@ -0,0 +1,57 @@
+"""Verifier audit: does an environment's own scorer accept wrong end states?
+
+A benchmark or RL environment ships a scorer that turns a finished episode
+into a reward. This kit tests that scorer, not an agent. It starts from a
+reference trajectory that the environment itself counts as correct, applies
+mechanical fault families to it (an unauthorized side effect, the wrong
+record, a duplicate write, a partial write, a write after the success claim,
+a success claim with no write), replays each edited trajectory from a fresh
+reset, and asks the environment's own scorer for a verdict.
+
+Ground truth never comes from the scorer. The target adapter reads the end
+state through its own path (a database diff, a model dump, a Table API read)
+and the kit compares that effect with the effect of the unedited reference.
+A fault whose effect matches the reference is degenerate and is excluded. A
+fault whose effect differs and that the scorer still accepts is a false
+accept, reported as a counterexample with the effect diff. A legitimate
+incidental change (an extra read, a repeated lookup) that the scorer rejects
+is a false reject.
+
+The kit is the method from "Finding Blind Spots in AppWorld and WorkArena
+Task Verifiers" (Abrich, 2026), generalized to any environment that can
+reset, replay a trajectory, score, and read back its state. See
+``docs/VERIFIER_AUDIT.md``.
+"""
+
+from __future__ import annotations
+
+from openadapt_flow.verifier_audit.families import (
+    DEFAULT_FAMILIES,
+    FaultFamily,
+    Polarity,
+    families_spec,
+)
+from openadapt_flow.verifier_audit.model import (
+    AuditTarget,
+    Call,
+    CallKind,
+    Episode,
+    Verdict,
+)
+from openadapt_flow.verifier_audit.report import wilson_interval
+from openadapt_flow.verifier_audit.runner import AuditConfig, run_audit
+
+__all__ = [
+    "AuditConfig",
+    "AuditTarget",
+    "Call",
+    "CallKind",
+    "DEFAULT_FAMILIES",
+    "Episode",
+    "FaultFamily",
+    "Polarity",
+    "Verdict",
+    "families_spec",
+    "run_audit",
+    "wilson_interval",
+]

packages/flow/openadapt_flow/verifier_audit/__main__.py

@@ -0,0 +1,88 @@
+"""``python -m openadapt_flow.verifier_audit``: audit an environment's scorer.
+
+    python -m openadapt_flow.verifier_audit families
+    python -m openadapt_flow.verifier_audit run \\
+        --target openadapt_flow.verifier_audit.targets.agentdojo:make_target \\
+        --target-arg suite=banking --out audit-out/agentdojo-banking
+
+``--target`` names a factory ``module:function`` that returns an
+:class:`~openadapt_flow.verifier_audit.model.AuditTarget`; each
+``--target-arg key=value`` is passed to it as a keyword argument.
+``--families-module`` adds a private family set (its ``FAMILIES`` tuple).
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import json
+import sys
+from pathlib import Path
+
+from openadapt_flow.verifier_audit.families import (
+    DEFAULT_FAMILIES,
+    families_spec,
+    load_families,
+)
+from openadapt_flow.verifier_audit.runner import AuditConfig, run_audit
+
+
+def _factory(spec: str):
+    module, _, attr = spec.partition(":")
+    if not attr:
+        raise SystemExit(f"--target must be module:function, got {spec!r}")
+    return getattr(importlib.import_module(module), attr)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="python -m openadapt_flow.verifier_audit")
+    sub = parser.add_subparsers(dest="cmd", required=True)
+    fam = sub.add_parser("families", help="print the family set as JSON")
+    fam.add_argument("--families-module")
+    run = sub.add_parser("run", help="audit a target")
+    run.add_argument("--target", required=True)
+    run.add_argument("--target-arg", action="append", default=[])
+    run.add_argument("--out", type=Path, required=True)
+    run.add_argument("--tasks", help="comma-separated task ids (default: all)")
+    run.add_argument(
+        "--families-module", help="module whose FAMILIES are added to the defaults"
+    )
+    run.add_argument("--only-families", help="comma-separated family ids to run")
+    run.add_argument("--max-candidates", type=int, default=6)
+    run.add_argument(
+        "--single-replay",
+        action="store_true",
+        help="score the census replay instead of replaying the chosen edit again",
+    )
+    run.add_argument(
+        "--redact", action="store_true", help="digest call arguments and effect rows"
+    )
+    args = parser.parse_args(argv)
+
+    families = tuple(DEFAULT_FAMILIES)
+    if args.families_module:
+        families += load_families(args.families_module)
+    if args.cmd == "families":
+        print(json.dumps(families_spec(families), indent=2))
+        return 0
+    if args.only_families:
+        wanted = set(args.only_families.split(","))
+        families = tuple(f for f in families if f.id in wanted)
+    kwargs = dict(kv.split("=", 1) for kv in args.target_arg)
+    target = _factory(args.target)(**kwargs)
+    cfg = AuditConfig(
+        families=families,
+        tasks=args.tasks.split(",") if args.tasks else None,
+        max_candidates=args.max_candidates,
+        single_replay=args.single_replay,
+        redact=args.redact,
+        out_dir=args.out,
+        log=lambda msg: print(msg, file=sys.stderr, flush=True),
+    )
+    report = run_audit(target, cfg)
+    print(json.dumps(report["summary"], indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())