[abrichr]

Closes #1982

The benchmark judge now evaluates task intent, structured actions, screenshots and explicitly sourced observations without agent final answers or per-turn narration. The input boundary covers action scoring, batched and fallback screenshot analysis, rubric rescoring and outcome verification. Measured HTTP status remains transport evidence, and legitimate record text is preserved.

Validation on WSL: 12 offline fake-client tests passed; changed-file and package Ruff checks passed; package formatting passed; mypy passed for 593 source files with the same-checkout types package; release consistency and engine generality passed.

No model calls or benchmark rescoring were performed. Historical protocol and result files retain their bytes. Observation provenance remains the responsibility of the harness adapter; the tests verify that every constructed model message respects that boundary.

[abrichr]

Independent adversarial review: APPROVE

Reviewed head 30147471e8423fa4f87d3a0f45466672824c0129. It is ad3cd1887 plus one change that only touches docs (JUDGE_INPUTS.md).

What I ran

• pytest tests/test_claim_blind_judge.py benchmark/verifier_baselines/test_uv_judge.py (packages/flow, uv run --frozen --extra dev): 12 passed.
• Two mutation checks against the structural test:
  1. Put state_description back into format_action_history.
  2. Append view["predicted_output"] to the task text in run_judge.judge_one.
     Both make test_every_model_stage_is_claim_blind[False|True] fail, so the test guards the real orchestration and not only one helper.
• ruff check benchmark/verifier_baselines tests/test_claim_blind_judge.py: clean.
• ruff format --check on the changed files: see P3 below.
• gh pr checks 1988: ci, packages-gate and run-ci fail only on the draft-PR gate. Real CI has not run yet.

Leak audit

• uv_judge.judge has only one route from the trajectory into a prompt: steps becomes action_history through format_action_history. That function now emits the URL, structured action args and observations with an allowed source.
  • state_description, error and tool_output are dropped.
  • Observations whose source is agent or missing are dropped.
• No $predicted* or $agent_predicted_output placeholders remain in uv_prompts.py. The judge() signature rejects predicted_output.
• Adapters:
  • medplum_views no longer copies turn.text or claim.final_text/note.
  • smoke_uv_practice no longer copies turn.text.
  • effect_e2e_arms drops the scripted narration and the terminate(status=...) self-verdict, and replaces them with a measured http observation.
• Screenshots are app start/end captures, not the agent's chat.
• scoreboard/claims_rejudge.py reads final_text, but it is a deterministic verify-run with no model judge, so it is out of scope.
• Genuine observations are kept. The test checks system-of-record and app content (including the words "claim saved success"), HTTP receipts and actions at every stage after the rubric.
• Production verifier and receipt code are untouched. Every change is under benchmark/verifier_baselines plus one test.

Findings (none blocking)

• P3 packages/flow/benchmark/verifier_baselines/uv_prompts.py:716: the action-only scorer still says "based on the Action History and Predicted Output". This is stale instruction text, not a leak.
• P3 packages/flow/benchmark/verifier_baselines/medplum_views.py:70: turns with only text become action_name: "reply". The judge learns that the agent replied, not what it said. That is acceptable, but consider "none" to match smoke_uv_practice.
• P3 ruff format --check would reformat smoke_uv_practice.py:106 and uv_judge.py:246. CI's format step does not cover benchmark/verifier_baselines, so this is cosmetic.
• Note: many rubric examples still give full credit when the "agent reports this" (for example, an unavailable item). With replies withheld, the judge will treat such tasks as having an evidence gap. JUDGE_INPUTS.md documents this. Expect lower recall on report-style criteria in new runs.