[MaxGhenis]

Follow-up to #235 (merged as 069f6a5). This is internal pilot tooling only: no board condition, published number or release changes.

Review r2 of #235 (Opus 5.5) agreed at b8231bf and listed low-severity follow-ups. This PR closes them.

Wording (v2-pilot only)

• Meals. The meal definitions now say "both breakfast and lunch". policyengine-us 2.41.0's state_universal_free_meals is "free breakfast and lunch to all students", and the copay parameter covers both meals. Without that, a model could count a breakfast-only program. No reference among the 100 households changes. The review checked all seven free references and found none reduced-price.
• Rate schedule. The d1263 line ("compute from the section 1 rate schedules, not the Tax Table") now also rides on federal_refundable_credits, since the refundable CTC depends on tax liability.

Harness

• Search-only turns for Claude. JSON-contract web arms now count turns that only searched toward the cap, and get a cap message once it is reached. Before, the cap bound only tool-contract models (Sol, Argon).
• Unknown calls. An answer that arrives beside an unknown function call now gets that call's result.
• Repairs. A repair waits while a server call is unfinished. Anthropic returns a 400 on any text after an unfinished server call.

Audit

• OpenAI source objects are kept whole. Two sources in the trial came back without a URL.
• reference_engine_entries counts searches and results that name PolicyEngine. These are not blocked or flagged, since a web+PE model may read the tool's docs.
• score prints a caveat when a provider returns no result URLs (Gemini).

Evidence for Claude's server-tool paths (review r2, finding 1)

• Offline test. A new test runs litellm's own Anthropic conversion on a moved deferred result, with thinking blocks and a nested (caller) search as real Opus replies have. It pins the block order.
• Live probes. Two probes used the board's Claude settings (response_format json_object and the board's thinking configuration). Each ran four requests with no errors: a code-execution call deferred beside our tool, its result moved next to its call, then a nested search and two more calculations. Every request and response body is saved in ~/reviews/policybench-tool-pilot-2026-10/web-arms/probes/. A third attempt timed out on its first request (600 s, an infrastructure failure) and is kept as mixed_probe_1_timeout.json.

Tests

• 139 passed at 7210d03, and 141 at 9249fc0.
• Five planted bugs, one at a time, were each caught: no cap text for JSON-contract web arms, an unknown call left unanswered, a repair not deferred, the source object dropped, and the meal wording reverted.

Review r3 (Opus 5.5) AGREED at 7210d03. Commit 9249fc0 then closes three of its low findings:

• Wording hash. Each record and manifest carries prompt_wording_sha256, a hash of every v2-pilot rewrite, and run refuses to resume across a wording change. The meal and refundable-credit lines changed after the v2 trial, so a resume into trial-v2b is now refused, not mixed.
• Repair rounds. A repair deferred behind an unfinished server call uses no repair round.
• Gemini caveat. The "no result URLs" caveat is written into score.json, not only printed.

Two things are left as they are. The web+PE notice still doesn't say that search-only turns share the 8-turn cap, because changing it would change the prompts the trial used. The run-level refusal has no test, because run needs the engine; the helpers behind it are tested. Tests: 141 passed. Review r4 covers this commit.

Deferred (r1 finding 8). There is no cross-check of Gemini's search count: litellm's own count and grounding metadata are empty on Gemini's function-calling turns (live probe, 2026-10-20).

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 21, 2026 12:47pm UTC

[MaxGhenis]

Merging at 9249fc0.

• CI: app, lint and test all pass on this head (run 39602284276).
• Reviews (Opus 5.5, because the Codex lanes were closed):
  • r3 AGREE at 7210d03;
  • r4 AGREE at 9249fc0, covering the delta that fixed r3's findings 1 and 2 and the score.json caveat.
• Where the reviews are: ~/reviews/policybench-tool-pilot-2026-10/web-arms/review/ (review_r3_opus.md, review_r4_opus.md).
• Left as they are (low):
  • The web+PE notice still doesn't mention that search-only turns share the cap, because changing it would change the prompts the trial used.
  • The wording hash covers the rewrites but not the search-rule text.
  • searches.csv keeps only source URLs; the full source object is in the record.

This is internal pilot tooling only. No board condition, published number or release changes.

🤖 Generated with Claude Code