[MaxGhenis]

This is an internal pilot, not a board condition. The board's no_tools condition is unchanged.

What

scripts/tool_pilot_argon.py run --arm {pe,web,web+pe}:

• web gives the provider's own web search in place of calculate_household.
• web+pe gives both.
• pe is unchanged. A test pins its notices to the sha256 the published runs recorded.

Rows are labeled <model>+web and <model>+web+pe-tool.

Why

Max asked what web access would cost, and whether web plus the PolicyEngine tool is cheaper than web alone. This PR adds the arms and was used for a 5-household trial. A 100-household run waits on his call.

How each provider is wired

Checked 2026-10-20 against each provider's docs and with live probes.

OpenAI (GPT-6 Sol)

• Uses the Responses API web_search tool with filters.blocked_domains.
• Requests include=["web_search_call.action.sources"] so the audit sees each search's result URLs.
• Billing counts every web_search_call at $10 per 1,000. That is an upper bound: the pricing page doesn't say whether open_page and find_in_page are billed. search_actions counts searches alone.

Anthropic (Opus 5.5)

• Uses web_search_20260209 (dynamic filtering) with blocked_domains.
• Billing reads usage.server_tool_use.web_search_requests at $10 per 1,000.
• Server calls (srvtoolu_*) get no client result.
• A reply is continued only when a server call has no result in the same response, the paused state Anthropic documents, at most 3 times. A finished search with no answer gets a nudge.
• No text follows an unfinished server call.
• When Claude calls a server tool beside a client tool, the next response carries the result without the call. litellm 1.88.1 then drops it from later requests, and the API refuses the third request with a 400 (reproduced live). The runner moves that result onto the message that made the call. Two live probes then ran four requests each, clean.

Gemini (Argon)

• Uses googleSearch with server-side tool invocations.
• litellm 1.88.1 drops search whenever function tools are present. The runner patches both drops, for that litellm version only.
• Billing is $14 per 1,000 non-empty queries, once per invocation id. The 5,000 free a month are ignored.

What the blocked domains actually cover

The list is policybench.org, github.com and githubusercontent.com:

• OpenAI: filters search results only. In the smoke and the trial, Sol opened two pages on github.com and raw.githubusercontent.com by URL. Both calls completed, and the response doesn't show whether the content was served.
• Anthropic: filters its searches, but Claude never searched, so this is untested live.
• Gemini: blocks nothing. The API refuses excludeDomains and returns no result URLs, so its reading can't be audited.
• *.github.io (PolicyEngine's generated docs) is not blocked.

score writes searches.csv, flags any household whose searches name PolicyBench or reach a blocked domain, and reports scores without those households. For Gemini it marks results as not visible.

Trial (households 000–004, list prices)

Model	No tools	PE tool	Web	Web + PE
GPT-6 Sol	$0.020	$0.041	$0.145 (5.6 searches)	$0.092 (1.6)
Claude Opus 5.5	$0.055	$0.057	$0.091 (0)	$0.100 (0)
Gemini 4 Argon	$0.027	$0.081	$0.523 (34.4)	$0.093 (0)

• Web plus the tool versus web alone: cheaper for Sol (36% less) and far cheaper for Argon, which never searched once it had the tool. No difference for Opus, which never searched at all.
• 100-household projection: about $118 at list for both arms on all three models. Argon's roughly 3,400 searches would fit inside Google's free 5,000 a month.
• Spent: about $7.50 on the v1 trial, smoke and probes.
• Accuracy: five households can't separate the arms.

What the searches show. 44% of Sol's and 56% of Argon's web-only queries name PolicyEngine. The prompts name PolicyEngine in four output definitions (d1705). In 2 of the 5 households, Argon searched for PolicyBench itself, for example "Inspect household #000" "policybench". That text is on policybench.org and not in the prompt.

Full record: ~/reviews/policybench-tool-pilot-2026-10/web-arms/README.md.

v2-pilot prompts and the no-answer-key rule (Max, 2026-10-21)

--prompt v2-pilot. There are no model-ready v2 prompts yet: #173 is an unactivated fact slice with no output request, and it names PolicyEngine throughout. So this option keeps the board's prompts and rewrites only what Max has ruled on:

• d1705. The four PolicyEngine-naming definitions (Medicaid, CHIP, free and reduced-price school meals) are rewritten neutrally. Each states the convention the reference follows where the law alone wouldn't settle it, read from policyengine-us 2.41.0:
  • everyone aged 5 to 17 counts as a K-12 student, and there are no meals without one;
  • free meals include universal-free-meals states and state-paid copays;
  • someone eligible for Medicaid is not CHIP-eligible;
  • the Medicare Savings Programs are not Medicaid.
• d1263. Federal income tax is computed from the section 1 rate schedules, not the Tax Table.

The rewrite also covers the answer and repair tools. A guard refuses any board text the model sees that still names PolicyEngine.

--forbid-benchmark-search. Tells a web arm not to search for PolicyBench or its answers.

Tests.

• All 100 households are rendered for each contract, and only those definition lines change.
• The meal wording is checked against every reference.
• Three more planted bugs are caught.

Trial on v2-pilot with the rule. Households 000–004, commit b8231bf, $4.86, cost per household at list (v1 → v2-pilot):

Model	Web	Web + PE	PE tool
GPT-6 Sol	$0.145 → $0.146	$0.092 → $0.140	$0.041 → $0.044
Claude Opus 5.5	$0.091 → $0.094	$0.100 → $0.135	$0.057 → $0.088
Gemini 4 Argon	$0.523 → $0.148	$0.093 → $0.107	$0.081 → $0.071

• No query named PolicyBench, against 6 by Argon on v1, and no household was flagged.
• No web-only query named PolicyEngine, against 44% for Sol and 56% for Argon on v1.
• Argon's web-only arm searched 6.4 times a household instead of 34.4.
• Web plus the tool is no longer cheaper than web alone for Sol. Without the PolicyEngine lead, Sol looks up the law instead.
• Misses. The remaining misses trace to the known "employer sponsored insurance premiums" label ambiguity, plus one $110 arithmetic error.
• 100-household projection: about $94 for the six web arms.

Earlier wording, found in the first v2 run: "in any eligibility group" led Sol to count QI as Medicaid. It is fixed.

Invariants (property tests)

• A household's cost_usd equals token_cost_usd plus searches × list price / 1,000, and the spend counter matches. This holds for every provider and both web arms, including households with repair turns and a truncated-and-retried turn.
• Gemini bills the non-empty queries of each invocation id's first appearance only.
• Out of scope but fixed here: paired_bootstrap's better/worse/same now come from one difference, so every household lands in exactly one. Hypothesis found a=0.125, b=a+1e-12 landing in none. The case is pinned as an example, and the published counts are unchanged.

Tests and review

• 133 passed locally; the 98 existing tests are unchanged.
• I planted fifteen bugs, one at a time, and the tests caught all fifteen.
• Review r1 (Opus 5.5) requested changes. Eight of its ten findings are resolved and one is partly resolved: search-only turns now force the answer for tool-contract models only. The Gemini search-count cross-check is deferred, because litellm's own count is empty on Gemini's function-calling turns.
• Review r2 (Opus 5.5, because the Codex lanes were closed until 18:00Z) AGREES at b8231bf: ~/reviews/policybench-tool-pilot-2026-10/web-arms/review/review_r2_opus.md. Its low-severity findings go to a follow-up PR.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 21, 2026 11:57am UTC

[MaxGhenis]

Merging at b8231bf.

• CI: app, lint and test all pass on this head (run 39595612642).
• Review r1 (Opus 5.5): REQUEST_CHANGES. Findings fixed in 7229a9d and 440de6f.
• Review r2 (Opus 5.5, because the Codex lanes were closed): AGREE at this head. Both reviews are in ~/reviews/policybench-tool-pilot-2026-10/web-arms/review/.
• Follow-up PR: r2's low-severity findings. These are evidence for Claude's server-tool paths, the meal wording naming breakfast and lunch, the rate-schedule line on refundable credits, search-only turns for JSON-contract models, and fuller audit logging.

This is internal pilot tooling only. No board condition, published number or release changes.

🤖 Generated with Claude Code