Repository navigation
Ready the tool pilot's v2-pilot run: prompt fixes, a no-tools arm, web timeout - #239
Merged
Merged
Conversation
…b timeout Max (2026-10-21, d1800): run all three models in all four arms on 100 households, on the v2-pilot wording. v2-pilot wording, from the pilot's misses, each checked against policyengine-us 2.41.0 or the reference rule: - Law date: "Apply the law and program parameters published before July 3, 2026; for SNAP, use the fiscal year 2026 values for all of 2026". That is the reference rule; the web arms had applied SNAP's FY2027 values. - "employer sponsored insurance premiums" now reads "employer-paid health insurance premiums (paid by the employer, not by the household)", as the engine documents the variable. Web arms had deducted them. - "spm unit energy subsidy" now reads "home energy assistance received". - The state income tax definition names Idaho's $10 permanent building fund tax, the one extra in the engine's state tax list. Run options: - --arm none sends the board's own no-tools request: the forced answer function, or Opus's JSON reply with no tools key. A differential test checks it against eval_no_tools' builders for all three contracts. - Web arms use a 900 s request timeout. Argon's web-only arm never finished scenario_005 within the board's 300 s. - Arms without the calculator start no engine workers. Tests: 144 passed. Five planted bugs were each caught. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- Law date: the sentence now gives the reference's fallback. Where no 2026
amount was published before July 3, 2026, the latest earlier one applies
(California, Idaho, Maryland, Michigan and Minnesota amounts, and the IRS
2025 sales tax tables, per each fix module's rule). It also states SNAP's
hold precisely: FY2026 allotments, deductions, utility allowances and asset
limits through December, with income limits on the 2026 poverty guidelines
from October ("Not held: the poverty guideline").
- Web arms don't retry a timed-out request. Three 900 s attempts would hold
a worker for 45 minutes; the household ends as provider_error, which a
resume reruns.
- The no-tools arm marks no prompt-cache breakpoint, as the board's request
marks none, and records its first turn as forced. The differential test now
compares the whole first request with the board's, key for key, across
all 100 households and three contracts.
- The energy relabel's comment now states the engine's mechanism: it deems
the utility allowance from the state flag and never reads the amount.
- The manifest records the timeout actually sent. Error records carry the
arm and prompt keys that resume and score check.
Tests: 147 passed. Three more planted bugs were each caught.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
MaxGhenis
marked this pull request as ready for review
October 22, 2026 11:20
Contributor
Author
|
Merging at 274527b.
This is internal pilot tooling only. No board condition, published number or release changes. 🤖 Generated with Claude Code |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This readies the 100-household run (Max, 2026-10-21, d1800): GPT-6 Sol, Claude Opus 5.5 and Gemini 4 Argon, each in four arms (no tools, PE tool, web, web+PE), on the v2-pilot wording, with the web arms told not to search for PolicyBench. It is internal pilot tooling only: no board condition, published number or release changes.
v2-pilot wording fixes
Each fix comes from the pilot's misses and is checked against policyengine-us 2.41.0 or the reference rule.
The law date. The prompt now says: "Apply the law and program parameters published before July 3, 2026. Where a 2026 amount had not been published by then, use the latest amount published before that date (for example, a state's 2025 inflation-adjusted amounts, or the IRS's 2025 optional sales tax tables). For SNAP, keep the fiscal year 2026 allotments, deductions, utility allowances and asset limits through December 2026; the income limits use the 2026 HHS poverty guidelines from October 2026."
Relabeled facts.
Idaho. The state income tax definition now names Idaho's $10 permanent building fund tax, the one non-income-tax entry in the engine's state tax list. All three Idaho households owe it.
Harness
The no-tools arm.
--arm nonesends the board's own first request: the forced answer function, or Opus's JSON reply with no tools and no cache mark. A differential test checks the whole request, key for key, againsteval_no_tools' builders for all 100 households and three contracts. Repairs follow the pilot's loop.A live check on the v1 wording reproduced the frozen board rows on 10 households:
Web arms.
Engine workers. Arms without the calculator start none.
Review and tests
🤖 Generated with Claude Code