Skip to content

Take the metrics an accept-when line reads on every case; say what best-on-the-evals and cost mean - #43

Merged
jithinAB merged 3 commits into
masterfrom
evals/selection-semantics
Nov 17, 2026
Merged

jithinAB merged 3 commits into
masterfrom
evals/selection-semantics

Conversation

@jithinAB

Copy link
Copy Markdown
Contributor

A metric a model is held to is taken on every case, and prefer: says what "best" means

  • The metrics accept-when: reads are taken on every case. evals.check returns at a case's first failed rule, so refund-tone.score >= 0.8 was held to the mean over only the cases a model got past their first failure. That is an easier subset, and it favours the model that failed more. check takes a new taken_on_every_case, and resolve.evaluate passes it from the agent's lines (selection.metrics_read). Each of those metrics is taken on a failed case too; the case still fails, for the reason it failed before. <metric>.score, .pass-rate and .cases therefore cover the whole suite. Any other metric still stops at the first failure, so a suite nobody holds a model to costs no more grading. A failed outcome now keeps the scores it took (CaseOutcome.scores), as a passed one does.
  • best-on-the-evals is defined as the largest share of the suite's cases passed, with the cheaper model winning a tie. This is what _cheapest_passing already does. The help said "the highest score", which could be read as a metric mean.
  • prefer: chooses only when model: is left out, as resolve already behaves. The help now says so.
  • cost is the agent's own spend per case, not what grading cost. The help now says so. A runtime that meters the judge (Bud Meta Harness does) reports that spend apart from cost.

Proof

  • test_a_metric_an_accept_line_reads_is_taken_on_every_case: two cases, one of which fails expect: first. Without taken_on_every_case, brevity.score is 1.0 over one case and brevity.score >= 0.8 holds. With it, the score covers two cases and the line does not hold. Pass and fail and the failure's reason are unchanged.
  • Focused: test_an_accept_line_means_one_thing.py, test_metrics_an_expert_brings.py, test_a_host_supplies_what_grades_a_case.py and test_judged_rules.py: 101 passed, 2 skipped (opt-in live judge). The full adapter suite is running and the result will be posted here.

Purpose checklist (BMH PACT v97 §0.2)

  1. P1 Declared: driven by the agent's own accept-when: lines.
  2. P2 No hidden divergence: Bud Meta Harness takes the same metrics on every case (its grader mirrors check) and reports the difference from the pinned reference until a re-pin includes this change.
  3. P3 Honest gaps: no new fields.
  4. P4 Host choices recorded: not applicable.
  5. P5 Model-visible ⟺ logged: not applicable.
  6. P6 Cross-runtime evidence: a Python test with the figures both runtimes compute (measured, held_after).
  7. P7 Versioned and checked: not applicable.
  8. P8 Reusable: not applicable.
  9. P9 Platform rules: not applicable (upstream).
  10. P10 Justified: portability and accuracy. The same suite yields the same mean in every runtime, and that mean no longer favours the model that failed more.
  11. P11 Lane hygiene: one change, on master.

🤖 Generated with Claude Code

…st-on-the-evals and cost mean

evals.check returned at a case's first failed rule, so a metric an accept-when
line holds a model to was averaged over the easier cases a model got past,
favouring the model that failed more. check now takes taken_on_every_case
(resolve.evaluate passes selection.metrics_read of the agent's lines) and takes
those metrics on a failed case too; the case still fails for the same reason.
The schema help says best-on-the-evals is the largest share of cases passed
(cheaper wins a tie), that prefer chooses only when model: is left out, and
that cost is the agent's own spend, not grading.
…antics

# Conflicts:
#	CHANGELOG.md
#	adapters/python/src/pact_adapters/selection.py
@jithinAB
jithinAB marked this pull request as ready for review November 17, 2026 03:42
@jithinAB
jithinAB merged commit a52dcdd into master Nov 17, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant