Skip to content

A measured figure says how it was measured, and every runtime holds a line to its interval - #41

Merged
jithinAB merged 6 commits into
masterfrom
evals/figure-measurement-fields
Nov 16, 2026
Merged

jithinAB merged 6 commits into
masterfrom
evals/figure-measurement-fields

Conversation

@jithinAB

@jithinAB jithinAB commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

A measured figure says how it was measured, and every runtime holds a line to its interval

A benchmark figure can now record provenance.sample-size:, population: and interval: {low, high, level}. On a figure with an interval:, a >/>= line holds only when value: and interval.low both hold, and a </<= line only when value: and interval.high both hold. = decides on value: alone. A figure without an interval decides on value: as before.

Why. Bud Meta Harness measures MMLU on a few hundred questions at onboarding and records the interval. It refuses a row whose interval reaches past the bar; the reference read only value: and accepted the same row, so one tree and one catalogue bound different models in two runtimes (BMH reports this today as diverges-from-the-reference). The rule is normative ("every runtime holds"). It is scoped to benchmark figures and applies to both ways of writing a bar (accept-when: and needs.scores:).

Changes

  • spec/schema.yaml: provenance.sample-size and population (integers, at least 1), provenance.interval (group figure-interval: low, high, level). The help states the rule and says "on a benchmark figure".
  • selection.py: Measurement, measurement_of(provenance, scale), reaches(op, have, measurement, bar) and within_noise(...). held_before's benchmark branch refuses a line the point clears and the interval does not, using BMH's wording: "has mmlu 78, on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1, which reaches past the bar, so mmlu.score > 75 holds only within noise".
  • resolve.py: ModelEntry.measurements (keyed like scores, carried to aliases through _canonical). _benchmarks drops a figure whose measurement is malformed, never reading it as its point figure. satisfies holds needs.scores: to the same bar through reaches, which fixes the disagreement loader/a-benchmark-held-twice exists to prevent.
  • crates/pact-loader/src/accept.rs recorded() (workspace layer, where the measurement is written) adds these errors: loader/an-interval-out-of-order, loader/a-confidence-level-that-is-not-a-share, loader/more-cases-than-the-benchmark-has, loader/an-interval-off-the-benchmark-scale and loader/a-measurement-on-a-figure-nothing-samples (reasoning, context window, speed). It adds one warning, loader/a-subset-figure-with-no-interval. All are in the module rule table.
  • spec/accept-when.yaml: figures: (decisions, including a <= row read on high, an = row read on the point, a subset row with no interval, two needs.scores rows and a fraction-scale row) and not-measurements: (refused rows, each with its rule:). Both ports read both sections: Rust in every_measurement_in_the_shared_table_is_checked_as_it_says, Python in test_an_accept_line_means_one_thing.py.
  • test_every_field_has_a_reader.py: the six DELEGATED lines are gone, because the resolver and the checker now read every field. docs/97-PRIMITIVES.md: Store + Check for each field (Store for level). site-docs: 70 kinds, plus the figure-interval row. CHANGELOG.md.
  • The golden tests/golden/schema-fields.txt re-sorts action.moves-hardware. This is harmless: field_inventory is ordered, and master had a line inserted by hand out of order.

Must not break (checked)

  • Every existing spec/accept-when.yaml row and every catalogue row without an interval decides on its point as before. The shipped models/catalog.yaml has no interval/sample-size key.
  • held_after (suite pass rate, metric score, cost) and benchmarks.record output are unchanged.

Proof

  • Fails before, passes after: with the resolver change removed, 13 of the new Python tests fail: the noise rows, the <= row, the fraction row, the needs.scores row, every not-measurements row and both authored-path tests. All pass with it.
  • Authored path: a workspace models/catalog.yaml row with provenance.interval and an agent with accept-when: [mmlu.score > 75] (and, separately, needs.scores: {mmlu: "> 75"}). pact check passes, pact show loads it, and scoring._bind refuses the row with the noise sentence.
  • scripts/test-all.sh: on 03971be: Rust tests, clippy (debug and release), every shipped tree under --deny-warnings, WebAssembly, TypeScript and the adapters: 3577 passed, 30 skipped, and 2 failed. The 2 failures were the documents' counts (the site still quoted 69 kinds in one sentence, and the README test count). The next commit fixes both with scripts/sync-counts.sh, and test_the_headline_test_count_is_the_count.py and test_the_documentation_site_tells_the_truth.py then pass (17 passed, 3 skipped).
  • Merged master (A runtime may serve a workspace without its broken agents (--leave-out-broken-agents) #44) on 2026-11-17: conflicts only in the counts and CHANGELOG; scripts/sync-counts.sh rewrote the counts (1722 Rust + 3609 adapter), and everything passed on a40ec9e.

Purpose checklist (BMH PACT v97 §0.2)

  1. P1 Declared: the measurement is a PACT field on the figure. Nothing is BMH-only; it replaces BMH's x-budmh-measurement.
  2. P2 No hidden divergence: this removes BMH's diverges-from-the-reference interval line, which BMH deletes after the re-pin.
  3. P3 Honest gaps: each new field has a reader (the resolver) and a checker rule. DELEGATED is back to 24.
  4. P4 Host choices recorded: no host choice is added. A refused row is named with its reason.
  5. P5 Model-visible ⟺ logged: no model-visible input changes.
  6. P6 Cross-runtime evidence: shared vectors (figures:, not-measurements:) read by Rust and Python. BMH's TypeScript port reads the same file.
  7. P7 Versioned and checked: pact check refuses a malformed measurement where it is written.
  8. P8 Reusable: catalogue rows hold in any workspace, bundle or host.
  9. P9 Platform rules: no change to platform rules (upstream spec).
  10. P10 Justified: portability, because one tree and one catalogue now bind one model in every runtime. There is no trade-off.
  11. P11 Lane hygiene: one change, on master.

🤖 Generated with Claude Code

Jithin VG and others added 6 commits November 16, 2026 14:16
On a benchmark figure with provenance.interval, a >/>= line holds only when
value and interval.low hold, and a </<= line only when value and interval.high
hold; = decides on value. needs.scores holds a benchmark to the same bar. The
checker refuses a measurement the resolver could not read (interval out of
order or off the benchmark's scale, a level that is not a share, more cases
than the benchmark has, a measurement on a figure nothing samples) and warns
on a subset figure with no interval; the resolver drops such a figure.
spec/accept-when.yaml gains figures: and not-measurements:, read by both
ports.
…ement-fields

# Conflicts:
#	CHANGELOG.md
#	README.md
#	docs/90-REVIEW.md
#	site-docs/index.md
#	site-docs/status/verified.md
@jithinAB
jithinAB marked this pull request as ready for review November 16, 2026 23:11
@jithinAB
jithinAB merged commit e932ad6 into master Nov 16, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant