Repository navigation
A measured figure says how it was measured, and every runtime holds a line to its interval - #41
Merged
Merged
Conversation
…ement-fields # Conflicts: # CHANGELOG.md
On a benchmark figure with provenance.interval, a >/>= line holds only when value and interval.low hold, and a </<= line only when value and interval.high hold; = decides on value. needs.scores holds a benchmark to the same bar. The checker refuses a measurement the resolver could not read (interval out of order or off the benchmark's scale, a level that is not a share, more cases than the benchmark has, a measurement on a figure nothing samples) and warns on a subset figure with no interval; the resolver drops such a figure. spec/accept-when.yaml gains figures: and not-measurements:, read by both ports.
…scripts/sync-counts.sh)
…ement-fields # Conflicts: # CHANGELOG.md # README.md # docs/90-REVIEW.md # site-docs/index.md # site-docs/status/verified.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A measured figure says how it was measured, and every runtime holds a line to its interval
A benchmark figure can now record
provenance.sample-size:,population:andinterval: {low, high, level}. On a figure with aninterval:, a>/>=line holds only whenvalue:andinterval.lowboth hold, and a</<=line only whenvalue:andinterval.highboth hold.=decides onvalue:alone. A figure without an interval decides onvalue:as before.Why. Bud Meta Harness measures MMLU on a few hundred questions at onboarding and records the interval. It refuses a row whose interval reaches past the bar; the reference read only
value:and accepted the same row, so one tree and one catalogue bound different models in two runtimes (BMH reports this today asdiverges-from-the-reference). The rule is normative ("every runtime holds"). It is scoped to benchmark figures and applies to both ways of writing a bar (accept-when:andneeds.scores:).Changes
spec/schema.yaml:provenance.sample-sizeandpopulation(integers, at least 1),provenance.interval(groupfigure-interval:low,high,level). The help states the rule and says "on a benchmark figure".selection.py:Measurement,measurement_of(provenance, scale),reaches(op, have, measurement, bar)andwithin_noise(...).held_before's benchmark branch refuses a line the point clears and the interval does not, using BMH's wording: "has mmlu 78, on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1, which reaches past the bar, sommlu.score > 75holds only within noise".resolve.py:ModelEntry.measurements(keyed likescores, carried to aliases through_canonical)._benchmarksdrops a figure whose measurement is malformed, never reading it as its point figure.satisfiesholdsneeds.scores:to the same bar throughreaches, which fixes the disagreementloader/a-benchmark-held-twiceexists to prevent.crates/pact-loader/src/accept.rsrecorded()(workspace layer, where the measurement is written) adds these errors:loader/an-interval-out-of-order,loader/a-confidence-level-that-is-not-a-share,loader/more-cases-than-the-benchmark-has,loader/an-interval-off-the-benchmark-scaleandloader/a-measurement-on-a-figure-nothing-samples(reasoning, context window, speed). It adds one warning,loader/a-subset-figure-with-no-interval. All are in the module rule table.spec/accept-when.yaml:figures:(decisions, including a<=row read onhigh, an=row read on the point, a subset row with no interval, twoneeds.scoresrows and a fraction-scale row) andnot-measurements:(refused rows, each with itsrule:). Both ports read both sections: Rust inevery_measurement_in_the_shared_table_is_checked_as_it_says, Python intest_an_accept_line_means_one_thing.py.test_every_field_has_a_reader.py: the six DELEGATED lines are gone, because the resolver and the checker now read every field.docs/97-PRIMITIVES.md: Store + Check for each field (Store forlevel).site-docs: 70 kinds, plus thefigure-intervalrow.CHANGELOG.md.tests/golden/schema-fields.txtre-sortsaction.moves-hardware. This is harmless:field_inventoryis ordered, and master had a line inserted by hand out of order.Must not break (checked)
spec/accept-when.yamlrow and every catalogue row without an interval decides on its point as before. The shippedmodels/catalog.yamlhas nointerval/sample-sizekey.held_after(suite pass rate, metric score,cost) andbenchmarks.recordoutput are unchanged.Proof
<=row, the fraction row, theneeds.scoresrow, everynot-measurementsrow and both authored-path tests. All pass with it.models/catalog.yamlrow withprovenance.intervaland an agent withaccept-when: [mmlu.score > 75](and, separately,needs.scores: {mmlu: "> 75"}).pact checkpasses,pact showloads it, andscoring._bindrefuses the row with the noise sentence.scripts/test-all.sh: on 03971be: Rust tests, clippy (debug and release), every shipped tree under--deny-warnings, WebAssembly, TypeScript and the adapters: 3577 passed, 30 skipped, and 2 failed. The 2 failures were the documents' counts (the site still quoted 69 kinds in one sentence, and the README test count). The next commit fixes both withscripts/sync-counts.sh, andtest_the_headline_test_count_is_the_count.pyandtest_the_documentation_site_tells_the_truth.pythen pass (17 passed, 3 skipped).scripts/sync-counts.shrewrote the counts (1722 Rust + 3609 adapter), andeverythingpassed on a40ec9e.Purpose checklist (BMH PACT v97 §0.2)
x-budmh-measurement.diverges-from-the-referenceinterval line, which BMH deletes after the re-pin.figures:,not-measurements:) read by Rust and Python. BMH's TypeScript port reads the same file.pact checkrefuses a malformed measurement where it is written.🤖 Generated with Claude Code