Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,43 @@
What changed in PACT, newest first. Each entry says what an author can now write, or what now
holds that did not; the commits are listed under it for whoever wants the detail.

## 2026-11 — a measured figure says how it was measured, and a line is held to its interval (16 November)

A figure's `provenance:` said where a number came from but not how much of the benchmark it rests on
or how sure it is. A runtime that measures figures itself (Bud Meta Harness measures MMLU on a few
hundred questions at onboarding) kept that under its own `x-` key and held a line to the interval,
while the reference held it to the point — so one tree and one catalogue could bind different models
in two runtimes.

### What an author can now write

- **`provenance.sample-size:` and `provenance.population:`** — on a benchmark figure, how many of the
benchmark's cases it was measured on, out of how many it has. Fewer than the population makes it a
subset figure, and a refusal that names the figure says so.
- **`provenance.interval: {low, high, level}`** — on a benchmark figure, the confidence interval
around `value:`, on the benchmark's own scale, with the level as a share (`0.95`).

### What holds now

- **Every runtime holds a line to the interval's far bound.** On a benchmark figure with an
`interval:`, a `>`/`>=` line holds only when `value:` and `interval.low` both hold, and a `<`/`<=`
line only when `value:` and `interval.high` both hold; `=` is decided on `value:`. `mmlu: 78` with a
95% interval of 71.6 to 83.1 does not clear `mmlu.score > 75`: *"has mmlu 78, on a subset of 200 of
14042 cases, 95% interval 71.6 to 83.1, which reaches past the bar, so `mmlu.score > 75` holds only
within noise"*. A figure without an interval decides on `value:` as before.
- **`needs.scores:` holds a benchmark to the same bar.** It compared the point alone, so one benchmark
under `scores:` and under `accept-when:` could decide differently.
- **The checker refuses a measurement that cannot be one**, where it is written:
`loader/an-interval-out-of-order`, `loader/a-confidence-level-that-is-not-a-share`,
`loader/more-cases-than-the-benchmark-has`, `loader/an-interval-off-the-benchmark-scale`, and
`loader/a-measurement-on-a-figure-nothing-samples` (a reasoning rung, a context window or a speed),
and warns `loader/a-subset-figure-with-no-interval`. The reference resolver drops such a figure, so
a line on it says the figure was never recorded rather than reading it as its point.
- `spec/accept-when.yaml` gains `figures:` (decisions) and `not-measurements:` (refusals), read by the
Rust checker and the Python resolver alike.
- The golden `tests/golden/schema-fields.txt` re-sorts `action.moves-hardware`: `field_inventory` is
ordered, and master had a hand-inserted line out of order.

## 2026-11 — a runtime may serve a workspace without its broken agents (16 November)

An error in any file of a workspace withdrew all of it: `pact discover` printed `skipping <root>: N
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1001,11 +1001,11 @@ framework is an extra.
|---|---|
| **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete |
| **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5312 tests (1721 Rust + 3591 adapter), clippy clean, TypeScript type-checked** |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5331 tests (1722 Rust + 3609 adapter), clippy clean, TypeScript type-checked** |
| **Adapters** | **All 7 named targets**, proven against one shared conformance suite |

```bash
./scripts/test-all.sh # 5312 tests, Rust + 7 adapters, fully offline
./scripts/test-all.sh # 5331 tests, Rust + 7 adapters, fully offline
```

**What is still owed** is not hidden — it lives in
Expand Down
52 changes: 42 additions & 10 deletions adapters/python/src/pact_adapters/resolve.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
from dataclasses import dataclass, field, replace
from functools import lru_cache
from pathlib import Path
from typing import Any, Callable, Mapping
from typing import Any, Callable, Mapping, TypeVar

import yaml

Expand Down Expand Up @@ -160,6 +160,11 @@ class ModelEntry:
output_cost: "float | None" = None
capabilities: frozenset[str] = frozenset()
scores: dict[str, float] = None # type: ignore[assignment]
#: How each benchmark figure in `scores` was measured, keyed like `scores`;
#: a figure published without `sample-size:`, `population:` or `interval:`
#: has no entry. `accept-when:` and `needs.scores:` hold a line to the
#: interval's far bound as well as the figure (`selection.reaches`).
measurements: "Mapping[str, _selection.Measurement]" = field(default_factory=dict)
#: How much this model can hold, in tokens. `None` means the catalogue says
#: `unknown` — not "small", and not "ask again later". It is here rather than
#: on a second record beside it because a window is a fact about a model, and
Expand Down Expand Up @@ -297,6 +302,11 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]:
op, value = wanted
if not _HOLDS[op](have, value):
return False, f"{metric} {have} is not {op} {value:g}"
# The bar `accept-when:` holds a benchmark to, so the same figure
# under either spelling decides the same way.
taken = self.measurements.get(metric)
if not _selection.reaches(op, have, taken, value):
return False, _selection.within_noise(metric, have, taken, f"scores.{metric}: {threshold}")
wanted = _tokens(needs.get("context-at-least"))
if wanted is not None and self.context_window is not None:
if self.context_window < wanted:
Expand Down Expand Up @@ -724,13 +734,15 @@ def _parse(text: str, shown: str) -> Catalogue:
window, trouble = _window(shown, lines, name, caps.get("context-window"))
if trouble is not None:
problems.append(trouble)
scores, measurements = _benchmarks(row.get("benchmarks"), names)
entry = ModelEntry(
name=name,
tier=str(row.get("tier") or "mid"),
cost=_cost(row.get("cost")),
output_cost=_cost(row.get("cost"), "output-per-mtok"),
capabilities=_capabilities(caps),
scores=_canonical(_benchmarks(row.get("benchmarks")), names),
scores=_canonical(scores, names),
measurements=_canonical(measurements, names),
context_window=window,
aliases=frozenset(
str(a) for a in (row.get("also-known-as") or []) if isinstance(a, str)
Expand Down Expand Up @@ -970,8 +982,10 @@ def _threshold(written: Any) -> "tuple[str, float] | None":
return None


def _benchmarks(written: Any) -> dict[str, float]:
"""Published benchmark figures for one row, by metric name.
def _benchmarks(
written: Any, names: "Mapping[str, tuple[str, str]]",
) -> "tuple[dict[str, float], dict[str, _selection.Measurement]]":
"""Published benchmark figures for one row, by metric name, and how each was measured.

This came back `{}` unconditionally — the argument was not even read — so
`satisfies()` answered *"no published {metric} score"* for every candidate on
Expand All @@ -989,18 +1003,36 @@ def _benchmarks(written: Any) -> dict[str, float]:
A figure written `unknown`, or one that is not a number, is DROPPED rather
than kept as zero. Zero would satisfy no threshold and read as a measurement;
absent is the truth, and `satisfies()` says *"no published X score"* about it.

A figure whose `provenance:` records a measurement that cannot be one
(`selection.measurement_of`: an interval out of order or off the benchmark's
scale named in `names`, a level that is not a share, more cases than the
benchmark has) is dropped the same way, never read as its point figure.
"""
if not isinstance(written, dict):
return {}
return {}, {}
out: dict[str, float] = {}
measured: dict[str, _selection.Measurement] = {}
for metric, figure in written.items():
provenance = None
if isinstance(figure, dict):
figure = figure.get("value")
figure, provenance = figure.get("value"), figure.get("provenance")
try:
out[str(metric)] = float(figure) # type: ignore[arg-type]
value = float(figure) # type: ignore[arg-type]
except (TypeError, ValueError):
continue # `unknown`, or prose — see the docstring
return out
known = names.get(str(metric).strip().lower())
try:
taken = _selection.measurement_of(provenance, known[1] if known else None)
except _selection.Unreadable:
continue # a measurement that cannot be one — see the docstring
out[str(metric)] = value
if taken is not None:
measured[str(metric)] = taken
return out, measured


_Figure = TypeVar("_Figure")


def _milliseconds(written: Any) -> "float | None":
Expand All @@ -1016,8 +1048,8 @@ def _milliseconds(written: Any) -> "float | None":
return None if found is None else found * 1000.0


def _canonical(scores: dict[str, float], names: dict[str, tuple[str, str]]) -> dict[str, float]:
"""A row's scores under the names it wrote AND each benchmark's own key."""
def _canonical(scores: "dict[str, _Figure]", names: "Mapping[str, tuple[str, str]]") -> "dict[str, _Figure]":
"""A row's scores, or their measurements, under the names it wrote AND each benchmark's own key."""
out = dict(scores)
for written, figure in scores.items():
key = names.get(written.strip().lower())
Expand Down
136 changes: 136 additions & 0 deletions adapters/python/src/pact_adapters/selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,18 @@
accepted by `mmlu.score > 50`, an unpriced one by `input-price < 3 USD`, one
whose cost was never metered by `cost < 0.05 USD`. Each says why, so the
author can record the figure (`benchmarks.record`) rather than guess.

## A measured figure

A benchmark figure may say how it was measured: `provenance.sample-size:` and
`population:` (how many of the benchmark's cases it rests on, out of how many),
and `provenance.interval:` (`low`, `high`, `level`). A line with `>` or `>=`
holds only when both `value:` and `interval.low` hold; one with `<` or `<=`
only when both `value:` and `interval.high` hold; `=` is decided on `value:`
alone. So a score of 78 with a 95% interval of 71.6 to 83.1 does not clear
`mmlu.score > 75`: the line holds only within noise, and the reason says so.
Every runtime holds the line this way, so one tree and one catalogue bind one
model wherever they are read.
"""

from __future__ import annotations
Expand All @@ -45,13 +57,17 @@
"AFTER",
"BEFORE",
"Line",
"Measurement",
"Unreadable",
"benchmarks_of",
"held_after",
"held_before",
"measured",
"measurement_of",
"parse",
"parse_all",
"reaches",
"within_noise",
]

#: The figures every model has, and the unit each is written in.
Expand Down Expand Up @@ -100,6 +116,123 @@ def holds(self, have: float) -> bool:
return _HOLDS["=" if self.op == "==" else self.op](have, self.value)


#: The highest a figure can be on each benchmark scale; the lowest is zero.
_SCALE_TOP = {"percent": 100.0, "fraction": 1.0}


@dataclass(frozen=True)
class Measurement:
"""How a recorded benchmark figure was measured (`provenance.sample-size`,
`population` and `interval`). A bound is on the figure's own scale."""

sample_size: "int | None" = None
population: "int | None" = None
low: "float | None" = None
high: "float | None" = None
#: The confidence level, as a share such as `0.95`.
level: "float | None" = None

@property
def subset(self) -> bool:
"""Taken on fewer cases than the benchmark has."""
return (
self.sample_size is not None
and self.population is not None
and self.sample_size < self.population
)

def text(self) -> str:
"""`on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1`, or `""`.

The wording Bud Meta Harness prints for the same figure, so a refusal
reads the same in both runtimes.
"""
parts = []
if self.sample_size is not None:
of = "" if self.population is None else f" of {self.population}"
parts.append(f"{'on a subset of' if self.subset else 'on'} {self.sample_size}{of} cases")
if self.low is not None and self.high is not None and self.level is not None:
parts.append(f"{self.level * 100:g}% interval {self.low:g} to {self.high:g}")
return ", ".join(parts)


def _count(written: Any, field: str) -> "int | None":
if written is None:
return None
if isinstance(written, bool) or not isinstance(written, int) or written < 1:
raise Unreadable(f"`{field}: {written}` is not a number of cases")
return written


def _bound(written: Any, field: str) -> float:
if isinstance(written, bool) or not isinstance(written, (int, float)) or not math.isfinite(written):
raise Unreadable(f"`interval.{field}: {written}` is not a figure")
return float(written)


def measurement_of(provenance: Any, scale: "str | None") -> "Measurement | None":
"""The measurement a figure's `provenance:` records, or `None` when it records none.

`scale` is the benchmark's `scale:` (`percent` or `fraction`), or `None`
for a name no benchmark has. `Unreadable` when the measurement cannot be
one: a count that is not a whole number above zero, more cases than the
benchmark has, an interval missing a bound or its level, bounds out of
order or off the benchmark's scale, or a level that is not a share. The
catalogue drops such a figure rather than read it as its point, so a line
on it says the figure was never recorded (`crates/pact-loader/src/accept.rs`
refuses the same rows where they are written).
"""
if not isinstance(provenance, Mapping):
return None
sample = _count(provenance.get("sample-size"), "sample-size")
population = _count(provenance.get("population"), "population")
if sample is not None and population is not None and sample > population:
raise Unreadable(f"{sample} cases is more than the benchmark's {population}")
interval = provenance.get("interval")
if interval is None:
if sample is None and population is None:
return None
return Measurement(sample_size=sample, population=population)
if not isinstance(interval, Mapping):
raise Unreadable("`interval:` is not a `low`, `high` and `level`")
low, high, level = (_bound(interval.get(k), k) for k in ("low", "high", "level"))
if low > high:
raise Unreadable(f"the interval's low {low:g} is above its high {high:g}")
if not 0.0 < level < 1.0:
raise Unreadable(f"a confidence level is a share such as 0.95, not {level:g}")
top = _SCALE_TOP.get(scale or "")
if top is not None and (low < 0.0 or high > top):
raise Unreadable(f"the interval {low:g} to {high:g} is off the {scale} scale, 0 to {top:g}")
return Measurement(sample_size=sample, population=population, low=low, high=high, level=level)


def reaches(op: str, have: float, measurement: "Measurement | None", bar: float) -> bool:
"""Whether a benchmark figure meets `<op> <bar>`, held to its interval's far bound.

`>`/`>=` need `have` and the interval's low to hold, `<`/`<=` `have` and
its high; `=` is decided on `have` alone, as is a figure with no interval.
`accept-when:` and `needs.scores:` both decide through this, so a
benchmark under either is held to one bar.
"""
from .resolve import _HOLDS

op = "=" if op == "==" else op
if not _HOLDS[op](have, bar):
return False
if measurement is None or measurement.low is None or measurement.high is None or op == "=":
return True
return _HOLDS[op](measurement.low if op in (">", ">=") else measurement.high, bar)


def within_noise(measure: str, have: float, measurement: "Measurement | None", written: str) -> str:
"""The reason a line the point clears but the interval does not is refused."""
said = measurement.text() if measurement is not None else ""
return (
f"has {measure} {have:g}, {said}, which reaches past the bar, so `{written}` holds "
f"only within noise"
)


def benchmarks_of(document: Mapping[str, Any] | None = None) -> dict[str, tuple[str, str]]:
"""Every benchmark name and alias, lower-cased, to `(its key, its scale)`.

Expand Down Expand Up @@ -243,6 +376,9 @@ def held_before(line: Line, entry: Any) -> "tuple[bool, str]":
if line.kind == "benchmark":
have = (entry.scores or {}).get(line.measure)
noun = f"no recorded {line.measure} score"
taken = (getattr(entry, "measurements", None) or {}).get(line.measure)
if have is not None and line.holds(float(have)) and not reaches(line.op, float(have), taken, line.value):
return False, within_noise(line.measure, float(have), taken, line.written)
elif line.measure == "input-price":
have = None if entry.cost is None else entry.cost * 1000.0
noun = "no published input price"
Expand Down
Loading
Loading