From 1c6976fb01f62c76dfb6f12936a39dfeed94622d Mon Sep 17 00:00:00 2001 From: Jithin VG Date: Mon, 16 Nov 2026 14:16:37 +0530 Subject: [PATCH 1/4] WIP: a measured figure says how it was measured (sample-size, population, interval) --- CHANGELOG.md | 25 +++++++++ .../tests/test_every_field_has_a_reader.py | 9 ++++ docs/97-PRIMITIVES.md | 6 +++ spec/schema.yaml | 52 +++++++++++++++++++ tests/golden/schema-fields.txt | 8 ++- 5 files changed, 99 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f69e7a..655da8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,31 @@ What changed in PACT, newest first. Each entry says what an author can now write, or what now holds that did not; the commits are listed under it for whoever wants the detail. +## 2026-11 — a measured figure says how it was measured (16 November) + +A figure's `provenance:` said where a number came from but not how much of the benchmark it rests on +or how sure it is. A runtime that measures figures itself (Bud Meta Harness measures MMLU on a few +hundred questions at onboarding) had to keep that under its own `x-` key, and a reader could not tell +a 200-question score from the published full one. + +### What an author can now write + +- **`provenance.sample-size:` and `provenance.population:`** — how many of the benchmark's cases a + measured figure was taken on, out of how many it has. Fewer than the population makes it a subset + figure, read as one. +- **`provenance.interval: {low, high, level}`** — the confidence interval around `value:`, on the + figure's own scale, with the level as a share (`0.95`). A runtime may hold an `accept-when:` line to + the whole interval, so a figure that clears a bar only within its own noise is not taken as clearing + it. + +### What holds now + +- The checker reads the three fields and the new `figure-interval` group like every other field: a + key the group does not have is refused (`'width' is not something the confidence interval around a + measured figure can have`). The reference resolver still holds a line to `value:`; the fields are + listed in `test_every_field_has_a_reader.py` as read by a reviewer, and in 97's table as Store + + Actor. + ## 2026-11 — an agent's words are filled by one renderer in every port (16 November) Each runtime filled an agent's words its own way. Python rendered a `jinja` workspace with Jinja2 and diff --git a/adapters/python/tests/test_every_field_has_a_reader.py b/adapters/python/tests/test_every_field_has_a_reader.py index fbd582e..aa5b50d 100644 --- a/adapters/python/tests/test_every_field_has_a_reader.py +++ b/adapters/python/tests/test_every_field_has_a_reader.py @@ -149,6 +149,15 @@ def identifiers() -> str: # from. Nothing at run time should behave differently because of it. "provenance.recorded-by": "who wrote the figure down, for a reviewer", "provenance.contamination": "a caveat on a benchmark figure, for a reviewer", + # How a measured figure was taken. The reference holds an accept-when line + # to `value:` as before; these are for a reviewer, and for a runtime that + # holds the line to the interval or reports a subset figure as one. + "provenance.sample-size": "cases a measured figure was taken on, for a reviewer", + "provenance.population": "cases the benchmark has, read against sample-size", + "provenance.interval": "the interval around a measured figure, for a reviewer", + "figure-interval.low": "an interval's lower bound, for a reviewer", + "figure-interval.high": "an interval's upper bound, for a reviewer", + "figure-interval.level": "an interval's confidence level, for a reviewer", # ---- Surfaced when this file stopped counting its own prose as a reader. ---- # # None of the six below is new: each has been unread for as long as it has diff --git a/docs/97-PRIMITIVES.md b/docs/97-PRIMITIVES.md index fb33b36..64a125f 100644 --- a/docs/97-PRIMITIVES.md +++ b/docs/97-PRIMITIVES.md @@ -1765,6 +1765,12 @@ words for a person, with no behaviour. | `provenance.date` | Store | | `provenance.as-of` | Store | | `provenance.recorded-by` | Store | +| `provenance.sample-size` | Store + Actor | +| `provenance.population` | Store + Actor | +| `provenance.interval` | Store + Actor | +| `figure-interval.low` | Store + Actor | +| `figure-interval.high` | Store + Actor | +| `figure-interval.level` | Store + Actor | | `model-cost.input-per-mtok` | Store | | `model-cost.output-per-mtok` | Store | | `needs.reasoning` | Check | diff --git a/spec/schema.yaml b/spec/schema.yaml index dcaacf7..cd9af2d 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -2073,6 +2073,58 @@ groups: surface: S-GOV tier: expert help: who wrote it down + sample-size: + type: integer + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: > + for a measured figure, how many of the benchmark's cases it was taken + on. Fewer than `population:` makes it a subset figure, which is read + as one: a score on 200 of 14,042 questions is not the benchmark's + score, and a reader that compares it with a published full figure + says so. + population: + type: integer + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: how many cases the whole benchmark has, so `sample-size:` can be read against it + interval: + type: group:figure-interval + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: > + for a measured figure, the confidence interval around `value:`, on + the figure's own scale. A runtime may hold an `accept-when:` line to + the whole interval rather than the point, so a figure that clears a + bar only within its own noise is not taken as clearing it. + + figure-interval: + describe: something the confidence interval around a measured figure can have + fields: + low: + type: number + required: yes + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: the interval's lower bound, on the figure's scale + high: + type: number + required: yes + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: the interval's upper bound, on the figure's scale + level: + type: number + required: yes + surface: S-GOV + tier: expert + kernel-op: actor.figures + help: the confidence level, as a share such as 0.95 model-cost: describe: something a model's price can have diff --git a/tests/golden/schema-fields.txt b/tests/golden/schema-fields.txt index 97ef799..09fed24 100644 --- a/tests/golden/schema-fields.txt +++ b/tests/golden/schema-fields.txt @@ -5,6 +5,7 @@ action.claim-lasts duration | S-EXEC | expert | optional action.description text | S-ROUTE | core | optional action.gets-hidden-values yes-no | S-GOV | expert | optional action.inspects list of text | S-EXEC | core | optional +action.moves-hardware yes-no | S-EXEC | core | optional action.needs-a-person yes-no | S-EXEC | core | optional action.passes map of text | S-CAP | expert | optional action.program text | S-EXEC | expert | optional @@ -14,7 +15,6 @@ action.remember-as text | S-EXEC | expert | optional action.same-request-key text | S-EXEC | core | optional action.same-request-key-across one-of | S-EXEC | expert | optional action.spends-money yes-no | S-EXEC | core | optional -action.moves-hardware yes-no | S-EXEC | core | optional action.status-check text | S-EXEC | expert | optional action.takes map of answer-shape | S-CAP | core | optional action.topics list of text | S-GOV | core | optional @@ -118,6 +118,9 @@ event.judged yes-no | S-EXEC | expert | optional event.more-than integer | S-CTRL | core | optional event.topic text | S-EXEC | core | optional event.when-this list of group:when-this | S-EXEC | core | optional +figure-interval.high number | S-GOV | expert | required +figure-interval.level number | S-GOV | expert | required +figure-interval.low number | S-GOV | expert | required figure.provenance group:provenance | S-GOV | expert | optional figure.value text | S-CAP | expert | required if-it-fails.after-that one-of | S-EXEC | core | required @@ -291,7 +294,10 @@ provenance.as-of text | S-GOV | expert | optional provenance.contamination text | S-GOV | expert | optional provenance.date text | S-GOV | expert | optional provenance.harness text | S-GOV | expert | optional +provenance.interval group:figure-interval | S-GOV | expert | optional +provenance.population integer | S-GOV | expert | optional provenance.recorded-by text | S-GOV | expert | optional +provenance.sample-size integer | S-GOV | expert | optional provenance.source text | S-GOV | expert | required published-tool.answers-with anything | S-CAP | expert | optional published-tool.description text | S-GEN | expert | optional From 03971bed09275d91ca365b71869a08d0f0cba900 Mon Sep 17 00:00:00 2001 From: Jithin VG Date: Mon, 16 Nov 2026 23:47:38 +0530 Subject: [PATCH 2/4] A line is held to a measured figure's interval, in every runtime On a benchmark figure with provenance.interval, a >/>= line holds only when value and interval.low hold, and a `/`>=` line holds only when `value:` and `interval.low` both hold, and a `<`/`<=` + line only when `value:` and `interval.high` both hold; `=` is decided on `value:`. `mmlu: 78` with a + 95% interval of 71.6 to 83.1 does not clear `mmlu.score > 75`: *"has mmlu 78, on a subset of 200 of + 14042 cases, 95% interval 71.6 to 83.1, which reaches past the bar, so `mmlu.score > 75` holds only + within noise"*. A figure without an interval decides on `value:` as before. +- **`needs.scores:` holds a benchmark to the same bar.** It compared the point alone, so one benchmark + under `scores:` and under `accept-when:` could decide differently. +- **The checker refuses a measurement that cannot be one**, where it is written: + `loader/an-interval-out-of-order`, `loader/a-confidence-level-that-is-not-a-share`, + `loader/more-cases-than-the-benchmark-has`, `loader/an-interval-off-the-benchmark-scale`, and + `loader/a-measurement-on-a-figure-nothing-samples` (a reasoning rung, a context window or a speed), + and warns `loader/a-subset-figure-with-no-interval`. The reference resolver drops such a figure, so + a line on it says the figure was never recorded rather than reading it as its point. +- `spec/accept-when.yaml` gains `figures:` (decisions) and `not-measurements:` (refusals), read by the + Rust checker and the Python resolver alike. +- The golden `tests/golden/schema-fields.txt` re-sorts `action.moves-hardware`: `field_inventory` is + ordered, and master had a hand-inserted line out of order. ## 2026-11 — PACT Core, first phase: actors and functions the checker reads, and a workspace that changes while it runs (16 November) diff --git a/adapters/python/src/pact_adapters/resolve.py b/adapters/python/src/pact_adapters/resolve.py index b92d538..0c4cc81 100644 --- a/adapters/python/src/pact_adapters/resolve.py +++ b/adapters/python/src/pact_adapters/resolve.py @@ -41,7 +41,7 @@ from dataclasses import dataclass, field, replace from functools import lru_cache from pathlib import Path -from typing import Any, Callable, Mapping +from typing import Any, Callable, Mapping, TypeVar import yaml @@ -160,6 +160,11 @@ class ModelEntry: output_cost: "float | None" = None capabilities: frozenset[str] = frozenset() scores: dict[str, float] = None # type: ignore[assignment] + #: How each benchmark figure in `scores` was measured, keyed like `scores`; + #: a figure published without `sample-size:`, `population:` or `interval:` + #: has no entry. `accept-when:` and `needs.scores:` hold a line to the + #: interval's far bound as well as the figure (`selection.reaches`). + measurements: "Mapping[str, _selection.Measurement]" = field(default_factory=dict) #: How much this model can hold, in tokens. `None` means the catalogue says #: `unknown` — not "small", and not "ask again later". It is here rather than #: on a second record beside it because a window is a fact about a model, and @@ -297,6 +302,11 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]: op, value = wanted if not _HOLDS[op](have, value): return False, f"{metric} {have} is not {op} {value:g}" + # The bar `accept-when:` holds a benchmark to, so the same figure + # under either spelling decides the same way. + taken = self.measurements.get(metric) + if not _selection.reaches(op, have, taken, value): + return False, _selection.within_noise(metric, have, taken, f"scores.{metric}: {threshold}") wanted = _tokens(needs.get("context-at-least")) if wanted is not None and self.context_window is not None: if self.context_window < wanted: @@ -724,13 +734,15 @@ def _parse(text: str, shown: str) -> Catalogue: window, trouble = _window(shown, lines, name, caps.get("context-window")) if trouble is not None: problems.append(trouble) + scores, measurements = _benchmarks(row.get("benchmarks"), names) entry = ModelEntry( name=name, tier=str(row.get("tier") or "mid"), cost=_cost(row.get("cost")), output_cost=_cost(row.get("cost"), "output-per-mtok"), capabilities=_capabilities(caps), - scores=_canonical(_benchmarks(row.get("benchmarks")), names), + scores=_canonical(scores, names), + measurements=_canonical(measurements, names), context_window=window, aliases=frozenset( str(a) for a in (row.get("also-known-as") or []) if isinstance(a, str) @@ -970,8 +982,10 @@ def _threshold(written: Any) -> "tuple[str, float] | None": return None -def _benchmarks(written: Any) -> dict[str, float]: - """Published benchmark figures for one row, by metric name. +def _benchmarks( + written: Any, names: "Mapping[str, tuple[str, str]]", +) -> "tuple[dict[str, float], dict[str, _selection.Measurement]]": + """Published benchmark figures for one row, by metric name, and how each was measured. This came back `{}` unconditionally — the argument was not even read — so `satisfies()` answered *"no published {metric} score"* for every candidate on @@ -989,18 +1003,36 @@ def _benchmarks(written: Any) -> dict[str, float]: A figure written `unknown`, or one that is not a number, is DROPPED rather than kept as zero. Zero would satisfy no threshold and read as a measurement; absent is the truth, and `satisfies()` says *"no published X score"* about it. + + A figure whose `provenance:` records a measurement that cannot be one + (`selection.measurement_of`: an interval out of order or off the benchmark's + scale named in `names`, a level that is not a share, more cases than the + benchmark has) is dropped the same way, never read as its point figure. """ if not isinstance(written, dict): - return {} + return {}, {} out: dict[str, float] = {} + measured: dict[str, _selection.Measurement] = {} for metric, figure in written.items(): + provenance = None if isinstance(figure, dict): - figure = figure.get("value") + figure, provenance = figure.get("value"), figure.get("provenance") try: - out[str(metric)] = float(figure) # type: ignore[arg-type] + value = float(figure) # type: ignore[arg-type] except (TypeError, ValueError): continue # `unknown`, or prose — see the docstring - return out + known = names.get(str(metric).strip().lower()) + try: + taken = _selection.measurement_of(provenance, known[1] if known else None) + except _selection.Unreadable: + continue # a measurement that cannot be one — see the docstring + out[str(metric)] = value + if taken is not None: + measured[str(metric)] = taken + return out, measured + + +_Figure = TypeVar("_Figure") def _milliseconds(written: Any) -> "float | None": @@ -1016,8 +1048,8 @@ def _milliseconds(written: Any) -> "float | None": return None if found is None else found * 1000.0 -def _canonical(scores: dict[str, float], names: dict[str, tuple[str, str]]) -> dict[str, float]: - """A row's scores under the names it wrote AND each benchmark's own key.""" +def _canonical(scores: "dict[str, _Figure]", names: "Mapping[str, tuple[str, str]]") -> "dict[str, _Figure]": + """A row's scores, or their measurements, under the names it wrote AND each benchmark's own key.""" out = dict(scores) for written, figure in scores.items(): key = names.get(written.strip().lower()) diff --git a/adapters/python/src/pact_adapters/selection.py b/adapters/python/src/pact_adapters/selection.py index 7918017..2a87191 100644 --- a/adapters/python/src/pact_adapters/selection.py +++ b/adapters/python/src/pact_adapters/selection.py @@ -30,6 +30,18 @@ accepted by `mmlu.score > 50`, an unpriced one by `input-price < 3 USD`, one whose cost was never metered by `cost < 0.05 USD`. Each says why, so the author can record the figure (`benchmarks.record`) rather than guess. + +## A measured figure + +A benchmark figure may say how it was measured: `provenance.sample-size:` and +`population:` (how many of the benchmark's cases it rests on, out of how many), +and `provenance.interval:` (`low`, `high`, `level`). A line with `>` or `>=` +holds only when both `value:` and `interval.low` hold; one with `<` or `<=` +only when both `value:` and `interval.high` hold; `=` is decided on `value:` +alone. So a score of 78 with a 95% interval of 71.6 to 83.1 does not clear +`mmlu.score > 75`: the line holds only within noise, and the reason says so. +Every runtime holds the line this way, so one tree and one catalogue bind one +model wherever they are read. """ from __future__ import annotations @@ -45,13 +57,17 @@ "AFTER", "BEFORE", "Line", + "Measurement", "Unreadable", "benchmarks_of", "held_after", "held_before", "measured", + "measurement_of", "parse", "parse_all", + "reaches", + "within_noise", ] #: The figures every model has, and the unit each is written in. @@ -100,6 +116,123 @@ def holds(self, have: float) -> bool: return _HOLDS["=" if self.op == "==" else self.op](have, self.value) +#: The highest a figure can be on each benchmark scale; the lowest is zero. +_SCALE_TOP = {"percent": 100.0, "fraction": 1.0} + + +@dataclass(frozen=True) +class Measurement: + """How a recorded benchmark figure was measured (`provenance.sample-size`, + `population` and `interval`). A bound is on the figure's own scale.""" + + sample_size: "int | None" = None + population: "int | None" = None + low: "float | None" = None + high: "float | None" = None + #: The confidence level, as a share such as `0.95`. + level: "float | None" = None + + @property + def subset(self) -> bool: + """Taken on fewer cases than the benchmark has.""" + return ( + self.sample_size is not None + and self.population is not None + and self.sample_size < self.population + ) + + def text(self) -> str: + """`on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1`, or `""`. + + The wording Bud Meta Harness prints for the same figure, so a refusal + reads the same in both runtimes. + """ + parts = [] + if self.sample_size is not None: + of = "" if self.population is None else f" of {self.population}" + parts.append(f"{'on a subset of' if self.subset else 'on'} {self.sample_size}{of} cases") + if self.low is not None and self.high is not None and self.level is not None: + parts.append(f"{self.level * 100:g}% interval {self.low:g} to {self.high:g}") + return ", ".join(parts) + + +def _count(written: Any, field: str) -> "int | None": + if written is None: + return None + if isinstance(written, bool) or not isinstance(written, int) or written < 1: + raise Unreadable(f"`{field}: {written}` is not a number of cases") + return written + + +def _bound(written: Any, field: str) -> float: + if isinstance(written, bool) or not isinstance(written, (int, float)) or not math.isfinite(written): + raise Unreadable(f"`interval.{field}: {written}` is not a figure") + return float(written) + + +def measurement_of(provenance: Any, scale: "str | None") -> "Measurement | None": + """The measurement a figure's `provenance:` records, or `None` when it records none. + + `scale` is the benchmark's `scale:` (`percent` or `fraction`), or `None` + for a name no benchmark has. `Unreadable` when the measurement cannot be + one: a count that is not a whole number above zero, more cases than the + benchmark has, an interval missing a bound or its level, bounds out of + order or off the benchmark's scale, or a level that is not a share. The + catalogue drops such a figure rather than read it as its point, so a line + on it says the figure was never recorded (`crates/pact-loader/src/accept.rs` + refuses the same rows where they are written). + """ + if not isinstance(provenance, Mapping): + return None + sample = _count(provenance.get("sample-size"), "sample-size") + population = _count(provenance.get("population"), "population") + if sample is not None and population is not None and sample > population: + raise Unreadable(f"{sample} cases is more than the benchmark's {population}") + interval = provenance.get("interval") + if interval is None: + if sample is None and population is None: + return None + return Measurement(sample_size=sample, population=population) + if not isinstance(interval, Mapping): + raise Unreadable("`interval:` is not a `low`, `high` and `level`") + low, high, level = (_bound(interval.get(k), k) for k in ("low", "high", "level")) + if low > high: + raise Unreadable(f"the interval's low {low:g} is above its high {high:g}") + if not 0.0 < level < 1.0: + raise Unreadable(f"a confidence level is a share such as 0.95, not {level:g}") + top = _SCALE_TOP.get(scale or "") + if top is not None and (low < 0.0 or high > top): + raise Unreadable(f"the interval {low:g} to {high:g} is off the {scale} scale, 0 to {top:g}") + return Measurement(sample_size=sample, population=population, low=low, high=high, level=level) + + +def reaches(op: str, have: float, measurement: "Measurement | None", bar: float) -> bool: + """Whether a benchmark figure meets ` `, held to its interval's far bound. + + `>`/`>=` need `have` and the interval's low to hold, `<`/`<=` `have` and + its high; `=` is decided on `have` alone, as is a figure with no interval. + `accept-when:` and `needs.scores:` both decide through this, so a + benchmark under either is held to one bar. + """ + from .resolve import _HOLDS + + op = "=" if op == "==" else op + if not _HOLDS[op](have, bar): + return False + if measurement is None or measurement.low is None or measurement.high is None or op == "=": + return True + return _HOLDS[op](measurement.low if op in (">", ">=") else measurement.high, bar) + + +def within_noise(measure: str, have: float, measurement: "Measurement | None", written: str) -> str: + """The reason a line the point clears but the interval does not is refused.""" + said = measurement.text() if measurement is not None else "" + return ( + f"has {measure} {have:g}, {said}, which reaches past the bar, so `{written}` holds " + f"only within noise" + ) + + def benchmarks_of(document: Mapping[str, Any] | None = None) -> dict[str, tuple[str, str]]: """Every benchmark name and alias, lower-cased, to `(its key, its scale)`. @@ -243,6 +376,9 @@ def held_before(line: Line, entry: Any) -> "tuple[bool, str]": if line.kind == "benchmark": have = (entry.scores or {}).get(line.measure) noun = f"no recorded {line.measure} score" + taken = (getattr(entry, "measurements", None) or {}).get(line.measure) + if have is not None and line.holds(float(have)) and not reaches(line.op, float(have), taken, line.value): + return False, within_noise(line.measure, float(have), taken, line.written) elif line.measure == "input-price": have = None if entry.cost is None else entry.cost * 1000.0 noun = "no published input price" diff --git a/adapters/python/tests/test_an_accept_line_means_one_thing.py b/adapters/python/tests/test_an_accept_line_means_one_thing.py index e7ad2a7..030b4da 100644 --- a/adapters/python/tests/test_an_accept_line_means_one_thing.py +++ b/adapters/python/tests/test_an_accept_line_means_one_thing.py @@ -51,6 +51,103 @@ def test_every_line_the_checker_refuses_is_refused_here_too(row: dict) -> None: assert not said or any(s in str(why.value) for s in said), str(why.value) +_MEASURED = ("sample-size", "population", "interval") + + +def _measured_row(tmp_path: Path, row: dict) -> resolve.ModelEntry: + """The table's world as a workspace catalogue whose one row records the vector's figure.""" + provenance = {"source": "measured here", **{k: row[k] for k in _MEASURED if k in row}} + catalogue = { + "version": 1, + "benchmarks": {name: {"scale": scale} for name, scale in TABLE["benchmarks"].items()}, + "models": {"m": {"benchmarks": {row.get("benchmark", "mmlu"): {"value": row["value"], "provenance": provenance}}}}, + } + (tmp_path / "models").mkdir() + (tmp_path / "models/catalog.yaml").write_text(yaml.safe_dump(catalogue, sort_keys=False)) + entry = resolve.load_catalogue(workspace=tmp_path).get("m") + assert entry is not None + return entry + + +def _bar(row: dict) -> dict: + """The vector's bar as the `needs` `satisfies` reads: an `accept-when:` line or `needs.scores:`.""" + if "scores" in row: + return {"scores": row["scores"]} + return {"accept-when": (_line(row["written"]),)} + + +@pytest.mark.parametrize("row", TABLE["figures"], ids=lambda r: f"{r.get('written') or r.get('scores')}: {r['because']}") +def test_a_measured_figure_is_held_to_its_interval_as_the_table_says(tmp_path: Path, row: dict) -> None: + ok, why = _measured_row(tmp_path, row).satisfies(_bar(row)) + assert ok == (row["holds"] is True), why + if "says" in row: + assert why == row["says"] + + +@pytest.mark.parametrize( + "row", [r for r in TABLE["not-measurements"] if "field" not in r], ids=lambda r: r["rule"] + ": " + r["because"], +) +def test_a_measurement_that_cannot_be_one_drops_its_figure(tmp_path: Path, row: dict) -> None: + benchmark = row.get("benchmark", "mmlu") + entry = _measured_row(tmp_path, row) + assert benchmark not in entry.scores and benchmark not in entry.measurements + ok, why = entry.satisfies({"accept-when": (_line(f"{benchmark} >= 0"),)}) + assert not ok and f"no recorded {benchmark} score" in why + + +AUTHORED_CATALOGUE = """\ +version: 1 +models: + measured-here: + description: A model this machine serves, with an MMLU figure taken on a subset. + served-by: + - { runtime: ollama, endpoint: local } + benchmarks: + mmlu: + value: "78" + provenance: + source: measured on this machine + sample-size: 200 + population: 14042 + interval: { low: 71.6, high: 83.1, level: 0.95 } +""" + + +@pytest.mark.parametrize( + "needs, written", + [ + ("because: it answers exam questions\naccept-when:\n - mmlu.score > 75\n", "mmlu.score > 75"), + ("because: it answers exam questions\nscores:\n mmlu: \"> 75\"\n", "scores.mmlu: > 75"), + ], + ids=["accept-when", "needs.scores"], +) +def test_an_authored_figure_that_clears_a_bar_only_within_noise_binds_no_model( + tmp_path: Path, needs: str, written: str, +) -> None: + """The authored path: `models/catalog.yaml` and `needs.yaml` written, checked by `pact check`, + shown by `pact show`, and bound the way the suite binds the model it scores on.""" + from pact_adapters.ir import AgentSpec + from pact_adapters.scoring import _bind + from trees import pact + + root = tmp_path / "w" + (root / "agents/desk").mkdir(parents=True) + (root / "models").mkdir() + (root / "workspace.yaml").write_text("name: measured\nallow-egress: []\n") + (root / "agents/desk/agent.yaml").write_text("description: answers\ninstructions: Answer.\nmodel: measured-here\n") + (root / "agents/desk/needs.yaml").write_text(needs) + (root / "models/catalog.yaml").write_text(AUTHORED_CATALOGUE) + checked = pact("check", str(root)) + assert checked.returncode == 0, checked.stdout + checked.stderr + doc = shown(root) + entry, said, problem = _bind(doc, AgentSpec.from_document(doc, "desk", source=root), "desk", "", "", root) + assert entry is None and problem is None + assert ( + "has mmlu 78, on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1, which reaches past " + f"the bar, so `{written}` holds only within noise" + ) in said, said + + def _entry(**kw) -> resolve.ModelEntry: return resolve.ModelEntry(**{"name": "m", "tier": "mid", "cost": 0.001, "scores": {}, **kw}) diff --git a/adapters/python/tests/test_every_field_has_a_reader.py b/adapters/python/tests/test_every_field_has_a_reader.py index aa5b50d..fbd582e 100644 --- a/adapters/python/tests/test_every_field_has_a_reader.py +++ b/adapters/python/tests/test_every_field_has_a_reader.py @@ -149,15 +149,6 @@ def identifiers() -> str: # from. Nothing at run time should behave differently because of it. "provenance.recorded-by": "who wrote the figure down, for a reviewer", "provenance.contamination": "a caveat on a benchmark figure, for a reviewer", - # How a measured figure was taken. The reference holds an accept-when line - # to `value:` as before; these are for a reviewer, and for a runtime that - # holds the line to the interval or reports a subset figure as one. - "provenance.sample-size": "cases a measured figure was taken on, for a reviewer", - "provenance.population": "cases the benchmark has, read against sample-size", - "provenance.interval": "the interval around a measured figure, for a reviewer", - "figure-interval.low": "an interval's lower bound, for a reviewer", - "figure-interval.high": "an interval's upper bound, for a reviewer", - "figure-interval.level": "an interval's confidence level, for a reviewer", # ---- Surfaced when this file stopped counting its own prose as a reader. ---- # # None of the six below is new: each has been unread for as long as it has diff --git a/crates/pact-loader/src/accept.rs b/crates/pact-loader/src/accept.rs index 9d853b2..595f44a 100644 --- a/crates/pact-loader/src/accept.rs +++ b/crates/pact-loader/src/accept.rs @@ -30,6 +30,18 @@ //! | `loader/two-metrics-one-name` | two of the suite's metrics would be called by one name | //! | `loader/a-metric-named-like-a-figure` | a metric is called by a benchmark's or a figure's name | //! | `loader/a-benchmark-nobody-has-measured` (warning) | no catalogue row records the benchmark a line names, so the line accepts no model yet | +//! | `loader/an-interval-out-of-order` | a workspace figure's `provenance.interval` has its `low` above its `high` | +//! | `loader/a-confidence-level-that-is-not-a-share` | an interval's `level` is not between zero and one (`95` for `0.95`) | +//! | `loader/more-cases-than-the-benchmark-has` | a figure's `sample-size` is above its `population` | +//! | `loader/an-interval-off-the-benchmark-scale` | a bound is below zero or above the top of the benchmark's `scale:` (100 for `percent`, 1 for `fraction`) | +//! | `loader/a-measurement-on-a-figure-nothing-samples` | `sample-size`, `population` or `interval` on a `reasoning:`, `context-window` or `first-words-within` figure | +//! | `loader/a-subset-figure-with-no-interval` (warning) | a figure taken on fewer cases than the benchmark has records no interval, so a line reads it as if it were the whole benchmark's | +//! +//! A benchmark figure's measurement (`provenance.sample-size`, `population`, +//! `interval`) changes what a line decides: `>`/`>=` hold only when `value:` +//! and `interval.low` hold, `<`/`<=` only when `value:` and `interval.high` +//! hold. The resolver drops a figure whose measurement is refused here, so a +//! line on it accepts no model. use pact_diag::{Diagnostic, Diagnostics, Span}; use pact_doc::{Node, Value}; @@ -395,8 +407,161 @@ pub fn check(document: &Node, builtin: &str, diags: &mut Diagnostics) { } } +/// The provenance fields that say how a figure was measured. +const MEASURED: [&str; 3] = ["sample-size", "population", "interval"]; + +/// The highest a figure can be on a benchmark `scale:`; the lowest is zero. +fn scale_top(scale: &str) -> Option { + match scale { + "percent" => Some(100.0), + "fraction" => Some(1.0), + _ => None, + } +} + +/// A number as YAML wrote it. +fn number_in(node: &Node) -> Option { + match node.value { + Value::Int(i) => Some(i as f64), + Value::Float(f) => Some(f), + _ => None, + } +} + +/// Refuse a measurement the resolver could not read on a workspace row's +/// benchmark figure (`selection.measurement_of`), where the author wrote it. +fn measurement(model: &str, name: &str, scale: &str, figure: &Node, diags: &mut Diagnostics) { + let Some(provenance) = figure.get("provenance") else { + return; + }; + let count = |key: &str| match provenance.get(key).map(|n| &n.value) { + Some(Value::Int(i)) => Some(*i), + _ => None, + }; + let (sample, population) = (count("sample-size"), count("population")); + if let (Some(sample), Some(population)) = (sample, population) + && sample > population + { + diags.push(Diagnostic::error( + "loader/more-cases-than-the-benchmark-has", + provenance + .get("sample-size") + .map_or(figure.span.clone(), |n| n.span.clone()), + format!( + "'{model}' says its {name} figure was taken on {sample} cases, and the \ + benchmark has {population}." + ), + "Write the number of cases the figure was measured on as `sample-size:`, and the \ + number the whole benchmark has as `population:`.", + )); + } + let Some(interval) = provenance.get("interval") else { + if let (Some(sample), Some(population)) = (sample, population) + && sample < population + { + diags.push(Diagnostic::warning( + "loader/a-subset-figure-with-no-interval", + provenance + .get("sample-size") + .map_or(figure.span.clone(), |n| n.span.clone()), + format!( + "'{model}' took its {name} figure on {sample} of {population} cases and \ + records no interval, so a line reads it as if it were the whole \ + benchmark's score." + ), + "Add `interval: {low, high, level}` from the measurement, so a line is held \ + to how far the figure could be off.", + )); + } + return; + }; + let bound = |key: &str| interval.get(key).and_then(number_in); + if let (Some(low), Some(high)) = (bound("low"), bound("high")) { + if low > high { + diags.push(Diagnostic::error( + "loader/an-interval-out-of-order", + interval.span.clone(), + format!( + "'{model}' gives its {name} figure an interval from {low} to {high}, and the \ + low end is above the high end." + ), + "Swap `low:` and `high:`.", + )); + } + if let Some(top) = scale_top(scale) + && (low < 0.0 || high > top) + { + diags.push(Diagnostic::error( + "loader/an-interval-off-the-benchmark-scale", + interval.span.clone(), + format!( + "'{model}' gives its {name} figure an interval from {low} to {high}, and \ + {name} is scored on the {scale} scale, from 0 to {top}." + ), + format!("Write both bounds on the benchmark's own scale, from 0 to {top}."), + )); + } + } + if let Some(level) = bound("level") + && !(level > 0.0 && level < 1.0) + { + diags.push(Diagnostic::error( + "loader/a-confidence-level-that-is-not-a-share", + interval + .get("level") + .map_or(interval.span.clone(), |n| n.span.clone()), + format!( + "'{model}' gives its {name} interval a confidence level of {level}, and a \ + level is a share between zero and one." + ), + "Write it as a share, like `0.95` for 95%.", + )); + } +} + +/// Refuse a measurement on a workspace row's figure that is not a benchmark +/// score: a reasoning rung, a context window and a speed are not sampled. +fn unsampled(model: &str, row: &Node, diags: &mut Diagnostics) { + let capabilities = row.get("capabilities"); + let speed = row.get("speed"); + for (field, figure) in [ + ("reasoning", row.get("reasoning")), + ( + "context-window", + capabilities.and_then(|c| c.get("context-window")), + ), + ( + "first-words-within", + speed.and_then(|s| s.get("first-words-within")), + ), + ] { + let Some(provenance) = figure + .and_then(|f| f.get("provenance")) + .and_then(Node::as_map) + else { + continue; + }; + for (key, entry) in provenance { + if MEASURED.contains(&key.as_str()) { + diags.push(Diagnostic::error( + "loader/a-measurement-on-a-figure-nothing-samples", + entry.key_span.clone(), + format!( + "'{model}' says how its {field} was sampled (`{key}:`), and {field} is \ + not taken on a sample of a benchmark's cases." + ), + format!( + "Take `{key}:` out; it belongs on a benchmark figure under `benchmarks:`." + ), + )); + } + } + } +} + /// Every benchmark some row of either catalogue has a score for, by its key — -/// refusing a workspace row's score under a name no benchmark has. +/// refusing a workspace row's score under a name no benchmark has, and a +/// workspace figure's measurement the resolver could not read. fn recorded( builtin: &str, document: &Node, @@ -410,6 +575,9 @@ fn recorded( continue; }; for (model, row) in rows { + if own { + unsampled(model, &row.node, diags); + } for (name, figure) in row .node .get("benchmarks") @@ -418,8 +586,11 @@ fn recorded( .flatten() { match benchmarks.get(&name.trim().to_lowercase()) { - Some((key, _)) => { + Some((key, scale)) => { out.insert(key.clone()); + if own { + measurement(model, name, scale, &figure.node, diags); + } } None if own => diags.push(Diagnostic::error( "loader/a-score-for-no-benchmark", @@ -506,4 +677,92 @@ mod tests { ); } } + + /// A workspace catalogue holding one row's figure, in the table's world, + /// checked the way `pact check` checks it: what `recorded` reports. + fn measured(table: &Node, row: &Node) -> Diagnostics { + use serde_json::{Map as J, json}; + let mut provenance = J::new(); + provenance.insert("source".into(), json!("measured here")); + for key in MEASURED { + if let Some(field) = row.get(key) { + provenance.insert(key.into(), field.to_json()); + } + } + let figure = + json!({"value": row.get("value").unwrap().to_json(), "provenance": provenance}); + let model = match row.get("field").and_then(Node::as_str) { + Some("context-window") => json!({"capabilities": {"context-window": figure}}), + Some("first-words-within") => json!({"speed": {"first-words-within": figure}}), + Some(other) => panic!("no figure called {other}"), + None => { + let on = row + .get("benchmark") + .and_then(Node::as_str) + .unwrap_or("mmlu"); + json!({"benchmarks": {on: figure}}) + } + }; + let scales: J = table + .get("benchmarks") + .and_then(Node::as_map) + .unwrap() + .iter() + .map(|(name, scale)| (name.clone(), json!({"scale": scale.node.as_str().unwrap()}))) + .collect(); + let document = json!({"models": {"benchmarks": scales, "models": {"m": model}}}); + let document = pact_doc::parse_yaml( + &document.to_string(), + camino::Utf8Path::new("workspace.yaml"), + ) + .unwrap(); + let mut diags = Diagnostics::new(); + recorded("", &document, &benchmarks("", &document), &mut diags); + diags + } + + /// The table's `figures:` are measurements the checker reads without an + /// error, and its `not-measurements:` are refused with their `rule:`. + #[test] + fn every_measurement_in_the_shared_table_is_checked_as_it_says() { + let table = + pact_doc::parse_yaml(VECTORS, camino::Utf8Path::new("accept-when.yaml")).unwrap(); + let rules = |d: &Diagnostics, severity: pact_diag::Severity| -> Vec<&'static str> { + d.items() + .iter() + .filter(|i| i.severity == severity) + .map(|i| i.rule) + .collect() + }; + for row in table.get("figures").and_then(Node::as_list).unwrap() { + let diags = measured(&table, row); + let said = row + .get("written") + .map_or_else(|| format!("{:?}", row.get("scores")), |w| format!("{w:?}")); + assert_eq!( + rules(&diags, pact_diag::Severity::Error), + Vec::<&str>::new(), + "{said}" + ); + let warns: Vec<&str> = row + .get("warns") + .and_then(Node::as_str) + .into_iter() + .collect(); + assert_eq!(rules(&diags, pact_diag::Severity::Warning), warns, "{said}"); + } + for row in table + .get("not-measurements") + .and_then(Node::as_list) + .unwrap() + { + let diags = measured(&table, row); + let rule = row.get("rule").and_then(Node::as_str).unwrap(); + assert_eq!( + rules(&diags, pact_diag::Severity::Error), + vec![rule], + "{rule}" + ); + } + } } diff --git a/docs/97-PRIMITIVES.md b/docs/97-PRIMITIVES.md index 64a125f..bc5b981 100644 --- a/docs/97-PRIMITIVES.md +++ b/docs/97-PRIMITIVES.md @@ -1765,12 +1765,12 @@ words for a person, with no behaviour. | `provenance.date` | Store | | `provenance.as-of` | Store | | `provenance.recorded-by` | Store | -| `provenance.sample-size` | Store + Actor | -| `provenance.population` | Store + Actor | -| `provenance.interval` | Store + Actor | -| `figure-interval.low` | Store + Actor | -| `figure-interval.high` | Store + Actor | -| `figure-interval.level` | Store + Actor | +| `provenance.sample-size` | Store + Check | +| `provenance.population` | Store + Check | +| `provenance.interval` | Store + Check | +| `figure-interval.low` | Store + Check | +| `figure-interval.high` | Store + Check | +| `figure-interval.level` | Store | | `model-cost.input-per-mtok` | Store | | `model-cost.output-per-mtok` | Store | | `needs.reasoning` | Check | diff --git a/site-docs/index.md b/site-docs/index.md index 97ff0d6..21585a2 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -33,7 +33,7 @@ requires it, and deleting it leaves a working agent. - **[What do you write?](concepts/what.md)** Files. One rule — *a directory is a field; a field may be a directory* — - and 69 kinds you never have to memorise. + and 70 kinds you never have to memorise. @@ -48,7 +48,7 @@ so plainly rather than leaving it to be discovered. | | | |---|---| | Tests | **5,310** — 1719 Rust, 3591 adapter | -| Schema kinds | **69** | +| Schema kinds | **70** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | | Worked example | 43 files, 498 settings, 1 payload script | diff --git a/site-docs/reference/kinds.md b/site-docs/reference/kinds.md index 2e55950..f3e0805 100644 --- a/site-docs/reference/kinds.md +++ b/site-docs/reference/kinds.md @@ -1,11 +1,11 @@ -# The 69 kinds +# The 70 kinds Generated from `spec/schema.yaml`. Every kind, what it is for, and how many fields it has. You do not need to learn these — `pact check` names the right one when you need it. -**69 kinds.** A kind is a *shape a document can take*, not a file you must +**70 kinds.** A kind is a *shape a document can take*, not a file you must write: most workspaces use a dozen. @@ -77,7 +77,8 @@ write: most workspaces use a dozen. | `model-cost` | 2 | — | | `model-speed` | 1 | — | | `needs` | 12 | `reasoning`, `tool-calling`, `images`, `audio`, `computer-use` | -| `provenance` | 6 | — | +| `provenance` | 9 | — | +| `figure-interval` | 3 | — | | `served-by` | 3 | — | ## Memory and context diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 82ee84b..6efe99a 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -18,7 +18,7 @@ from an earlier document. | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` | -| Schema kinds | **69** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | +| Schema kinds | **70** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | | Fields that resolve a name | **78** | `grep -cE '^\s*names:' spec/schema.yaml` | ## What is proven, and what "proven" means diff --git a/spec/accept-when.yaml b/spec/accept-when.yaml index c75b7bf..ffac1b6 100644 --- a/spec/accept-when.yaml +++ b/spec/accept-when.yaml @@ -47,3 +47,44 @@ not-lines: - {written: "banana > 3", rule: loader/no-such-measure, because: "nothing is called that"} - {written: "mmlu", rule: loader/not-a-comparison, because: "no comparison at all"} - {written: "mmlu >", rule: loader/not-a-comparison, because: "a comparison with nothing to compare against"} + +# A benchmark figure that says how it was measured (`provenance.sample-size`, +# `population`, `interval`). Each row is one catalogue figure — `value:` and the +# provenance fields beside it, on `benchmark:` (default `mmlu`) — and one bar, +# written as an `accept-when:` line (`written:`) or a `needs.scores:` entry +# (`scores:`). `holds:` is the decision; `says:` is the reason both runtimes +# print, word for word, when it is given; `warns:` is the checker's warning on +# the figure. `>`/`>=` hold only when `value:` and `interval.low` hold, `<`/`<=` +# only when `value:` and `interval.high` hold, and `=` is decided on `value:`. +figures: + - written: "mmlu.score > 75" + value: "78" + sample-size: 200 + population: 14042 + interval: {low: 71.6, high: 83.1, level: 0.95} + holds: no + says: "has mmlu 78, on a subset of 200 of 14042 cases, 95% interval 71.6 to 83.1, which reaches past the bar, so `mmlu.score > 75` holds only within noise" + because: the point clears the bar and the interval's low end does not + - {written: "mmlu.score > 75", value: "78", interval: {low: 76, high: 83.1, level: 0.95}, holds: yes, because: the whole interval clears the bar} + - {written: "mmlu.score > 80", value: "78", interval: {low: 71.6, high: 83.1, level: 0.95}, holds: no, says: "has mmlu 78, so `mmlu.score > 80` does not hold", because: the point itself falls short} + - {written: "mmlu.score <= 80", value: "78", interval: {low: 71.6, high: 83.1, level: 0.95}, holds: no, because: "a `<=` line is read on the interval's high end"} + - {written: "mmlu.score <= 85", value: "78", interval: {low: 71.6, high: 83.1, level: 0.95}, holds: yes, because: the high end is under the bar too} + - {written: "mmlu.score = 78", value: "78", interval: {low: 71.6, high: 83.1, level: 0.95}, holds: yes, because: "`=` is decided on the point alone"} + - {written: "win-rate >= 60%", benchmark: win-rate, value: "0.65", interval: {low: 0.55, high: 0.75, level: 0.9}, holds: no, because: a fraction benchmark's interval is out of one} + - {written: "mmlu.score > 75", value: "78", sample-size: 200, population: 14042, holds: yes, warns: loader/a-subset-figure-with-no-interval, because: with no interval the line decides on the point} + - {written: "mmlu.score > 75", value: "78", sample-size: 14042, population: 14042, holds: yes, because: a figure on every case is no subset} + - {scores: {mmlu: "> 75"}, value: "78", interval: {low: 71.6, high: 83.1, level: 0.95}, holds: no, says: "has mmlu 78, 95% interval 71.6 to 83.1, which reaches past the bar, so `scores.mmlu: > 75` holds only within noise", because: "`needs.scores:` holds a benchmark to the bar `accept-when:` does"} + - {scores: {mmlu: "> 75"}, value: "78", interval: {low: 76, high: 83.1, level: 0.95}, holds: yes, because: the whole interval clears the bar} + +# A measurement that cannot be one. The checker refuses it with `rule:` where +# it is written; the reference resolver drops the figure, so a line on it +# accepts no model (`no recorded mmlu score`). `field:` names a figure other than +# a benchmark score, which nothing samples. +not-measurements: + - {value: "78", interval: {low: 83.1, high: 71.6, level: 0.95}, rule: loader/an-interval-out-of-order, because: the low end is above the high end} + - {value: "78", interval: {low: 71.6, high: 83.1, level: 95}, rule: loader/a-confidence-level-that-is-not-a-share, because: a level is a share such as 0.95} + - {value: "78", sample-size: 20000, population: 14042, rule: loader/more-cases-than-the-benchmark-has, because: more cases than the benchmark has} + - {value: "78", interval: {low: 71.6, high: 101, level: 0.95}, rule: loader/an-interval-off-the-benchmark-scale, because: a percent benchmark is out of 100} + - {benchmark: win-rate, value: "0.65", interval: {low: 0.55, high: 1.2, level: 0.95}, rule: loader/an-interval-off-the-benchmark-scale, because: a fraction benchmark is out of one} + - {field: context-window, value: "128k", interval: {low: 120000, high: 130000, level: 0.95}, rule: loader/a-measurement-on-a-figure-nothing-samples, because: a context window is not sampled} + - {field: first-words-within, value: "800ms", sample-size: 50, rule: loader/a-measurement-on-a-figure-nothing-samples, because: a speed is a length of time, and a bare low and high would be ambiguous} diff --git a/spec/schema.yaml b/spec/schema.yaml index 59ba73a..d49a183 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -2075,31 +2075,34 @@ groups: help: who wrote it down sample-size: type: integer + at-least: 1 surface: S-GOV tier: expert kernel-op: actor.figures help: > - for a measured figure, how many of the benchmark's cases it was taken - on. Fewer than `population:` makes it a subset figure, which is read - as one: a score on 200 of 14,042 questions is not the benchmark's - score, and a reader that compares it with a published full figure - says so. + on a benchmark figure, how many of the benchmark's cases it was + measured on. Fewer than `population:` makes it a subset figure: a + score on 200 of 14,042 questions is not the benchmark's score, and a + refusal that names the figure says it was taken on a subset. population: type: integer + at-least: 1 surface: S-GOV tier: expert kernel-op: actor.figures - help: how many cases the whole benchmark has, so `sample-size:` can be read against it + help: on a benchmark figure, how many cases the whole benchmark has, so `sample-size:` can be read against it interval: type: group:figure-interval surface: S-GOV tier: expert kernel-op: actor.figures help: > - for a measured figure, the confidence interval around `value:`, on - the figure's own scale. A runtime may hold an `accept-when:` line to - the whole interval rather than the point, so a figure that clears a - bar only within its own noise is not taken as clearing it. + on a benchmark figure, the confidence interval around `value:`, on + the benchmark's own scale. A line is held to the interval's far bound + as well as the value: `mmlu.score > 75` holds only when `value:` and + `low:` are both above 75, `<` and `<=` read `high:`, and `=` reads + `value:` alone. So a figure that clears a bar only within its own + noise is not taken as clearing it, in every runtime. figure-interval: describe: something the confidence interval around a measured figure can have @@ -2110,14 +2113,14 @@ groups: surface: S-GOV tier: expert kernel-op: actor.figures - help: the interval's lower bound, on the figure's scale + help: the interval's lower bound, on the benchmark's scale; a `>` or `>=` line holds only when this does high: type: number required: yes surface: S-GOV tier: expert kernel-op: actor.figures - help: the interval's upper bound, on the figure's scale + help: the interval's upper bound, on the benchmark's scale; a `<` or `<=` line holds only when this does level: type: number required: yes From a8bcd003c4f0e7f43277cbd8559fe5e0980582c9 Mon Sep 17 00:00:00 2001 From: Jithin VG Date: Tue, 17 Nov 2026 01:08:34 +0530 Subject: [PATCH 3/4] Count the new kind and the new tests where the documents state them (scripts/sync-counts.sh) --- README.md | 4 ++-- docs/90-REVIEW.md | 4 ++-- site-docs/index.md | 4 ++-- site-docs/status/verified.md | 6 +++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index d589ddb..9e22816 100644 --- a/README.md +++ b/README.md @@ -1001,11 +1001,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5310 tests (1719 Rust + 3591 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5329 tests (1720 Rust + 3609 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5310 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5329 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index 4b85429..0a7e736 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,8 +80,8 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1719 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | -| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3591 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | +| Rust | 5 crates, **1720 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3609 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | | Worked example | 42 files, 1128 lines, **132 distinct authored keys**, `pact check` → OK (498 settings) | diff --git a/site-docs/index.md b/site-docs/index.md index 21585a2..4bca284 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -47,7 +47,7 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,310** — 1719 Rust, 3591 adapter | +| Tests | **5,329** — 1720 Rust, 3609 adapter | | Schema kinds | **70** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | @@ -68,7 +68,7 @@ so plainly rather than leaving it to be discovered. [Gaps and blockers](status/gaps.md), in that order - **Comparing with Vercel Eve?** [Compared to Eve](status/eve.md) — a measured comparison, including where Eve is better -- **Building on it?** [The 69 kinds](reference/kinds.md) and +- **Building on it?** [The 70 kinds](reference/kinds.md) and [Adapters](reference/adapters.md) --- diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 6efe99a..62784d9 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,9 +12,9 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1719** | `cargo test --workspace` | -| Adapter tests | **3591** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,310** | both of the above | +| Rust tests | **1720** | `cargo test --workspace` | +| Adapter tests | **3609** | `cd adapters/python && uv run pytest tests/ -q` | +| Total | **5,329** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` | From a40ec9e5c048a4a4edbb8695ed5980b368b6adcf Mon Sep 17 00:00:00 2001 From: Jithin VG Date: Tue, 17 Nov 2026 04:25:58 +0530 Subject: [PATCH 4/4] Count the merged tests where the documents state them (scripts/sync-counts.sh) --- README.md | 4 ++-- docs/90-REVIEW.md | 2 +- site-docs/index.md | 2 +- site-docs/status/verified.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 9e22816..225e0cf 100644 --- a/README.md +++ b/README.md @@ -1001,11 +1001,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5329 tests (1720 Rust + 3609 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5331 tests (1722 Rust + 3609 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5329 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5331 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index 0a7e736..463b8e6 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,7 +80,7 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1720 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Rust | 5 crates, **1722 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | | Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3609 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | diff --git a/site-docs/index.md b/site-docs/index.md index 4bca284..5b8e95b 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -47,7 +47,7 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,329** — 1720 Rust, 3609 adapter | +| Tests | **5,331** — 1722 Rust, 3609 adapter | | Schema kinds | **70** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 62784d9..e1a97ef 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,9 +12,9 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1720** | `cargo test --workspace` | +| Rust tests | **1722** | `cargo test --workspace` | | Adapter tests | **3609** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,329** | both of the above | +| Total | **5,331** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` |