diff --git a/CHANGELOG.md b/CHANGELOG.md index df05679..d915d23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,28 @@ memory it took, so a local row the machine could not load was bound and failed o - **The `deepeval` extra installs `datasets` and `pandas`.** DeepEval's benchmarks import them when they are built and its package declares neither, so a benchmark run failed on its first line. +## 2026-11 — a copied catalogue row says which row it copied, and check says when that row moves on (17 November) + +Recording one benchmark figure for a shipped model copies the distribution's whole row into the +workspace's `models/catalog.yaml` first, because a workspace row replaces the shipped one whole. The +copy then kept the price, the window and the serving it was copied with, and when the distribution +corrected that row the workspace went on binding against the old facts with nothing said. + +### What an author can now write + +- **`copied-from: {digest}`** on a row of the workspace's own catalogue: `sha256:` and the hex + digest of the distribution's row as it was copied (its JSON as `pact show` prints it, keys sorted, + no spaces). `benchmarks.record` writes it whenever it copies a row. + +### What holds now + +- **`pact check` warns `loader/a-copied-row-behind-the-distribution`** when the distribution's row + of that name (or alias) no longer has that digest, naming the fields where the copy and the + shipped row now differ and giving the digest to write once they are reconciled; and when the + distribution no longer has the row at all. +- `spec/copied-rows.yaml` holds the Rust checker and the Python writer to the same digests, for a row + with whole numbers, decimals, a share, a date and non-ASCII text. + ## 2026-11 — a measured figure says how it was measured, and a line is held to its interval (16 November) A figure's `provenance:` said where a number came from but not how much of the benchmark it rests on diff --git a/README.md b/README.md index f362e78..3b1cc8b 100644 --- a/README.md +++ b/README.md @@ -1001,11 +1001,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5375 tests (1725 Rust + 3650 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5383 tests (1728 Rust + 3655 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5375 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5383 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/adapters/python/src/pact_adapters/benchmarks.py b/adapters/python/src/pact_adapters/benchmarks.py index 6897a5e..aae8b84 100644 --- a/adapters/python/src/pact_adapters/benchmarks.py +++ b/adapters/python/src/pact_adapters/benchmarks.py @@ -23,6 +23,10 @@ import datetime as _dt import enum +import hashlib +import json +import subprocess +import tempfile import typing from collections.abc import Callable, Mapping from dataclasses import dataclass @@ -31,6 +35,7 @@ import yaml +from .loader import pact_binary from .providers import AskModel from .selection import benchmarks_of @@ -162,8 +167,14 @@ def record( # A workspace row REPLACES the distribution's row of the same name — the catalogue is # layered row by row (`resolve._layer`) — so a row holding only this figure would lose # where the model is served, its price and its window. The distribution's row is copied - # first, and the reviewer reads all of it in the diff. + # first, and the reviewer reads all of it in the diff. `copied-from:` + # says which version of that row it was, so `pact check` can say when + # the distribution's row moves on without the copy + # (`loader/a-copied-row-behind-the-distribution`). rows[model] = _shipped_row(model) + shown = _shipped_as_shown(model) + if rows[model] and shown is not None: + rows[model]["copied-from"] = {"digest": row_digest(shown)} row = rows[model] row.setdefault("benchmarks", {})[key] = figure path.parent.mkdir(parents=True, exist_ok=True) @@ -200,6 +211,44 @@ def _shipped_row(model: str) -> dict[str, Any]: return {} +def row_digest(row: Mapping[str, Any]) -> str: + """The digest `copied-from.digest` records: `sha256:` and the hex SHA-256 of + `row`'s JSON with every mapping's keys sorted and no spaces. + + `row` is the JSON `pact show` printed for the row, so its numbers are the + ones the loader read; digesting a YAML reading of its own would let a date + or a `1e3` read differently here than in the checker (`spec/copied-rows.yaml`). + """ + text = json.dumps(row, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _shipped_as_shown(model: str) -> "dict[str, Any] | None": + """The distribution's row for `model` as `pact show` prints it, or `None` with no loader here. + + `pact show` loads a workspace, so the distribution's catalogue is shown + from a scratch workspace that holds nothing else. + """ + from .resolve import BUILTIN_CATALOGUE + + binary = pact_binary() + if binary is None: + return None + with tempfile.TemporaryDirectory() as scratch: + root = Path(scratch) + (root / "models").mkdir() + (root / "models" / "catalog.yaml").write_text(Path(BUILTIN_CATALOGUE).read_text(encoding="utf-8"), encoding="utf-8") + (root / "workspace.yaml").write_text("name: shipped-catalogue\n", encoding="utf-8") + shown = subprocess.run([str(binary), "show", str(root)], capture_output=True, text=True) + if shown.returncode != 0: + return None + rows = ((json.loads(shown.stdout) or {}).get("models") or {}).get("models") or {} + for name, row in rows.items(): + if name == model or model in (row or {}).get("also-known-as", []): + return row + return None + + def run_deepeval( benchmark: Benchmark, model: str, ask: Callable[[str, str], str] ) -> float: # pragma: no cover - downloads a dataset and asks a live model diff --git a/adapters/python/tests/test_a_copied_row_says_which_row_it_copied.py b/adapters/python/tests/test_a_copied_row_says_which_row_it_copied.py new file mode 100644 index 0000000..9fbce11 --- /dev/null +++ b/adapters/python/tests/test_a_copied_row_says_which_row_it_copied.py @@ -0,0 +1,73 @@ +"""A copied catalogue row records which version of the distribution's row it copied. + +`benchmarks.record` copies the distribution's row before adding a figure, and +writes `copied-from.digest` beside it. The digest is taken over the JSON +`pact show` prints, so it is the one the checker computes +(`crates/pact-loader/src/copies.rs`): `spec/copied-rows.yaml` holds both +runtimes to the same digests, and `pact check` stays quiet on a fresh copy. +""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +import pytest +import yaml + +from pact_adapters import benchmarks +from pact_adapters.loader import pact_binary + +REPO = Path(__file__).resolve().parents[3] +TABLE = yaml.safe_load((REPO / "spec" / "copied-rows.yaml").read_text(encoding="utf-8")) +BINARY = pact_binary() +needs_binary = pytest.mark.skipif(BINARY is None, reason="no `pact` binary is built here") + + +def _shown_row(tmp_path: Path, row: dict) -> dict: + (tmp_path / "models").mkdir() + (tmp_path / "workspace.yaml").write_text("name: probe\n", encoding="utf-8") + (tmp_path / "models" / "catalog.yaml").write_text( + yaml.safe_dump({"version": 1, "models": {"probe": row}}, allow_unicode=True, sort_keys=False), + encoding="utf-8", + ) + shown = subprocess.run([str(BINARY), "show", str(tmp_path)], capture_output=True, text=True, check=True) + return json.loads(shown.stdout)["models"]["models"]["probe"] + + +@needs_binary +@pytest.mark.parametrize("case", TABLE["digests"], ids=lambda c: c["because"]) +def test_a_row_digests_as_the_shared_table_says(case, tmp_path): + assert benchmarks.row_digest(_shown_row(tmp_path, case["row"])) == case["digest"] + + +@needs_binary +def test_recording_a_figure_writes_which_row_was_copied_and_check_is_quiet(tmp_path): + path = benchmarks.record(tmp_path, "qwen2.5-7b-instruct", "mmlu", 70.0, source="a run on this machine") + row = yaml.safe_load(path.read_text(encoding="utf-8"))["models"]["qwen2.5-7b-instruct"] + shipped = benchmarks._shipped_as_shown("qwen2.5-7b-instruct") + assert shipped is not None + assert row["copied-from"] == {"digest": benchmarks.row_digest(shipped)} + (tmp_path / "workspace.yaml").write_text("name: copied\n", encoding="utf-8") + checked = subprocess.run([str(BINARY), "check", str(tmp_path)], capture_output=True, text=True) + assert "a-copied-row-behind-the-distribution" not in checked.stdout + checked.stderr + + +@needs_binary +def test_a_copy_whose_digest_is_not_the_shipped_rows_is_reported(tmp_path): + path = benchmarks.record(tmp_path, "qwen2.5-7b-instruct", "mmlu", 70.0, source="a run on this machine") + data = yaml.safe_load(path.read_text(encoding="utf-8")) + data["models"]["qwen2.5-7b-instruct"]["copied-from"] = {"digest": "sha256:" + "0" * 64} + path.write_text(yaml.safe_dump(data, sort_keys=False, allow_unicode=True), encoding="utf-8") + (tmp_path / "workspace.yaml").write_text("name: copied\n", encoding="utf-8") + checked = subprocess.run([str(BINARY), "check", str(tmp_path)], capture_output=True, text=True) + assert "loader/a-copied-row-behind-the-distribution" in checked.stdout + checked.stderr + assert "`benchmarks`" in checked.stdout + checked.stderr + + +def test_a_second_figure_does_not_copy_the_row_again(tmp_path): + first = benchmarks.record(tmp_path, "qwen2.5-7b-instruct", "mmlu", 70.0, source="a run on this machine") + before = yaml.safe_load(first.read_text(encoding="utf-8"))["models"]["qwen2.5-7b-instruct"].get("copied-from") + again = benchmarks.record(tmp_path, "qwen2.5-7b-instruct", "gsm8k", 80.0, source="a run on this machine") + assert yaml.safe_load(again.read_text(encoding="utf-8"))["models"]["qwen2.5-7b-instruct"].get("copied-from") == before diff --git a/crates/pact-cli/src/main.rs b/crates/pact-cli/src/main.rs index 0c960f9..f227ed2 100644 --- a/crates/pact-cli/src/main.rs +++ b/crates/pact-cli/src/main.rs @@ -2078,6 +2078,9 @@ fn validate_leaving_out( // What a model needs of the machine that runs it, read the way a runtime // holding a machine to it will read it (`spec/hardware-fit.yaml`). pact_loader::hardware::check(root, &mut diags); + // A row of the workspace's own catalogue copied from the distribution's, + // which the distribution has corrected since. + pact_loader::copies::check(root, BUILTIN_CATALOGUE, &mut diags); // 96 §B5: a `settings:` line the pinned model would throw away. pact_loader::settings::check(root, BUILTIN_CATALOGUE, &mut diags); diags.sort(); diff --git a/crates/pact-loader/src/copies.rs b/crates/pact-loader/src/copies.rs new file mode 100644 index 0000000..17a25ba --- /dev/null +++ b/crates/pact-loader/src/copies.rs @@ -0,0 +1,231 @@ +//! **A copied catalogue row that has fallen behind** the distribution's. +//! +//! A workspace row replaces the distribution's row of the same name whole, so +//! a figure is never two sources mixed. Recording one benchmark figure for a +//! shipped model therefore copies the shipped row first (`benchmarks.record`), +//! and the copy keeps the price, the window and the serving it was copied with. +//! When the distribution later corrects that row, the workspace goes on binding +//! against the old facts with nothing said. `copied-from.digest` records which +//! version of the row was copied; this compares it with the row the +//! distribution ships now and warns, naming the fields that now differ. +//! +//! The digest is `sha256:` and the hex SHA-256 of the row's JSON as `pact show` +//! prints it, with every mapping's keys sorted and no spaces. The Python writer +//! digests the JSON `pact show` gave it, so neither runtime reads YAML numbers +//! for itself; `spec/copied-rows.yaml` holds both to the same digests. +//! +//! | Rule | Fires when | +//! |---|---| +//! | `loader/a-copied-row-behind-the-distribution` (warning) | a workspace row's `copied-from.digest` is not the digest of the distribution's row of that name as shipped now, or the distribution no longer has that row | + +use pact_diag::{Diagnostic, Diagnostics}; +use pact_doc::Node; +use serde_json::{Map, Value}; +use sha2::{Digest, Sha256}; + +/// The digest `copied-from.digest` records for a distribution row. +pub fn digest(row: &Node) -> String { + let text = serde_json::to_string(&sorted(row.to_json())).unwrap_or_default(); + format!("sha256:{:x}", Sha256::digest(text.as_bytes())) +} + +/// `value` with every object's keys in sorted order. `serde_json` keeps +/// insertion order here (`preserve_order`), so the order is rebuilt by hand. +fn sorted(value: Value) -> Value { + match value { + Value::Object(map) => { + let mut entries: Vec<(String, Value)> = map.into_iter().collect(); + entries.sort_by(|a, b| a.0.cmp(&b.0)); + Value::Object( + entries + .into_iter() + .map(|(k, v)| (k, sorted(v))) + .collect::>(), + ) + } + Value::Array(items) => Value::Array(items.into_iter().map(sorted).collect()), + other => other, + } +} + +/// The distribution's row a workspace row of this name copied: the row of that +/// key, or the one that lists it under `also-known-as:`. +fn shipped_row<'a>(shipped: &'a Node, name: &str) -> Option<&'a Node> { + let rows = shipped.get("models").and_then(Node::as_map)?; + rows.iter().find_map(|(key, row)| { + let aliases = row + .node + .get("also-known-as") + .and_then(Node::as_list) + .unwrap_or(&[]); + (key == name || aliases.iter().any(|a| a.as_str() == Some(name))).then_some(&row.node) + }) +} + +/// Warn on every workspace row whose `copied-from.digest` no longer matches the +/// distribution's row it was copied from. +pub fn check(document: &Node, builtin: &str, diags: &mut Diagnostics) { + let Some(rows) = document + .get("models") + .and_then(|c| c.get("models")) + .and_then(Node::as_map) + else { + return; + }; + let Ok(shipped) = pact_doc::parse_yaml(builtin, camino::Utf8Path::new("models/catalog.yaml")) + else { + return; + }; + for (model, row) in rows { + let Some(copied) = row.node.get("copied-from").and_then(|c| c.get("digest")) else { + continue; + }; + let recorded = copied.as_str().unwrap_or("").trim(); + let Some(now) = shipped_row(&shipped, model) else { + diags.push(Diagnostic::warning( + "loader/a-copied-row-behind-the-distribution", + copied.span.clone(), + format!( + "'{model}' was copied from the distribution's row of that name, and the \ + distribution no longer has one, so nothing here keeps its figures current." + ), + format!( + "Check the row's figures against their sources, then take `copied-from:` out \ + of '{model}' so it reads as a row of your own." + ), + )); + continue; + }; + if digest(now) == recorded { + continue; + } + let differ = differing(&row.node, now); + diags.push(Diagnostic::warning( + "loader/a-copied-row-behind-the-distribution", + copied.span.clone(), + format!( + "'{model}' is a copy of the distribution's row, and that row has changed since it \ + was copied: {} now differ{}.", + if differ.is_empty() { + "no field of the copy".to_string() + } else { + differ.join(", ") + }, + if differ.len() == 1 { "s" } else { "" } + ), + format!( + "Compare each with the distribution's row, keep what you changed on purpose, take \ + the rest from it, and write `copied-from: {{digest: {}}}`.", + digest(now) + ), + )); + } +} + +/// The top-level fields where the workspace row and the distribution's row +/// now disagree, sorted; `copied-from` is the copy's own. +fn differing(copy: &Node, shipped: &Node) -> Vec { + let (copy, shipped) = (copy.to_json(), shipped.to_json()); + let empty = Map::new(); + let (copy, shipped) = ( + copy.as_object().unwrap_or(&empty), + shipped.as_object().unwrap_or(&empty), + ); + let mut keys: Vec<&String> = copy + .keys() + .chain(shipped.keys()) + .filter(|k| *k != "copied-from") + .collect(); + keys.sort(); + keys.dedup(); + keys.into_iter() + .filter(|k| { + copy.get(*k).map(|v| sorted(v.clone())) != shipped.get(*k).map(|v| sorted(v.clone())) + }) + .map(|k| format!("`{k}`")) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + const VECTORS: &str = include_str!("../../../spec/copied-rows.yaml"); + + fn table() -> Node { + pact_doc::parse_yaml(VECTORS, camino::Utf8Path::new("spec/copied-rows.yaml")).unwrap() + } + + #[test] + fn every_row_in_the_shared_table_digests_as_it_says() { + for case in table().get("digests").and_then(Node::as_list).unwrap() { + let want = case.get("digest").and_then(Node::as_str).unwrap(); + assert_eq!( + digest(case.get("row").unwrap()), + want, + "{}", + case.get("because").unwrap() + ); + } + } + + fn checked(workspace: &str, shipped: &str) -> Diagnostics { + let doc = pact_doc::parse_yaml(workspace, camino::Utf8Path::new("workspace.yaml")).unwrap(); + let mut diags = Diagnostics::new(); + check(&doc, shipped, &mut diags); + diags + } + + #[test] + fn a_copy_of_the_current_row_is_quiet_and_a_stale_one_names_what_differs() { + let shipped = "models:\n m:\n tier: small\n also-known-as: [m-alias]\n cost: {input-per-mtok: 0.10 USD, output-per-mtok: 0.40 USD}\n"; + let row = pact_doc::parse_yaml(shipped, camino::Utf8Path::new("c.yaml")).unwrap(); + let current = digest(row.get("models").and_then(|m| m.get("m")).unwrap()); + let copy = |digest: &str, cost: &str| { + format!( + "models:\n models:\n m:\n tier: small\n also-known-as: [m-alias]\n cost: {cost}\n benchmarks: {{mmlu: {{value: \"80\"}}}}\n copied-from: {{digest: \"{digest}\"}}\n" + ) + }; + let quiet = checked( + ©( + ¤t, + "{input-per-mtok: 0.10 USD, output-per-mtok: 0.40 USD}", + ), + shipped, + ); + assert!(quiet.is_empty(), "{:?}", quiet.items()); + + let moved = "models:\n m:\n tier: small\n also-known-as: [m-alias]\n cost: {input-per-mtok: 0.08 USD, output-per-mtok: 0.40 USD}\n"; + let stale = checked( + ©( + ¤t, + "{input-per-mtok: 0.10 USD, output-per-mtok: 0.40 USD}", + ), + moved, + ); + let [warning] = stale.items() else { + panic!("{:?}", stale.items()) + }; + assert_eq!(warning.rule, "loader/a-copied-row-behind-the-distribution"); + assert!( + warning.message.contains("`benchmarks`, `cost` now differ"), + "{}", + warning.message + ); + + let by_alias = format!( + "models:\n models:\n m-alias:\n tier: small\n also-known-as: [m-alias]\n cost: {{input-per-mtok: 0.10 USD, output-per-mtok: 0.40 USD}}\n copied-from: {{digest: \"{current}\"}}\n" + ); + assert!(checked(&by_alias, shipped).is_empty()); + + let gone = checked(©(¤t, "{}"), "models:\n other: {tier: small}\n"); + assert_eq!(gone.items().len(), 1); + assert!(gone.items()[0].message.contains("no longer has one")); + } + + #[test] + fn a_row_without_copied_from_is_not_judged() { + let ws = "models:\n models:\n m: {tier: frontier}\n"; + assert!(checked(ws, "models:\n m: {tier: small}\n").is_empty()); + } +} diff --git a/crates/pact-loader/src/lib.rs b/crates/pact-loader/src/lib.rs index 365fa39..5c2a77d 100644 --- a/crates/pact-loader/src/lib.rs +++ b/crates/pact-loader/src/lib.rs @@ -144,6 +144,7 @@ pub mod clauses; pub mod closed_world; pub mod code; pub mod conditions; +pub mod copies; pub mod currency; pub mod deciding; pub mod derive; diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index 2cd49c9..6a73358 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,8 +80,8 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1725 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | -| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3650 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | +| Rust | 5 crates, **1728 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3655 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | | Worked example | 42 files, 1128 lines, **132 distinct authored keys**, `pact check` → OK (498 settings) | diff --git a/docs/97-PRIMITIVES.md b/docs/97-PRIMITIVES.md index 47a7a2e..a330830 100644 --- a/docs/97-PRIMITIVES.md +++ b/docs/97-PRIMITIVES.md @@ -1745,6 +1745,7 @@ words for a person, with no behaviour. | `model.speed` | Store + Actor | | `model.size` | Store | | `model.requires` | Store + Actor | +| `model.copied-from` | Store + Check | | `served-by.runtime` | Actor | | `served-by.endpoint` | Actor + Allowance | | `served-by.region` | Allowance + Store + Actor | @@ -1763,6 +1764,7 @@ words for a person, with no behaviour. | `model-speed.first-words-within` | Store | | `model-requires.memory` | Store + Actor | | `model-requires.accelerator` | Actor | +| `copied-from.digest` | Store + Check | | `figure.value` | Store + Actor | | `figure.provenance` | Store + Actor | | `provenance.source` | Store | diff --git a/site-docs/index.md b/site-docs/index.md index 59a64a3..cc70ca9 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -33,7 +33,7 @@ requires it, and deleting it leaves a working agent. - **[What do you write?](concepts/what.md)** Files. One rule — *a directory is a field; a field may be a directory* — - and 71 kinds you never have to memorise. + and 72 kinds you never have to memorise. @@ -47,8 +47,8 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,375** — 1725 Rust, 3650 adapter | -| Schema kinds | **71** | +| Tests | **5,383** — 1728 Rust, 3655 adapter | +| Schema kinds | **72** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | | Worked example | 43 files, 498 settings, 1 payload script | @@ -68,7 +68,7 @@ so plainly rather than leaving it to be discovered. [Gaps and blockers](status/gaps.md), in that order - **Comparing with Vercel Eve?** [Compared to Eve](status/eve.md) — a measured comparison, including where Eve is better -- **Building on it?** [The 71 kinds](reference/kinds.md) and +- **Building on it?** [The 72 kinds](reference/kinds.md) and [Adapters](reference/adapters.md) --- diff --git a/site-docs/reference/kinds.md b/site-docs/reference/kinds.md index 8bf92a5..afa6255 100644 --- a/site-docs/reference/kinds.md +++ b/site-docs/reference/kinds.md @@ -1,11 +1,11 @@ -# The 71 kinds +# The 72 kinds Generated from `spec/schema.yaml`. Every kind, what it is for, and how many fields it has. You do not need to learn these — `pact check` names the right one when you need it. -**71 kinds.** A kind is a *shape a document can take*, not a file you must +**72 kinds.** A kind is a *shape a document can take*, not a file you must write: most workspaces use a dozen. @@ -70,9 +70,10 @@ write: most workspaces use a dozen. | `call-order` | 2 | `call`, `first` | | `benchmark` | 5 | — | | `catalog` | 4 | — | +| `copied-from` | 1 | — | | `figure` | 2 | — | | `limits` | 24 | `feel`, `finishes-within`, `starts-from`, `paused-during`, `milestones` | -| `model` | 13 | — | +| `model` | 14 | — | | `model-can` | 12 | — | | `model-cost` | 2 | — | | `model-requires` | 2 | — | diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index c3d3d6e..1c34d06 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,13 +12,13 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1725** | `cargo test --workspace` | -| Adapter tests | **3650** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,375** | both of the above | +| Rust tests | **1728** | `cargo test --workspace` | +| Adapter tests | **3655** | `cd adapters/python && uv run pytest tests/ -q` | +| Total | **5,383** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` | -| Schema kinds | **71** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | +| Schema kinds | **72** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | | Fields that resolve a name | **78** | `grep -cE '^\s*names:' spec/schema.yaml` | ## What is proven, and what "proven" means diff --git a/spec/copied-rows.yaml b/spec/copied-rows.yaml new file mode 100644 index 0000000..158e9c2 --- /dev/null +++ b/spec/copied-rows.yaml @@ -0,0 +1,34 @@ +# The digest `copied-from.digest` records for a distribution row, held to one +# value in every runtime. It is `sha256:` and the hex SHA-256 of the row's JSON +# as `pact show` prints it, with every mapping's keys sorted and no spaces. +# The Rust checker (`crates/pact-loader/src/copies.rs`) digests each `row:` as +# loaded; the Python writer (`pact_adapters.benchmarks.row_digest`) digests the +# JSON `pact show` prints for the same row. Both must give `digest:`. +digests: + - because: whole numbers, decimals and a share stay as the loader read them + row: + tier: small + description: "A model written with the words an author uses — café, naïve, 日本語" + capabilities: + dimensions: 768 + context-window: + value: 32k + provenance: {source: the model card, as-of: 2026-07-28} + benchmarks: + mmlu: + value: "78" + provenance: + source: measured here + sample-size: 200 + population: 14042 + interval: {low: 71.6, high: 83.1, level: 0.95} + cost: {input-per-mtok: 0.10 USD, output-per-mtok: 0.40 USD} + digest: "sha256:f0774b8ee0a8ce8d2fb2802eef45f95c05fe4cf1603e30abc930d9e989a75ff7" + - because: key order in the file does not move the digest, and list order does + row: + served-by: + - {runtime: ollama, endpoint: local} + - {endpoint: local, runtime: vllm} + also-known-as: ["b-model:7b", b-model] + tier: mid + digest: "sha256:bdf87889208feeb0553891b9cf6cc1499dd82e53d3831e801c9f29f754f1a5b7" diff --git a/spec/schema.yaml b/spec/schema.yaml index 5bc3f09..a33abbe 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -1866,6 +1866,18 @@ groups: (`endpoint: local`). A runtime that knows the machine leaves out a local model the machine cannot hold, and says so; a model with no figure here is not left out. + copied-from: + type: group:copied-from + surface: S-GOV + tier: expert + help: > + on a row of your own `models/catalog.yaml` that began as a copy of the + distribution's row of the same name: which version of that row it + copied. A workspace row replaces the distribution's whole, so a copy + keeps the price, the window and the serving it was copied with; `pact + check` warns when the distribution's row has changed since, naming + the fields that now differ. `benchmarks.record` writes it when it + copies a row. served-by: describe: something a way of serving a model can have @@ -2061,6 +2073,18 @@ groups: the accelerator's memory and a machine without one cannot run it. Leave it out, or write `none`, when it runs from the machine's own memory, whether or not an accelerator speeds it up. + copied-from: + describe: something a record of where a copied catalogue row came from can have + fields: + digest: + type: text + required: yes + surface: S-GOV + tier: expert + help: > + `sha256:` and the hex digest of the distribution's row as it was when + it was copied: its JSON as `pact show` prints it, with every mapping's + keys sorted and no spaces. figure: describe: something a recorded figure can have diff --git a/tests/golden/schema-fields.txt b/tests/golden/schema-fields.txt index ef5740b..d5362b8 100644 --- a/tests/golden/schema-fields.txt +++ b/tests/golden/schema-fields.txt @@ -84,6 +84,7 @@ context-policy.if-it-still-does-not-fit one-of | S-GOV | core | optional context-policy.summarised-by text | S-CAP | expert | optional context-policy.then list of group:tidy-step | S-CTRL | expert | optional context-policy.when-full percent | S-CTRL | core | optional +copied-from.digest text | S-GOV | expert | required credential-reference.by-reference text | S-CAP | core | required cycle-limits.evals integer | S-GOV | core | optional cycle-limits.per-cycle integer | S-GOV | core | optional @@ -214,6 +215,7 @@ model-speed.first-words-within group:figure | S-CAP | expert | optional model.also-known-as list of text | S-CAP | expert | optional model.benchmarks map of group:figure | S-CAP | expert | optional model.capabilities group:model-can | S-CAP | expert | optional +model.copied-from group:copied-from | S-GOV | expert | optional model.cost group:model-cost | S-GOV | expert | optional model.description text | S-GEN | expert | optional model.family text | S-GEN | expert | optional