diff --git a/CHANGELOG.md b/CHANGELOG.md index a522730..df05679 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,35 @@ What changed in PACT, newest first. Each entry says what an author can now write, or what now holds that did not; the commits are listed under it for whoever wants the detail. +## 2026-11 — a catalogue row says what a model needs of the machine that runs it (17 November) + +A runtime choosing among models served on its own machine could not tell whether the machine could +hold one: a row said where a model was served and what it cost, never how big it was or how much +memory it took, so a local row the machine could not load was bound and failed on its first call. + +### What an author can now write + +- **`size:`** — a figure: how many parameters the model has, as a count (`7.6B`, `405B`, `500M`), + with its provenance, or `unknown`. +- **`requires: {memory, accelerator}`** — `memory:` is a figure: how much memory loading and running + the model as served takes (`4.7 GB`, `9 GiB`); `accelerator: gpu` says it runs only on an + accelerator, so `memory:` is the accelerator's memory. + +### What holds now + +- **A runtime that knows its machine leaves out a model served on that machine alone that the + machine cannot hold**, with a sentence: *"needs 9 GB of memory, and this machine has 8 GB"*, *"runs + only on an accelerator, and this machine has none"*. A row with no figure, a machine figure the + runtime does not know, and a row some runtime serves elsewhere are never left out for it. The + reference reads the machine's memory when it scores (`pact_adapters.machine.this_machine`); it does + not know an accelerator, so it decides nothing on one. +- **The checker refuses a figure no runtime could read**, where it is written: + `loader/a-parameter-count-that-does-not-read` and `loader/a-memory-figure-that-does-not-read` (a + bare `8` could be bytes or gigabytes). +- `spec/hardware-fit.yaml` holds the readings and the decisions; the Rust checker reads its value + sections and the Python resolver all of it. The shipped rows record no `size:` or `requires:` yet: + each figure needs a source somebody else can follow. + ## 2026-11 — a metric a model is held to is taken on every case, and `prefer:` says what "best" means (16 November) ### What holds now diff --git a/README.md b/README.md index 4592b26..f362e78 100644 --- a/README.md +++ b/README.md @@ -1001,11 +1001,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5336 tests (1722 Rust + 3614 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5375 tests (1725 Rust + 3650 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5336 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5375 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/adapters/python/src/pact_adapters/machine.py b/adapters/python/src/pact_adapters/machine.py new file mode 100644 index 0000000..89248c4 --- /dev/null +++ b/adapters/python/src/pact_adapters/machine.py @@ -0,0 +1,104 @@ +"""What a model needs of the machine that runs it, and whether this machine has it. + +A catalogue row may say how many parameters a model has (`size:`) and what a +machine needs to run it where it is served on that machine (`requires: +{memory, accelerator}`). A runtime that knows its machine leaves out a model +served only here that the machine cannot hold, with a sentence saying why; a +runtime that does not know, or a row that records nothing, leaves nothing out. + +`spec/hardware-fit.yaml` holds the readings and the decisions, and the +checker (`crates/pact-loader/src/hardware.rs`) refuses a figure this module +could not read, where it is written. +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass + +#: A parameter count: a number and an optional `K`, `M`, `B` or `T` +#: (thousand, million, billion, trillion), either case — `7.6B`, `405b`, `500M`. +_COUNT = re.compile(r"^(\d+(?:\.\d+)?)\s*([kKmMbBtT]?)$") +_COUNT_SCALE = {"": 1, "k": 10**3, "m": 10**6, "b": 10**9, "t": 10**12} + +#: An amount of memory: a number and a unit, with or without a space between — +#: decimal `KB`, `MB`, `GB`, `TB` or binary `KiB`, `MiB`, `GiB`, `TiB`. A bare +#: number is refused: bytes and gigabytes differ by a factor nobody should guess. +_BYTES = re.compile(r"^(\d+(?:\.\d+)?)\s*([KMGT]i?B)$") +_BYTES_SCALE = { + "KB": 10**3, "MB": 10**6, "GB": 10**9, "TB": 10**12, + "KiB": 2**10, "MiB": 2**20, "GiB": 2**30, "TiB": 2**40, +} + + +def parameters_of(written: object) -> "float | None": + """A `size:` figure's value as a number of parameters, or `None` when it does not read. + + `unknown` reads as `None` too; the checker tells the two apart. + """ + if isinstance(written, bool) or not isinstance(written, str | int | float): + return None + match = _COUNT.match(str(written).strip()) + if match is None: + return None + count = float(match.group(1)) * _COUNT_SCALE[match.group(2).lower()] + return count if count > 0 else None + + +def bytes_of(written: object) -> "int | None": + """A `requires.memory:` figure's value in bytes, or `None` when it does not read.""" + if not isinstance(written, str): + return None + match = _BYTES.match(written.strip()) + if match is None: + return None + amount = round(float(match.group(1)) * _BYTES_SCALE[match.group(2)]) + return amount if amount > 0 else None + + +def shown(amount: int) -> str: + """Bytes as a sentence prints them: gigabytes to one decimal, `.0` dropped — `8 GB`, `15.6 GB`.""" + text = f"{amount / 10**9:.1f}" + return f"{text[:-2] if text.endswith('.0') else text} GB" + + +@dataclass(frozen=True) +class Machine: + """What a runtime knows about the machine it runs on. + + `None` means the runtime does not know, and nothing is judged on it. + `accelerator_memory_bytes` of `0` means the machine has no accelerator. + """ + + memory_bytes: "int | None" = None + accelerator_memory_bytes: "int | None" = None + + +def this_machine() -> Machine: + """The memory this process's machine reports. Its accelerator is not known here.""" + try: + pages, size = os.sysconf("SC_PHYS_PAGES"), os.sysconf("SC_PAGE_SIZE") + except (AttributeError, ValueError, OSError): # a platform with no sysconf names + return Machine() + return Machine(memory_bytes=pages * size if pages > 0 and size > 0 else None) + + +def fits(memory_written: str, memory_bytes: "int | None", accelerator: str, machine: Machine) -> tuple[bool, str]: + """Whether `machine` can run a model that needs `memory_bytes` (written `memory_written`). + + `accelerator` is the row's `requires.accelerator:` (`gpu` or `none`). A + figure the row does not record, or a machine figure the runtime does not + know, is never a reason to leave the model out. + """ + if accelerator == "gpu": + have = machine.accelerator_memory_bytes + if have == 0: + return False, "runs only on an accelerator, and this machine has none" + if have is not None and memory_bytes is not None and memory_bytes > have: + return False, f"needs {memory_written} of accelerator memory, and this machine has {shown(have)}" + return True, "" + have = machine.memory_bytes + if have is not None and memory_bytes is not None and memory_bytes > have: + return False, f"needs {memory_written} of memory, and this machine has {shown(have)}" + return True, "" diff --git a/adapters/python/src/pact_adapters/resolve.py b/adapters/python/src/pact_adapters/resolve.py index 2b08b12..1b0d8aa 100644 --- a/adapters/python/src/pact_adapters/resolve.py +++ b/adapters/python/src/pact_adapters/resolve.py @@ -46,6 +46,7 @@ import yaml from . import egress as _egress +from . import machine as _machine from . import selection as _selection from .diagnostics import Problem from .evals import ( @@ -226,6 +227,27 @@ class ModelEntry: languages: tuple[str, ...] = () sessions: frozenset[str] = frozenset() dimensions: "int | None" = None + #: `size:` as a number of parameters, or `None` when the row does not say. + parameters: "float | None" = None + #: `requires.memory:` in bytes and as written, and `requires.accelerator:` + #: (`gpu` or `none`). `fits_machine` holds a row served only on this machine + #: to them; a row that does not say is never left out for it. + memory_bytes: "int | None" = None + memory_written: str = "" + accelerator: str = "none" + #: Some `served-by:` entry reaches it somewhere other than this machine, so + #: what this machine can hold does not decide whether it can be used. + served_elsewhere: bool = False + + def fits_machine(self, machine: "_machine.Machine") -> tuple[bool, str]: + """Whether `machine` can run this model where it is served, and why not. + + Only a row served on this machine alone is judged: one some runtime + serves elsewhere is reached there whatever this machine holds. + """ + if not self.served_locally or self.served_elsewhere: + return True, "" + return _machine.fits(self.memory_written, self.memory_bytes, self.accelerator, machine) def fits_voice(self, plan: Any, role: str) -> tuple[bool, str]: """Whether this model can play `role` in voice plan `plan` (`voice.VoicePlan`), and why not. @@ -252,7 +274,7 @@ def fits_voice(self, plan: Any, role: str) -> tuple[bool, str]: return False, "cannot listen while it speaks (its row has no `sessions: [duplex]`)" return True, "" - def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]: + def satisfies(self, needs: dict[str, Any], machine: "_machine.Machine | None" = None) -> tuple[bool, str]: """Cheap pre-filter. Never a substitute for an eval verdict (FR-2.1.2). `needs` is the flat shape `needs_of` produces from the author's own @@ -264,6 +286,9 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]: Two rules, and the difference between them is the whole of Y12: a figure the catalogue RECORDS and that falls short **filters**; a figure the catalogue says is `unknown` **binds and ranks last**. + + `machine`, when the runtime knows it, also leaves out a row served on + this machine alone that the machine cannot hold (`fits_machine`). """ # 96 §A2. An agent's model reads and writes. A row that embeds, # transcribes, speaks or reranks computes a different function, and the @@ -326,6 +351,8 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]: "is only reachable over the network, and this workspace does " "not let the model call leave the box" ) + if machine is not None: + return self.fits_machine(machine) return True, "" def ignored_settings(self, settings: "Mapping[str, Any] | None") -> list[str]: @@ -762,6 +789,7 @@ def _parse(text: str, shown: str) -> Catalogue: languages=_words(caps.get("languages")), sessions=frozenset(_words(caps.get("sessions"))), dimensions=_dimensions(caps.get("dimensions")), + **_hardware(row), ) entries.append(entry) for key in (name, *entry.aliases): @@ -777,6 +805,31 @@ def _parse(text: str, shown: str) -> Catalogue: return Catalogue(tuple(entries), default, tuple(problems), by_id) +def _hardware(row: dict[str, Any]) -> dict[str, Any]: + """`size:`, `requires:` and whether some `served-by:` entry is off this machine, as `ModelEntry` fields. + + A figure that does not read is left unsaid here; the checker refuses it + where it is written (`loader/a-parameter-count-that-does-not-read`, + `loader/a-memory-figure-that-does-not-read`). + """ + size = row.get("size") + requires = row.get("requires") if isinstance(row.get("requires"), dict) else {} + memory = requires.get("memory") if isinstance(requires.get("memory"), dict) else {} + written = str(memory.get("value") or "").strip() + amount = _machine.bytes_of(written) + served = row.get("served-by") if isinstance(row.get("served-by"), list) else [] + return { + "parameters": _machine.parameters_of(size.get("value")) if isinstance(size, dict) else None, + "memory_bytes": amount, + "memory_written": written if amount is not None else "", + "accelerator": "gpu" if str(requires.get("accelerator") or "").strip() == "gpu" else "none", + "served_elsewhere": any( + isinstance(entry, dict) and str(entry.get("endpoint") or "").strip() != "local" + for entry in served + ), + } + + def _words(written: Any) -> tuple[str, ...]: """A `list of text` line of a row, each word trimmed; anything else is not said.""" if not isinstance(written, list): diff --git a/adapters/python/src/pact_adapters/scoring.py b/adapters/python/src/pact_adapters/scoring.py index 2104572..3e13be4 100644 --- a/adapters/python/src/pact_adapters/scoring.py +++ b/adapters/python/src/pact_adapters/scoring.py @@ -49,6 +49,7 @@ import yaml from . import egress as _egress +from . import machine as _machine from .diagnostics import Problem from .evals import ( Case, @@ -1114,12 +1115,15 @@ def _bind( catalogue = load_catalogue(workspace=root) needs = needs_of(document, key) written = _needs_file(root, key) + # A model served on this machine alone is held to what the machine has + # (`requires:`); this machine's memory is what the scoring run will use. + machine = _machine.this_machine() wanted, because = asked, "you asked for it with `--model`" if not wanted and spec.model: wanted, because = spec.model, f"`model:` in agents/{key}/agent.yaml pins it" if not wanted: - picked = _cheapest_that_qualifies(catalogue.entries, needs) + picked = _cheapest_that_qualifies(catalogue.entries, needs, machine=machine) if picked is None: return None, ( f"no model in `models/catalog.yaml` meets what {written} asks for, " @@ -1132,7 +1136,7 @@ def _bind( # picks". Where it does not qualify, picking it anyway would be picking a # model the author's own `needs:` refuses. fallback = catalogue.get(catalogue.default) - if fallback is not None and fallback.satisfies(needs)[0]: + if fallback is not None and fallback.satisfies(needs, machine)[0]: picked = fallback wanted = picked.name because = ( @@ -1150,9 +1154,9 @@ def _bind( + ", ".join(sorted(e.name for e in catalogue.entries)), ) - ok, why = entry.satisfies(needs) + ok, why = entry.satisfies(needs, machine) if not ok: - better = _cheapest_that_qualifies(catalogue.entries, needs, exclude=entry.name) + better = _cheapest_that_qualifies(catalogue.entries, needs, exclude=entry.name, machine=machine) advice = ( f" `{better.name}` does meet it — score on that with " f"`--model {better.name}`." @@ -1170,7 +1174,7 @@ def _bind( if not asked: for fallback in spec.models[1:]: row = catalogue.get(fallback) - meets, lacks = row.satisfies(needs) if row is not None else (False, "") + meets, lacks = row.satisfies(needs, machine) if row is not None else (False, "") if not meets: return None, ( f"`{fallback}`, a fallback in `model:` for {key}, " @@ -1240,11 +1244,11 @@ def _needs_file(root: Path, key: str) -> str: def _cheapest_that_qualifies( - entries, needs: dict[str, Any], exclude: str = "" + entries, needs: dict[str, Any], exclude: str = "", machine: "_machine.Machine | None" = None ) -> "ModelEntry | None": - """The first model D11 would offer: cheapest that meets the author's needs.""" + """The first model D11 would offer: cheapest that meets the author's needs and fits `machine`.""" ranked = sorted( - (e for e in entries if e.name != exclude and e.satisfies(needs)[0]), + (e for e in entries if e.name != exclude and e.satisfies(needs, machine)[0]), key=lambda e: e.ranks_after(needs), ) return ranked[0] if ranked else None diff --git a/adapters/python/tests/test_a_local_model_must_fit_the_machine.py b/adapters/python/tests/test_a_local_model_must_fit_the_machine.py new file mode 100644 index 0000000..84f3ce4 --- /dev/null +++ b/adapters/python/tests/test_a_local_model_must_fit_the_machine.py @@ -0,0 +1,108 @@ +"""A catalogue row's `size:` and `requires:` mean one thing in every runtime. + +`spec/hardware-fit.yaml` is the table; the Rust checker reads its value +sections and this file reads all of it, so a figure the checker accepts is a +figure the resolver reads, and a machine that cannot hold a model served on it +alone leaves that model out with the same sentence everywhere. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import yaml + +from pact_adapters import machine as m +from pact_adapters.resolve import ModelEntry, _hardware, _served_locally, load_catalogue, needs_of + +TABLE = yaml.safe_load( + (Path(__file__).resolve().parents[3] / "spec" / "hardware-fit.yaml").read_text(encoding="utf-8") +) + + +def _machine(written: dict) -> m.Machine: + accelerator = written.get("accelerator-memory") + return m.Machine( + memory_bytes=m.bytes_of(written["memory"]) if "memory" in written else None, + accelerator_memory_bytes=0 if accelerator == "none" else (m.bytes_of(accelerator) if accelerator else None), + ) + + +@pytest.mark.parametrize("row", TABLE["sizes"], ids=lambda r: r["written"]) +def test_a_parameter_count_reads_as_the_table_says(row): + want = None if row["reads"] == "unknown" else float(row["reads"]) + assert m.parameters_of(row["written"]) == want + + +@pytest.mark.parametrize("row", TABLE["not-sizes"], ids=lambda r: r["written"]) +def test_a_parameter_count_the_table_refuses_does_not_read(row): + assert m.parameters_of(row["written"]) is None + + +@pytest.mark.parametrize("row", TABLE["memories"], ids=lambda r: r["written"]) +def test_a_memory_figure_reads_as_the_table_says(row): + want = None if row["reads"] == "unknown" else int(row["reads"]) + assert m.bytes_of(row["written"]) == want + + +@pytest.mark.parametrize("row", TABLE["not-memories"], ids=lambda r: r["written"]) +def test_a_memory_figure_the_table_refuses_does_not_read(row): + assert m.bytes_of(row["written"]) is None + + +def _entry(row: dict) -> ModelEntry: + memory = str(row["memory"]) + served = [{"runtime": "ollama", "endpoint": "local"}] + if row.get("served-elsewhere") is True: + served.append({"runtime": "vllm", "endpoint": "https://models.example"}) + written = {"requires": {"memory": {"value": memory}}, "served-by": served} + if "accelerator" in row: + written["requires"]["accelerator"] = row["accelerator"] + return ModelEntry(name="local-model", tier="small", cost=0.0, served_locally=_served_locally(served), **_hardware(written)) + + +@pytest.mark.parametrize("row", TABLE["fits"], ids=lambda r: f"{r['memory']}-{r['because']}") +def test_a_machine_holds_a_model_served_on_it_as_the_table_says(row): + fits, why = _entry(row).fits_machine(_machine(row["machine"])) + assert fits is row["fits"], row["because"] + if "says" in row: + assert why == row["says"] + if fits: + assert why == "" + + +def test_satisfies_leaves_out_a_local_model_the_machine_cannot_hold(tmp_path): + (tmp_path / "models").mkdir() + (tmp_path / "models" / "catalog.yaml").write_text( + yaml.safe_dump( + { + "version": 1, + "models": { + "big-local": { + "tier": "mid", + "served-by": [{"runtime": "ollama", "endpoint": "local"}], + "size": {"value": "32B", "provenance": {"source": "the model card"}}, + "requires": {"memory": {"value": "20 GB", "provenance": {"source": "measured with the runtime"}}}, + "cost": {"input-per-mtok": "0 USD", "output-per-mtok": "0 USD"}, + } + }, + } + ), + encoding="utf-8", + ) + row = load_catalogue(workspace=tmp_path).get("big-local") + assert row is not None and row.parameters == 32e9 and row.memory_bytes == 20 * 10**9 + needs = needs_of({"agents": {"a": {"instructions": "x"}}}, "a") + assert row.satisfies(needs) == (True, "") + assert row.satisfies(needs, m.Machine(memory_bytes=8 * 10**9)) == ( + False, + "needs 20 GB of memory, and this machine has 8 GB", + ) + assert row.satisfies(needs, m.Machine(memory_bytes=64 * 10**9)) == (True, "") + + +def test_this_machine_reports_its_memory_and_not_its_accelerator(): + here = m.this_machine() + assert here.accelerator_memory_bytes is None + assert here.memory_bytes is None or here.memory_bytes > 0 diff --git a/crates/pact-cli/src/main.rs b/crates/pact-cli/src/main.rs index 6392269..0c960f9 100644 --- a/crates/pact-cli/src/main.rs +++ b/crates/pact-cli/src/main.rs @@ -2075,6 +2075,9 @@ fn validate_leaving_out( // 96 §B4: what an agent accepts a model on — a benchmark, a metric of // its suite, a price — read the way the resolver will decide it. pact_loader::accept::check(root, BUILTIN_CATALOGUE, &mut diags); + // What a model needs of the machine that runs it, read the way a runtime + // holding a machine to it will read it (`spec/hardware-fit.yaml`). + pact_loader::hardware::check(root, &mut diags); // 96 §B5: a `settings:` line the pinned model would throw away. pact_loader::settings::check(root, BUILTIN_CATALOGUE, &mut diags); diags.sort(); diff --git a/crates/pact-loader/src/hardware.rs b/crates/pact-loader/src/hardware.rs new file mode 100644 index 0000000..09132ab --- /dev/null +++ b/crates/pact-loader/src/hardware.rs @@ -0,0 +1,234 @@ +//! **What a model needs of the machine that runs it**: a catalogue row's +//! `size:` (how many parameters) and `requires: {memory, accelerator}`. +//! +//! A runtime that knows its machine leaves out a model served on that machine +//! alone when the machine cannot hold it (`pact_adapters.machine.fits`). So a +//! figure written here has to read the same way in every runtime, and one that +//! does not read is refused where it is written rather than skipped where it is +//! used — a skipped memory figure would bind a model the machine cannot load. +//! `spec/hardware-fit.yaml` holds the readings; this reads its value sections. +//! +//! | Rule | Fires when | +//! |---|---| +//! | `loader/a-parameter-count-that-does-not-read` | a workspace row's `size.value` is not a count such as `7.6B`, `500M` or `8000000000`, nor `unknown` | +//! | `loader/a-memory-figure-that-does-not-read` | a workspace row's `requires.memory.value` is not an amount such as `4.7 GB` or `9 GiB`, nor `unknown` | + +use pact_diag::{Diagnostic, Diagnostics}; +use pact_doc::Node; + +/// `size.value` as a number of parameters: a number and an optional `K`, `M`, +/// `B` or `T` (either case), above zero. `None` when it does not read. +pub fn parameters(text: &str) -> Option { + let text = text.trim(); + let (number, scale) = match text.chars().last()? { + 'k' | 'K' => (&text[..text.len() - 1], 1e3), + 'm' | 'M' => (&text[..text.len() - 1], 1e6), + 'b' | 'B' => (&text[..text.len() - 1], 1e9), + 't' | 'T' => (&text[..text.len() - 1], 1e12), + _ => (text, 1.0), + }; + let count = decimal(number.trim_end())? * scale; + (count > 0.0).then_some(count) +} + +/// `requires.memory.value` in bytes: a number, an optional space, and `KB`, +/// `MB`, `GB`, `TB` or `KiB`, `MiB`, `GiB`, `TiB`, above zero. A bare number +/// is refused — bytes and gigabytes differ by a factor nobody should guess. +pub fn bytes(text: &str) -> Option { + let text = text.trim(); + let (number, unit) = text.split_at(text.find(|c: char| c.is_ascii_alphabetic())?); + let scale: f64 = match unit { + "KB" => 1e3, + "MB" => 1e6, + "GB" => 1e9, + "TB" => 1e12, + "KiB" => 1024f64, + "MiB" => 1024f64.powi(2), + "GiB" => 1024f64.powi(3), + "TiB" => 1024f64.powi(4), + _ => return None, + }; + let amount = (decimal(number.trim_end())? * scale).round(); + (amount > 0.0).then_some(amount as u64) +} + +/// Digits with an optional fraction (`7`, `7.6`); no sign, exponent or spaces +/// inside — the same `\d+(\.\d+)?` the Python reader matches. +fn decimal(text: &str) -> Option { + let (whole, fraction) = text.split_once('.').unwrap_or((text, "0")); + let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()); + (digits(whole) && digits(fraction)) + .then(|| text.parse().ok()) + .flatten() +} + +/// Refuse a workspace row's `size:` or `requires.memory:` figure that does not +/// read. The distribution's rows are held to the same reading by a test. +pub fn check(document: &Node, diags: &mut Diagnostics) { + let Some(rows) = document + .get("models") + .and_then(|c| c.get("models")) + .and_then(Node::as_map) + else { + return; + }; + for (model, row) in rows { + figures(model, &row.node, diags); + } +} + +fn figures(model: &str, row: &Node, diags: &mut Diagnostics) { + if let Some(value) = row.get("size").and_then(|f| f.get("value")) { + let text = value + .as_str() + .map(str::to_owned) + .unwrap_or_else(|| value.to_string()); + if text.trim() != "unknown" && parameters(&text).is_none() { + diags.push(Diagnostic::error( + "loader/a-parameter-count-that-does-not-read", + value.span.clone(), + format!( + "'{model}' gives its size as `{}`, which is not a count of parameters.", + text.trim() + ), + "Write it like `7.6B`, `405B`, `500M` or `8000000000`, or `unknown`.".to_string(), + )); + } + } + let memory = row + .get("requires") + .and_then(|r| r.get("memory")) + .and_then(|f| f.get("value")); + if let Some(value) = memory { + let text = value + .as_str() + .map(str::to_owned) + .unwrap_or_else(|| value.to_string()); + if text.trim() != "unknown" && bytes(&text).is_none() { + diags.push(Diagnostic::error( + "loader/a-memory-figure-that-does-not-read", + value.span.clone(), + format!( + "'{model}' says it needs `{}` of memory, which is not an amount of memory a \ + runtime can hold a machine to.", + text.trim() + ), + "Write it with its unit, like `4.7 GB`, `9 GiB` or `800 MB`, or `unknown`." + .to_string(), + )); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const VECTORS: &str = include_str!("../../../spec/hardware-fit.yaml"); + const SHIPPED: &str = include_str!("../../../models/catalog.yaml"); + + fn section(name: &str) -> Vec { + let table = + pact_doc::parse_yaml(VECTORS, camino::Utf8Path::new("spec/hardware-fit.yaml")).unwrap(); + table.get(name).and_then(Node::as_list).unwrap().to_vec() + } + + fn text(row: &Node, key: &str) -> String { + let value = row.get(key).unwrap(); + value + .as_str() + .map(str::to_owned) + .unwrap_or_else(|| value.to_string()) + } + + #[test] + fn every_size_and_memory_in_the_shared_table_reads_as_it_says() { + for row in section("sizes") { + let (written, reads) = (text(&row, "written"), text(&row, "reads")); + let want = (reads != "unknown").then(|| reads.parse::().unwrap()); + let got = if written == "unknown" { + None + } else { + parameters(&written) + }; + assert_eq!(got, want, "size {written}"); + } + for row in section("not-sizes") { + assert_eq!( + parameters(&text(&row, "written")), + None, + "{}", + text(&row, "because") + ); + } + for row in section("memories") { + let (written, reads) = (text(&row, "written"), text(&row, "reads")); + let want = (reads != "unknown").then(|| reads.parse::().unwrap()); + let got = if written == "unknown" { + None + } else { + bytes(&written) + }; + assert_eq!(got, want, "memory {written}"); + } + for row in section("not-memories") { + assert_eq!( + bytes(&text(&row, "written")), + None, + "{}", + text(&row, "because") + ); + } + } + + fn checked(catalogue: &str) -> Diagnostics { + let wrapped = format!( + "models:\n{}", + catalogue + .lines() + .map(|l| format!(" {l}\n")) + .collect::() + ); + let doc = pact_doc::parse_yaml(&wrapped, camino::Utf8Path::new("workspace.yaml")).unwrap(); + let mut diags = Diagnostics::default(); + check(&doc, &mut diags); + diags + } + + #[test] + fn a_figure_that_does_not_read_is_refused_where_it_is_written() { + let diags = checked( + "models:\n m:\n size: {value: 7.6 GB}\n requires:\n memory: {value: \"8\"}\n", + ); + let rules: Vec<_> = diags.items().iter().map(|d| d.rule).collect(); + assert_eq!( + rules, + [ + "loader/a-parameter-count-that-does-not-read", + "loader/a-memory-figure-that-does-not-read" + ] + ); + let fine = checked( + "models:\n m:\n size: {value: 7.6B}\n requires:\n memory: {value: 4.7 GB}\n accelerator: gpu\n n:\n size: {value: unknown}\n requires:\n memory: {value: unknown}\n", + ); + assert!(fine.is_empty()); + } + + #[test] + fn every_figure_the_distribution_ships_reads() { + let doc = pact_doc::parse_yaml( + &format!( + "models:\n{}", + SHIPPED + .lines() + .map(|l| format!(" {l}\n")) + .collect::() + ), + camino::Utf8Path::new("w.yaml"), + ) + .unwrap(); + let mut diags = Diagnostics::default(); + check(&doc, &mut diags); + assert!(diags.is_empty(), "{:?}", diags.items()); + } +} diff --git a/crates/pact-loader/src/lib.rs b/crates/pact-loader/src/lib.rs index 05de6a2..365fa39 100644 --- a/crates/pact-loader/src/lib.rs +++ b/crates/pact-loader/src/lib.rs @@ -154,6 +154,7 @@ pub mod failures; pub mod fill; pub mod firstfile; pub mod handover; +pub mod hardware; pub mod holes; pub mod interfaces; pub mod jinja; diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index e9d4e0b..2cd49c9 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,8 +80,8 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1722 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | -| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3614 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | +| Rust | 5 crates, **1725 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3650 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | | Worked example | 42 files, 1128 lines, **132 distinct authored keys**, `pact check` → OK (498 settings) | diff --git a/docs/97-PRIMITIVES.md b/docs/97-PRIMITIVES.md index bc5b981..47a7a2e 100644 --- a/docs/97-PRIMITIVES.md +++ b/docs/97-PRIMITIVES.md @@ -354,8 +354,10 @@ one thing only: how they decide. - **May it park?** Whether it can wait on a person or an event. A `run-code` snippet may only call Actors that never wait. - **Figures.** Cost, latency, context size, capabilities and benchmark scores. - Each carries where the number came from (`model-cost`, `model-speed`, `figure`, - `provenance`). + Each carries where the number came from (`model-cost`, `model-speed`, + `model-requires`, `figure`, `provenance`). What a model needs of the machine + that runs it (`size`, `requires`) leaves out a model served on that machine + alone when the machine cannot hold it. - **Identity.** Who acted, recorded in the run record. Added in review: - **on whose behalf** it acted; - **how strongly** its identity was established, which `needs-signature` @@ -1741,6 +1743,8 @@ words for a person, with no behaviour. | `model.kind` | Actor | | `model.benchmarks` | Store + Actor | | `model.speed` | Store + Actor | +| `model.size` | Store | +| `model.requires` | Store + Actor | | `served-by.runtime` | Actor | | `served-by.endpoint` | Actor + Allowance | | `served-by.region` | Allowance + Store + Actor | @@ -1757,6 +1761,8 @@ words for a person, with no behaviour. | `model-can.dimensions` | Shape + Actor | | `model-can.context-window` | Store + Actor | | `model-speed.first-words-within` | Store | +| `model-requires.memory` | Store + Actor | +| `model-requires.accelerator` | Actor | | `figure.value` | Store + Actor | | `figure.provenance` | Store + Actor | | `provenance.source` | Store | @@ -2486,7 +2492,7 @@ Generated by `python3 scripts/pattern-coverage.py --ops` from `spec/kernel.yaml` | `actor.effect` | Actor | none, reads, writes, spends, irreversible; with undo and status-check; readable by a Check | computed from reads-only, spends-money, undone-by; Check-readable as `effect:` in `when-this`; shown by `pact discover` | `action.reads-only`, `action.spends-money`, `action.undone-by`, `action.status-check`, `when-this.effect`, `workflow.undone-by`, `stage.undone-by` | | `actor.write-set` | Actor | the Store paths an Actor declares it writes | `writes: [, ...]` on an action; overlapping branches are warned | `action.writes` | | `actor.park` | Actor | may wait on a person or an event | ask-someone, answer-within | `question.answer-within`, `port.answer-within`, `stage.does` | -| `actor.figures` | Actor | cost, latency, context, capabilities, benchmark scores, with provenance | model-cost, model-speed, figure, provenance | `model.reasoning`, `model.cost`, `model.benchmarks`, `model.speed`, `model-can.context-window`, `figure.value`, `figure.provenance` | +| `actor.figures` | Actor | cost, latency, context, capabilities, benchmark scores, with provenance | model-cost, model-speed, model-requires, figure, provenance | `model.reasoning`, `model.cost`, `model.benchmarks`, `model.speed`, `model.size`, `model.requires`, `model-requires.memory`, `model-can.context-window`, `figure.value`, `figure.provenance` | | `actor.identity` | Actor | who acted, on whose behalf, how strongly established | needs-signature, owned-by, not-the-same-as | `question.not-the-same-as`, `question.needs-signature`, `ownership.owned-by` | | `actor.location` | Actor | where it runs | served-by.region, runs-in | `served-by.endpoint`, `served-by.region`, `stage.runs-in` | | `actor.by-requirement` | Actor | chosen from a catalogue by a Check and a preference | needs, prefer, interface, implements, chosen-by | `agent.needs`, `needs.prefer`, `needs.accept-when`, `needs.scores`, `knowledge.interface`, `knowledge.chosen-by`, `knowledge.implements`, `skill.interface`, `skill.chosen-by`, `skill.implements`, `tool.interface`, `tool.chosen-by`, `tool.implements` | diff --git a/site-docs/index.md b/site-docs/index.md index d1dbc53..59a64a3 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -33,7 +33,7 @@ requires it, and deleting it leaves a working agent. - **[What do you write?](concepts/what.md)** Files. One rule — *a directory is a field; a field may be a directory* — - and 70 kinds you never have to memorise. + and 71 kinds you never have to memorise. @@ -47,8 +47,8 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,336** — 1722 Rust, 3614 adapter | -| Schema kinds | **70** | +| Tests | **5,375** — 1725 Rust, 3650 adapter | +| Schema kinds | **71** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | | Worked example | 43 files, 498 settings, 1 payload script | @@ -68,7 +68,7 @@ so plainly rather than leaving it to be discovered. [Gaps and blockers](status/gaps.md), in that order - **Comparing with Vercel Eve?** [Compared to Eve](status/eve.md) — a measured comparison, including where Eve is better -- **Building on it?** [The 70 kinds](reference/kinds.md) and +- **Building on it?** [The 71 kinds](reference/kinds.md) and [Adapters](reference/adapters.md) --- diff --git a/site-docs/reference/kinds.md b/site-docs/reference/kinds.md index f3e0805..8bf92a5 100644 --- a/site-docs/reference/kinds.md +++ b/site-docs/reference/kinds.md @@ -1,11 +1,11 @@ -# The 70 kinds +# The 71 kinds Generated from `spec/schema.yaml`. Every kind, what it is for, and how many fields it has. You do not need to learn these — `pact check` names the right one when you need it. -**70 kinds.** A kind is a *shape a document can take*, not a file you must +**71 kinds.** A kind is a *shape a document can take*, not a file you must write: most workspaces use a dozen. @@ -72,9 +72,10 @@ write: most workspaces use a dozen. | `catalog` | 4 | — | | `figure` | 2 | — | | `limits` | 24 | `feel`, `finishes-within`, `starts-from`, `paused-during`, `milestones` | -| `model` | 11 | — | +| `model` | 13 | — | | `model-can` | 12 | — | | `model-cost` | 2 | — | +| `model-requires` | 2 | — | | `model-speed` | 1 | — | | `needs` | 12 | `reasoning`, `tool-calling`, `images`, `audio`, `computer-use` | | `provenance` | 9 | — | diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 7414411..c3d3d6e 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,13 +12,13 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1722** | `cargo test --workspace` | -| Adapter tests | **3614** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,336** | both of the above | +| Rust tests | **1725** | `cargo test --workspace` | +| Adapter tests | **3650** | `cd adapters/python && uv run pytest tests/ -q` | +| Total | **5,375** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` | -| Schema kinds | **70** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | +| Schema kinds | **71** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | | Fields that resolve a name | **78** | `grep -cE '^\s*names:' spec/schema.yaml` | ## What is proven, and what "proven" means diff --git a/spec/hardware-fit.yaml b/spec/hardware-fit.yaml new file mode 100644 index 0000000..1b79452 --- /dev/null +++ b/spec/hardware-fit.yaml @@ -0,0 +1,76 @@ +# What a model needs of the machine that runs it (`size:`, `requires:` on a +# catalogue row), held to one table. The checker +# (`crates/pact-loader/src/hardware.rs`) reads `sizes:`, `not-sizes:`, +# `memories:` and `not-memories:`; the Python resolver +# (`pact_adapters.machine`) reads every section. A runtime that binds a model +# for a machine it knows decides each `fits:` row as written, and prints `says:` +# word for word. + +# `size.value:` — a parameter count, a number with an optional K, M, B or T. +sizes: + - {written: "7.6B", reads: 7600000000} + - {written: "405b", reads: 405000000000} + - {written: "500M", reads: 500000000} + - {written: "1.24 B", reads: 1240000000} + - {written: "8000000000", reads: 8000000000} + - {written: "1.8T", reads: 1800000000000} + - {written: "unknown", reads: unknown} + +not-sizes: + - {written: "7.6 GB", because: a size counts parameters, not bytes} + - {written: "big", because: not a count} + - {written: "0B", because: a model has at least one parameter} + - {written: "-7B", because: a count is not negative} + - {written: "7.6 billion", because: "the unit is one letter: `7.6B`"} + +# `requires.memory.value:` — an amount of memory with its unit; decimal GB and +# binary GiB are both read, and a bare number is refused. +memories: + - {written: "4.7 GB", reads: 4700000000} + - {written: "9 GiB", reads: 9663676416} + - {written: "800 MB", reads: 800000000} + - {written: "16GB", reads: 16000000000} + - {written: "1 TB", reads: 1000000000000} + - {written: "512 MiB", reads: 536870912} + - {written: "unknown", reads: unknown} + +not-memories: + - {written: "8", because: a bare number could be bytes or gigabytes} + - {written: "8 G", because: "the unit is `GB` or `GiB`"} + - {written: "8 gb", because: "units are written `GB`, `GiB`, as a datasheet writes them"} + - {written: "0 GB", because: running a model takes some memory} + - {written: "7.6B", because: a memory figure is an amount of memory, not a parameter count} + +# A row served on this machine alone, against what the runtime knows of the +# machine (`memory:`, and `accelerator-memory:`, where `none` means it has no +# accelerator). A machine figure that is left out is not known, and nothing is +# decided on it. +fits: + - {memory: "4.7 GB", machine: {memory: "16 GB"}, fits: yes, because: the machine holds it} + - memory: "9 GB" + machine: {memory: "8 GB"} + fits: no + says: "needs 9 GB of memory, and this machine has 8 GB" + because: the model needs more memory than the machine has + - {memory: "9 GB", machine: {}, fits: yes, because: a runtime that does not know the machine leaves nothing out} + - {memory: unknown, machine: {memory: "8 GB"}, fits: yes, because: a row that records no figure is not left out} + - memory: "6 GB" + accelerator: gpu + machine: {memory: "64 GB", accelerator-memory: none} + fits: no + says: "runs only on an accelerator, and this machine has none" + because: the machine's own memory does not run an accelerator-only model + - {memory: "6 GB", accelerator: gpu, machine: {memory: "64 GB", accelerator-memory: "8 GB"}, fits: yes, because: the accelerator holds it} + - memory: "12 GB" + accelerator: gpu + machine: {memory: "64 GB", accelerator-memory: "8 GB"} + fits: no + says: "needs 12 GB of accelerator memory, and this machine has 8 GB" + because: the accelerator is too small + - {memory: "12 GB", accelerator: gpu, machine: {memory: "64 GB"}, fits: yes, because: "the runtime does not know the accelerator, so it does not decide on it"} + - memory: "16 GiB" + machine: {memory: "15.6 GB"} + fits: no + says: "needs 16 GiB of memory, and this machine has 15.6 GB" + because: GiB are binary, so 16 GiB is more than 15.6 GB + - {memory: "900 GB", served-elsewhere: yes, machine: {memory: "8 GB"}, fits: yes, because: a runtime reaches it off this machine} diff --git a/spec/schema.yaml b/spec/schema.yaml index 6474dd4..5bc3f09 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -1848,6 +1848,24 @@ groups: how quickly it answers, and where those figures came from. Read when an agent says `prefer: fastest`, and when a `feel: voice` agent is matched against how long the first words may take. + size: + type: group:figure + surface: S-CAP + tier: expert + kernel-op: actor.figures + help: > + how many parameters it has, as a count — `7.6B`, `405B`, `500M` — with + where that figure came from. Write `unknown` if nobody has published it. + requires: + type: group:model-requires + surface: S-CAP + tier: expert + kernel-op: actor.figures + help: > + what a machine needs to run it where it is served on that machine + (`endpoint: local`). A runtime that knows the machine leaves out a + local model the machine cannot hold, and says so; a model with no + figure here is not left out. served-by: describe: something a way of serving a model can have @@ -2020,6 +2038,30 @@ groups: how long before the first words (or the first sound) come back, as a length of time — `400ms` — with where that figure came from + model-requires: + describe: something a model's needs of the machine that runs it can have + fields: + memory: + type: group:figure + surface: S-CAP + tier: expert + kernel-op: actor.figures + help: > + how much memory it takes to load the model as it is served and answer + — `4.7 GB`, `9 GiB`, `800 MB` — with where that figure came from. + With `accelerator: gpu` this is the accelerator's memory; otherwise it + is the machine's own. + accelerator: + type: one-of + choices: [none, gpu] + surface: S-CAP + tier: expert + help: > + `gpu` when the model runs only on an accelerator, so `memory:` is + the accelerator's memory and a machine without one cannot run it. + Leave it out, or write `none`, when it runs from the machine's own + memory, whether or not an accelerator speeds it up. + figure: describe: something a recorded figure can have fields: diff --git a/tests/golden/schema-fields.txt b/tests/golden/schema-fields.txt index 09fed24..ef5740b 100644 --- a/tests/golden/schema-fields.txt +++ b/tests/golden/schema-fields.txt @@ -208,6 +208,8 @@ model-can.tool-calling one-of | S-CAP | expert | optional model-can.voices list of text | S-CAP | expert | optional model-cost.input-per-mtok text | S-GOV | expert | optional model-cost.output-per-mtok text | S-GOV | expert | optional +model-requires.accelerator one-of | S-CAP | expert | optional +model-requires.memory group:figure | S-CAP | expert | optional model-speed.first-words-within group:figure | S-CAP | expert | optional model.also-known-as list of text | S-CAP | expert | optional model.benchmarks map of group:figure | S-CAP | expert | optional @@ -217,7 +219,9 @@ model.description text | S-GEN | expert | optional model.family text | S-GEN | expert | optional model.kind one-of | S-CAP | expert | optional model.reasoning group:figure | S-CAP | expert | optional +model.requires group:model-requires | S-CAP | expert | optional model.served-by list of group:served-by | S-CAP | expert | optional +model.size group:figure | S-CAP | expert | optional model.speed group:model-speed | S-CAP | expert | optional model.tier one-of | S-CAP | expert | optional moment.after duration | S-CTRL | core | optional