Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,35 @@
What changed in PACT, newest first. Each entry says what an author can now write, or what now
holds that did not; the commits are listed under it for whoever wants the detail.

## 2026-11 — a catalogue row says what a model needs of the machine that runs it (17 November)

A runtime choosing among models served on its own machine could not tell whether the machine could
hold one: a row said where a model was served and what it cost, never how big it was or how much
memory it took, so a local row the machine could not load was bound and failed on its first call.

### What an author can now write

- **`size:`** — a figure: how many parameters the model has, as a count (`7.6B`, `405B`, `500M`),
with its provenance, or `unknown`.
- **`requires: {memory, accelerator}`** — `memory:` is a figure: how much memory loading and running
the model as served takes (`4.7 GB`, `9 GiB`); `accelerator: gpu` says it runs only on an
accelerator, so `memory:` is the accelerator's memory.

### What holds now

- **A runtime that knows its machine leaves out a model served on that machine alone that the
machine cannot hold**, with a sentence: *"needs 9 GB of memory, and this machine has 8 GB"*, *"runs
only on an accelerator, and this machine has none"*. A row with no figure, a machine figure the
runtime does not know, and a row some runtime serves elsewhere are never left out for it. The
reference reads the machine's memory when it scores (`pact_adapters.machine.this_machine`); it does
not know an accelerator, so it decides nothing on one.
- **The checker refuses a figure no runtime could read**, where it is written:
`loader/a-parameter-count-that-does-not-read` and `loader/a-memory-figure-that-does-not-read` (a
bare `8` could be bytes or gigabytes).
- `spec/hardware-fit.yaml` holds the readings and the decisions; the Rust checker reads its value
sections and the Python resolver all of it. The shipped rows record no `size:` or `requires:` yet:
each figure needs a source somebody else can follow.

## 2026-11 — a metric a model is held to is taken on every case, and `prefer:` says what "best" means (16 November)

### What holds now
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1001,11 +1001,11 @@ framework is an extra.
|---|---|
| **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete |
| **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5336 tests (1722 Rust + 3614 adapter), clippy clean, TypeScript type-checked** |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5375 tests (1725 Rust + 3650 adapter), clippy clean, TypeScript type-checked** |
| **Adapters** | **All 7 named targets**, proven against one shared conformance suite |

```bash
./scripts/test-all.sh # 5336 tests, Rust + 7 adapters, fully offline
./scripts/test-all.sh # 5375 tests, Rust + 7 adapters, fully offline
```

**What is still owed** is not hidden — it lives in
Expand Down
104 changes: 104 additions & 0 deletions adapters/python/src/pact_adapters/machine.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
"""What a model needs of the machine that runs it, and whether this machine has it.

A catalogue row may say how many parameters a model has (`size:`) and what a
machine needs to run it where it is served on that machine (`requires:
{memory, accelerator}`). A runtime that knows its machine leaves out a model
served only here that the machine cannot hold, with a sentence saying why; a
runtime that does not know, or a row that records nothing, leaves nothing out.

`spec/hardware-fit.yaml` holds the readings and the decisions, and the
checker (`crates/pact-loader/src/hardware.rs`) refuses a figure this module
could not read, where it is written.
"""

from __future__ import annotations

import os
import re
from dataclasses import dataclass

#: A parameter count: a number and an optional `K`, `M`, `B` or `T`
#: (thousand, million, billion, trillion), either case — `7.6B`, `405b`, `500M`.
_COUNT = re.compile(r"^(\d+(?:\.\d+)?)\s*([kKmMbBtT]?)$")
_COUNT_SCALE = {"": 1, "k": 10**3, "m": 10**6, "b": 10**9, "t": 10**12}

#: An amount of memory: a number and a unit, with or without a space between —
#: decimal `KB`, `MB`, `GB`, `TB` or binary `KiB`, `MiB`, `GiB`, `TiB`. A bare
#: number is refused: bytes and gigabytes differ by a factor nobody should guess.
_BYTES = re.compile(r"^(\d+(?:\.\d+)?)\s*([KMGT]i?B)$")
_BYTES_SCALE = {
"KB": 10**3, "MB": 10**6, "GB": 10**9, "TB": 10**12,
"KiB": 2**10, "MiB": 2**20, "GiB": 2**30, "TiB": 2**40,
}


def parameters_of(written: object) -> "float | None":
"""A `size:` figure's value as a number of parameters, or `None` when it does not read.

`unknown` reads as `None` too; the checker tells the two apart.
"""
if isinstance(written, bool) or not isinstance(written, str | int | float):
return None
match = _COUNT.match(str(written).strip())
if match is None:
return None
count = float(match.group(1)) * _COUNT_SCALE[match.group(2).lower()]
return count if count > 0 else None


def bytes_of(written: object) -> "int | None":
"""A `requires.memory:` figure's value in bytes, or `None` when it does not read."""
if not isinstance(written, str):
return None
match = _BYTES.match(written.strip())
if match is None:
return None
amount = round(float(match.group(1)) * _BYTES_SCALE[match.group(2)])
return amount if amount > 0 else None


def shown(amount: int) -> str:
"""Bytes as a sentence prints them: gigabytes to one decimal, `.0` dropped — `8 GB`, `15.6 GB`."""
text = f"{amount / 10**9:.1f}"
return f"{text[:-2] if text.endswith('.0') else text} GB"


@dataclass(frozen=True)
class Machine:
"""What a runtime knows about the machine it runs on.

`None` means the runtime does not know, and nothing is judged on it.
`accelerator_memory_bytes` of `0` means the machine has no accelerator.
"""

memory_bytes: "int | None" = None
accelerator_memory_bytes: "int | None" = None


def this_machine() -> Machine:
"""The memory this process's machine reports. Its accelerator is not known here."""
try:
pages, size = os.sysconf("SC_PHYS_PAGES"), os.sysconf("SC_PAGE_SIZE")
except (AttributeError, ValueError, OSError): # a platform with no sysconf names
return Machine()
return Machine(memory_bytes=pages * size if pages > 0 and size > 0 else None)


def fits(memory_written: str, memory_bytes: "int | None", accelerator: str, machine: Machine) -> tuple[bool, str]:
"""Whether `machine` can run a model that needs `memory_bytes` (written `memory_written`).

`accelerator` is the row's `requires.accelerator:` (`gpu` or `none`). A
figure the row does not record, or a machine figure the runtime does not
know, is never a reason to leave the model out.
"""
if accelerator == "gpu":
have = machine.accelerator_memory_bytes
if have == 0:
return False, "runs only on an accelerator, and this machine has none"
if have is not None and memory_bytes is not None and memory_bytes > have:
return False, f"needs {memory_written} of accelerator memory, and this machine has {shown(have)}"
return True, ""
have = machine.memory_bytes
if have is not None and memory_bytes is not None and memory_bytes > have:
return False, f"needs {memory_written} of memory, and this machine has {shown(have)}"
return True, ""
55 changes: 54 additions & 1 deletion adapters/python/src/pact_adapters/resolve.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import yaml

from . import egress as _egress
from . import machine as _machine
from . import selection as _selection
from .diagnostics import Problem
from .evals import (
Expand Down Expand Up @@ -226,6 +227,27 @@ class ModelEntry:
languages: tuple[str, ...] = ()
sessions: frozenset[str] = frozenset()
dimensions: "int | None" = None
#: `size:` as a number of parameters, or `None` when the row does not say.
parameters: "float | None" = None
#: `requires.memory:` in bytes and as written, and `requires.accelerator:`
#: (`gpu` or `none`). `fits_machine` holds a row served only on this machine
#: to them; a row that does not say is never left out for it.
memory_bytes: "int | None" = None
memory_written: str = ""
accelerator: str = "none"
#: Some `served-by:` entry reaches it somewhere other than this machine, so
#: what this machine can hold does not decide whether it can be used.
served_elsewhere: bool = False

def fits_machine(self, machine: "_machine.Machine") -> tuple[bool, str]:
"""Whether `machine` can run this model where it is served, and why not.

Only a row served on this machine alone is judged: one some runtime
serves elsewhere is reached there whatever this machine holds.
"""
if not self.served_locally or self.served_elsewhere:
return True, ""
return _machine.fits(self.memory_written, self.memory_bytes, self.accelerator, machine)

def fits_voice(self, plan: Any, role: str) -> tuple[bool, str]:
"""Whether this model can play `role` in voice plan `plan` (`voice.VoicePlan`), and why not.
Expand All @@ -252,7 +274,7 @@ def fits_voice(self, plan: Any, role: str) -> tuple[bool, str]:
return False, "cannot listen while it speaks (its row has no `sessions: [duplex]`)"
return True, ""

def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]:
def satisfies(self, needs: dict[str, Any], machine: "_machine.Machine | None" = None) -> tuple[bool, str]:
"""Cheap pre-filter. Never a substitute for an eval verdict (FR-2.1.2).

`needs` is the flat shape `needs_of` produces from the author's own
Expand All @@ -264,6 +286,9 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]:
Two rules, and the difference between them is the whole of Y12:
a figure the catalogue RECORDS and that falls short **filters**; a figure
the catalogue says is `unknown` **binds and ranks last**.

`machine`, when the runtime knows it, also leaves out a row served on
this machine alone that the machine cannot hold (`fits_machine`).
"""
# 96 §A2. An agent's model reads and writes. A row that embeds,
# transcribes, speaks or reranks computes a different function, and the
Expand Down Expand Up @@ -326,6 +351,8 @@ def satisfies(self, needs: dict[str, Any]) -> tuple[bool, str]:
"is only reachable over the network, and this workspace does "
"not let the model call leave the box"
)
if machine is not None:
return self.fits_machine(machine)
return True, ""

def ignored_settings(self, settings: "Mapping[str, Any] | None") -> list[str]:
Expand Down Expand Up @@ -762,6 +789,7 @@ def _parse(text: str, shown: str) -> Catalogue:
languages=_words(caps.get("languages")),
sessions=frozenset(_words(caps.get("sessions"))),
dimensions=_dimensions(caps.get("dimensions")),
**_hardware(row),
)
entries.append(entry)
for key in (name, *entry.aliases):
Expand All @@ -777,6 +805,31 @@ def _parse(text: str, shown: str) -> Catalogue:
return Catalogue(tuple(entries), default, tuple(problems), by_id)


def _hardware(row: dict[str, Any]) -> dict[str, Any]:
"""`size:`, `requires:` and whether some `served-by:` entry is off this machine, as `ModelEntry` fields.

A figure that does not read is left unsaid here; the checker refuses it
where it is written (`loader/a-parameter-count-that-does-not-read`,
`loader/a-memory-figure-that-does-not-read`).
"""
size = row.get("size")
requires = row.get("requires") if isinstance(row.get("requires"), dict) else {}
memory = requires.get("memory") if isinstance(requires.get("memory"), dict) else {}
written = str(memory.get("value") or "").strip()
amount = _machine.bytes_of(written)
served = row.get("served-by") if isinstance(row.get("served-by"), list) else []
return {
"parameters": _machine.parameters_of(size.get("value")) if isinstance(size, dict) else None,
"memory_bytes": amount,
"memory_written": written if amount is not None else "",
"accelerator": "gpu" if str(requires.get("accelerator") or "").strip() == "gpu" else "none",
"served_elsewhere": any(
isinstance(entry, dict) and str(entry.get("endpoint") or "").strip() != "local"
for entry in served
),
}


def _words(written: Any) -> tuple[str, ...]:
"""A `list of text` line of a row, each word trimmed; anything else is not said."""
if not isinstance(written, list):
Expand Down
20 changes: 12 additions & 8 deletions adapters/python/src/pact_adapters/scoring.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import yaml

from . import egress as _egress
from . import machine as _machine
from .diagnostics import Problem
from .evals import (
Case,
Expand Down Expand Up @@ -1114,12 +1115,15 @@ def _bind(
catalogue = load_catalogue(workspace=root)
needs = needs_of(document, key)
written = _needs_file(root, key)
# A model served on this machine alone is held to what the machine has
# (`requires:`); this machine's memory is what the scoring run will use.
machine = _machine.this_machine()

wanted, because = asked, "you asked for it with `--model`"
if not wanted and spec.model:
wanted, because = spec.model, f"`model:` in agents/{key}/agent.yaml pins it"
if not wanted:
picked = _cheapest_that_qualifies(catalogue.entries, needs)
picked = _cheapest_that_qualifies(catalogue.entries, needs, machine=machine)
if picked is None:
return None, (
f"no model in `models/catalog.yaml` meets what {written} asks for, "
Expand All @@ -1132,7 +1136,7 @@ def _bind(
# picks". Where it does not qualify, picking it anyway would be picking a
# model the author's own `needs:` refuses.
fallback = catalogue.get(catalogue.default)
if fallback is not None and fallback.satisfies(needs)[0]:
if fallback is not None and fallback.satisfies(needs, machine)[0]:
picked = fallback
wanted = picked.name
because = (
Expand All @@ -1150,9 +1154,9 @@ def _bind(
+ ", ".join(sorted(e.name for e in catalogue.entries)),
)

ok, why = entry.satisfies(needs)
ok, why = entry.satisfies(needs, machine)
if not ok:
better = _cheapest_that_qualifies(catalogue.entries, needs, exclude=entry.name)
better = _cheapest_that_qualifies(catalogue.entries, needs, exclude=entry.name, machine=machine)
advice = (
f" `{better.name}` does meet it — score on that with "
f"`--model {better.name}`."
Expand All @@ -1170,7 +1174,7 @@ def _bind(
if not asked:
for fallback in spec.models[1:]:
row = catalogue.get(fallback)
meets, lacks = row.satisfies(needs) if row is not None else (False, "")
meets, lacks = row.satisfies(needs, machine) if row is not None else (False, "")
if not meets:
return None, (
f"`{fallback}`, a fallback in `model:` for {key}, "
Expand Down Expand Up @@ -1240,11 +1244,11 @@ def _needs_file(root: Path, key: str) -> str:


def _cheapest_that_qualifies(
entries, needs: dict[str, Any], exclude: str = ""
entries, needs: dict[str, Any], exclude: str = "", machine: "_machine.Machine | None" = None
) -> "ModelEntry | None":
"""The first model D11 would offer: cheapest that meets the author's needs."""
"""The first model D11 would offer: cheapest that meets the author's needs and fits `machine`."""
ranked = sorted(
(e for e in entries if e.name != exclude and e.satisfies(needs)[0]),
(e for e in entries if e.name != exclude and e.satisfies(needs, machine)[0]),
key=lambda e: e.ranks_after(needs),
)
return ranked[0] if ranked else None
Expand Down
Loading
Loading