Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,24 @@
What changed in PACT, newest first. Each entry says what an author can now write, or what now
holds that did not; the commits are listed under it for whoever wants the detail.

## 2026-11 — a benchmark named in any script, a figure recorded under an alias, and DeepEval's ARC challenge set (16 November)

### What holds now

- **An `accept-when:` name is everything before the comparison, in both ports.** The Python reader
took only `[A-Za-z0-9 _.()+-]`, so `τ-bench.score > 50` — an alias the catalogue publishes for
tau-bench — was not a comparison there while the checker read it. `spec/accept-when.yaml` gains the
row, and `> 50` (no name at all) as a refused one.
- **`benchmarks.record` writes a figure under the row's own id.** Recorded under an alias
(`qwen2.5:7b-instruct`), it made a second workspace row beside the distribution's, and the
catalogue, layered by id, offered both as candidates. It now finds the row the name answers to, in
the workspace or the distribution, and writes there.
- **The shipped `arc` row runs the challenge set.** DeepEval's ARC runs the easy set unless told
otherwise, so a figure measured for the row (`ARC-Challenge`) was the easy set's. The row now says
`with: {mode: challenge}`, read into DeepEval's `ARCMode`.
- **The `deepeval` extra installs `datasets` and `pandas`.** DeepEval's benchmarks import them when
they are built and its package declares neither, so a benchmark run failed on its first line.

## 2026-11 — a measured figure says how it was measured, and a line is held to its interval (16 November)

A figure's `provenance:` said where a number came from but not how much of the benchmark it rests on
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1001,11 +1001,11 @@ framework is an extra.
|---|---|
| **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete |
| **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5331 tests (1722 Rust + 3609 adapter), clippy clean, TypeScript type-checked** |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5335 tests (1722 Rust + 3613 adapter), clippy clean, TypeScript type-checked** |
| **Adapters** | **All 7 named targets**, proven against one shared conformance suite |

```bash
./scripts/test-all.sh # 5331 tests, Rust + 7 adapters, fully offline
./scripts/test-all.sh # 5335 tests, Rust + 7 adapters, fully offline
```

**What is still owed** is not hidden — it lives in
Expand Down
4 changes: 3 additions & 1 deletion adapters/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,9 @@ langchain = ["langchain-core>=1.5.1"]
# else's dependency graph is one re-lock from breaking.
openai-agents = ["openai-agents>=0.18.3", "openai>=2.16.0"]
# The eval metrics `providers.py` reaches for, imported lazily when asked.
deepeval = ["deepeval>=4.1.3"]
# DeepEval's benchmarks import `datasets` (every one) and `pandas` (MMLU) when
# they are built, and its own package declares neither.
deepeval = ["deepeval>=4.1.3", "datasets>=2.19", "pandas>=2.0"]
all = ["pact-adapters[anthropic,autogen,langgraph,langchain,openai-agents,deepeval]"]
# `pydantic-handlebars` is what `Agent.from_spec` compiles a `{{name}}` template with (the
# `spec` extra of `pydantic-ai-slim`): the parity suite loads exported specs whose holes are
Expand Down
22 changes: 22 additions & 0 deletions adapters/python/src/pact_adapters/benchmarks.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,10 @@ def record(
data = data if isinstance(data, dict) else {}
data.setdefault("version", 1)
rows = data.setdefault("models", {})
# Under the row's own id, whichever of its ids the caller wrote: the catalogue
# is layered by row id (`resolve._layer`), so a row written under an alias
# would be a second candidate beside the distribution's, not the same model.
model = _row_id(model, rows)
if model not in rows:
# A workspace row REPLACES the distribution's row of the same name — the catalogue is
# layered row by row (`resolve._layer`) — so a row holding only this figure would lose
Expand All @@ -167,6 +171,24 @@ def record(
return path


def _row_id(model: str, rows: Mapping[str, Any]) -> str:
"""The id of the row `model` names, in the workspace's `rows` or the distribution's.

A row answers to its id and to every `also-known-as:` entry. A name no row
answers to is a new row's id, as written.
"""
from .resolve import BUILTIN_CATALOGUE

shipped = (yaml.safe_load(Path(BUILTIN_CATALOGUE).read_text(encoding="utf-8")) or {}).get("models") or {}
for layer in (rows, shipped):
if model in layer:
return model
for name, row in layer.items():
if isinstance(row, Mapping) and model in (row.get("also-known-as") or []):
return str(name)
return model


def _shipped_row(model: str) -> dict[str, Any]:
"""The distribution's own row for `model` (under its id or any alias), or a new empty one."""
from .resolve import BUILTIN_CATALOGUE
Expand Down
6 changes: 4 additions & 2 deletions adapters/python/src/pact_adapters/selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,9 @@
AFTER: dict[str, str] = {"pass-rate": "fraction", "cost": "money"}

_OPS = (">=", "<=", "==", ">", "<", "=")
_LINE = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9 _.()+-]*?)\s*(>=|<=|==|>|<|=)\s*(.+?)\s*$")
#: A name is everything before the first `>`, `<` or `=`, as `accept.rs` reads
#: it, so a benchmark published as `τ-bench` is a name in both ports.
_LINE = re.compile(r"^\s*([^<>=]*?)\s*(>=|<=|==|>|<|=)\s*(.+?)\s*$")


class Unreadable(ValueError):
Expand Down Expand Up @@ -327,7 +329,7 @@ def parse(
) -> Line:
"""One line, or `Unreadable` saying why it is not one."""
found = _LINE.match(str(written))
if found is None:
if found is None or not found.group(1).strip():
raise Unreadable(
f"`{written}` is not a comparison: write a name, then `>`, `>=`, `<`, `<=` or `=`, "
f"then a figure — `mmlu.score > 50`"
Expand Down
24 changes: 24 additions & 0 deletions adapters/python/tests/test_an_accept_line_means_one_thing.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,30 @@ def test_the_shipped_benchmarks_say_how_each_is_measured() -> None:
for b in known.values():
if b.runner == "deepeval":
assert hasattr(shipped, b.runs.split(":", 1)[1]), b.name
# DeepEval's ARC runs the easy set by default; the row's figure is the challenge set's.
from deepeval.benchmarks.modes import ARCMode

assert benchmarks._settings_for(shipped.ARC, known["arc"].with_) == {"mode": ARCMode.CHALLENGE}


def test_deepeval_comes_with_what_its_benchmarks_import() -> None:
"""`datasets` (every benchmark) and `pandas` (MMLU) are imported when a benchmark is built,
and DeepEval's own package declares neither, so the `deepeval` extra does."""
import importlib

for module in ("datasets", "pandas"):
importlib.import_module(module)


def test_a_figure_recorded_under_an_alias_lands_on_the_rows_own_id(tmp_path: Path) -> None:
path = benchmarks.record(tmp_path, "qwen2.5:7b-instruct", "mmlu", 70.0, source="a run on this machine")
assert list(yaml.safe_load(path.read_text())["models"]) == ["qwen2.5-7b-instruct"]
rows = [e for e in resolve.load_catalogue(workspace=tmp_path).entries if "qwen2.5:7b-instruct" in e.aliases]
assert [e.name for e in rows] == ["qwen2.5-7b-instruct"], "one candidate, not the alias beside the row"
assert rows[0].scores["mmlu"] == pytest.approx(70.0)
again = benchmarks.record(tmp_path, "qwen2.5-7b-instruct", "gsm8k", 80.0, source="a run on this machine")
figures = yaml.safe_load(again.read_text())["models"]["qwen2.5-7b-instruct"]["benchmarks"]
assert set(figures) == {"mmlu", "gsm8k"}


def test_a_bound_model_says_which_of_an_agents_settings_it_would_throw_away() -> None:
Expand Down
Loading
Loading