Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,36 @@
What changed in PACT, newest first. Each entry says what an author can now write, or what now
holds that did not; the commits are listed under it for whoever wants the detail.

## 2026-11 — one detector dialect, read alike by every runtime (17 November)

A workspace's `detectors:` wrote their `looks-like:` patterns as any regular expression Python's `re`
compiled. A runtime written in another language reads them with its own engine, and engines disagree
on lookaround, back-references, `\b` beside non-ASCII letters, `$` before a final newline and what an
empty match steps past, so one tree could hide a card number on one runtime and let it through on
another.

### What holds now

- **`pact check` holds every `looks-like:` line to the detector dialect** (spec/schema.yaml,
`detector.looks-like`; `crates/pact-loader/src/dialect.rs`) and refuses the rest with
`loader/looks-like-outside-the-dialect`, naming what the pattern writes and where. The dialect is
what Python's `re` and the Rust `regex` crate both read, with Python's meaning: literals, `.`, `^`,
`$`, `\A`, `\Z`, `\b`, `\B`, the class escapes, `[...]` with ranges, character escapes, groups,
`(?P<name>...)`, `(?i:...)`, `(?m:...)`, `(?s:...)`, alternation and the quantifiers with their lazy
forms. A pattern that can match an empty value is outside it.
- **A TypeScript port** (`adapters/typescript/src/detectors.ts`) translates exactly the patterns in
the dialect to JavaScript that finds what Python's `re` finds, refuses the rest with the checker's
reasons, compiles a workspace's `detectors:` (holding each to its `examples:`) and carries the five
built-in recognisers.
- **Shared cases** (`tests/conformance/detectors.json`): 87 patterns over 68 texts, each in or outside
the dialect with its reason, and every match `re.finditer` yields for the ones in it. The checker
(`detector_dialect_cases.rs`), the reference and the TypeScript port
(`test_every_port_reads_one_detector_dialect.py`) read every case alike.

A tree whose detector writes a pattern outside the dialect no longer loads: rewrite it in the dialect
(a lookbehind usually becomes a group around the part before, a `(?<name>...)` becomes
`(?P<name>...)`).

## 2026-11 — a catalogue row says what a model needs of the machine that runs it (17 November)

A runtime choosing among models served on its own machine could not tell whether the machine could
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1001,11 +1001,11 @@ framework is an extra.
|---|---|
| **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete |
| **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5383 tests (1728 Rust + 3655 adapter), clippy clean, TypeScript type-checked** |
| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5392 tests (1733 Rust + 3659 adapter), clippy clean, TypeScript type-checked** |
| **Adapters** | **All 7 named targets**, proven against one shared conformance suite |

```bash
./scripts/test-all.sh # 5383 tests, Rust + 7 adapters, fully offline
./scripts/test-all.sh # 5392 tests, Rust + 7 adapters, fully offline
```

**What is still owed** is not hidden — it lives in
Expand Down
107 changes: 107 additions & 0 deletions adapters/python/tests/test_every_port_reads_one_detector_dialect.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
"""One detector dialect, read alike by the checker, Python and TypeScript.

A workspace's `detectors:` write the patterns a thing looks like, and a hiding
rule replaces every value one of them finds. A pattern that Python's `re` and
another runtime's engine read differently hides a value on one runtime and
lets it through on another, so `pact check` holds every `looks-like:` line to
the detector dialect (spec/schema.yaml, `detectors`) and refuses the rest with
`loader/looks-like-outside-the-dialect`.

`tests/conformance/detectors.json` holds the cases. This suite holds the
reference to them (each pattern in the dialect, compiled by
`detectors.declared_in`, finds exactly the listed matches), the TypeScript
port (`adapters/typescript/src/detectors.ts`, through `read-detectors.mts`:
the same refusals with the same reasons, the same matches), and the checker
(a tree whose detector writes a pattern outside the dialect is refused). The
Rust reader's refusals are held to the same cases by
`crates/pact-loader/tests/detector_dialect_cases.rs`.

Skips, stated rather than hidden: no `node` able to strip types, and no built
`pact` binary.
"""

from __future__ import annotations

import json
import re
import shutil
import subprocess
from pathlib import Path

import pytest

from pact_adapters.detectors import declared_in

REPO = Path(__file__).resolve().parents[3]
HELD = json.loads((REPO / "tests" / "conformance" / "detectors.json").read_text(encoding="utf-8"))
PACT = REPO / "target" / "debug" / "pact"


def _expected(case: dict) -> dict:
if case["outside"]:
return {"outside": True, "reason": case["reason"]}
return {"outside": False, "finds": case["finds"]}


def test_the_reference_finds_what_every_case_in_the_dialect_lists() -> None:
wrong = []
for case in HELD["cases"]:
if case["outside"]:
continue
pattern = declared_in({"detectors": {"thing": {"looks-like": [case["looks-like"]]}}})["thing"]
finds = []
for index, text in enumerate(HELD["texts"]):
found = [match.group(0) for match in re.finditer(pattern, text)]
if found:
finds.append({"in": index, "found": found})
if finds != case["finds"]:
wrong.append(case["looks-like"])
assert not wrong, wrong


def test_no_case_in_the_dialect_can_match_an_empty_value() -> None:
# The dialect leaves out patterns that can match nothing: Python's re and
# JavaScript step past an empty match differently, so their matches differ.
for case in HELD["cases"]:
if not case["outside"]:
assert all("" not in find["found"] for find in case["finds"]), case["looks-like"]


def test_the_typescript_port_reads_every_case_as_held() -> None:
if shutil.which("node") is None:
pytest.skip("node is not on PATH, so the TypeScript port cannot run")
run = subprocess.run(
["node", "--experimental-strip-types", "--no-warnings", str(REPO / "adapters" / "typescript" / "read-detectors.mts")],
input=json.dumps({"texts": HELD["texts"], "cases": HELD["cases"]}),
capture_output=True,
text=True,
)
assert run.returncode == 0, run.stderr
port = json.loads(run.stdout)
wrong = [
case["looks-like"] for case, answer in zip(HELD["cases"], port, strict=True) if answer != _expected(case)
]
assert not wrong, wrong


def test_the_checker_refuses_a_tree_whose_detector_is_outside_the_dialect(tmp_path: Path) -> None:
if not PACT.exists():
pytest.skip("no built pact binary at target/debug/pact")
case = next(c for c in HELD["cases"] if c["outside"] and c["looks-like"] == "(?<=x)y")
(tmp_path / "workspace.yaml").write_text(
"name: Desk\n"
"detectors:\n"
" order reference:\n"
" description: An order reference.\n"
f" looks-like:\n - {json.dumps(case['looks-like'])}\n"
" examples:\n - xy\n",
encoding="utf-8",
)
agent = tmp_path / "agents" / "desk"
agent.mkdir(parents=True)
(agent / "agent.yaml").write_text("description: Answers.\ninstructions: Answer.\n", encoding="utf-8")
run = subprocess.run([str(PACT), "check", str(tmp_path)], capture_output=True, text=True)
said = run.stdout + run.stderr
assert run.returncode != 0, said
assert "loader/looks-like-outside-the-dialect" in said, said
assert case["reason"] in said, said
22 changes: 22 additions & 0 deletions adapters/typescript/read-detectors.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// Read every case of `tests/conformance/detectors.json`, given on stdin, with
// the TypeScript port: whether `translateDetectorPattern` refuses the pattern
// and why, and otherwise every match it finds in each text, in order. The
// Python suite compares the answers with the cases, in
// `test_every_port_reads_one_detector_dialect.py`.
import { readFileSync } from "node:fs";
import { translateDetectorPattern } from "./src/detectors.ts";

const given = JSON.parse(readFileSync(0, "utf8")) as {
texts: string[];
cases: { "looks-like": string }[];
};
const answers = given.cases.map((c) => {
const translation = translateDetectorPattern(c["looks-like"]);
if (!translation.ok) return { outside: true, reason: translation.reason };
const finds = given.texts.flatMap((text, index) => {
const found = [...text.matchAll(new RegExp(translation.source, "gu"))].map((match) => match[0]);
return found.length === 0 ? [] : [{ in: index, found }];
});
return { outside: false, finds };
});
process.stdout.write(JSON.stringify(answers));
Loading
Loading