diff --git a/CHANGELOG.md b/CHANGELOG.md index 3dac6cc..a6ae42a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,30 @@ What changed in PACT, newest first. Each entry says what an author can now write, or what now holds that did not; the commits are listed under it for whoever wants the detail. +## 2026-11 — a runtime may serve a workspace without its broken agents (16 November) + +An error in any file of a workspace withdrew all of it: `pact discover` printed `skipping : N +problem(s)` and `pact show` exited 1, so a runtime serving a tree that is edited while it runs stopped +serving every agent the moment one agent was edited badly, agents nobody had touched among them. + +### What holds now + +- **`--leave-out-broken-agents` leaves out an agent whose own folder holds the errors, and serves the + rest.** With it, when every error lies in some agents' folders (`agents//…`), `pact show` and + `pact discover` validate the workspace again without those agents and serve it if it then loads, + exit 0. Each left-out agent's problems print on stderr as warnings, with `loader/agent-left-out` + naming it. +- **A runtime hears which agents were left out.** `pact discover --leave-out-broken-agents` adds + `"left-out": [{ "id": "pact:", "problems": N }]` to the workspace's inventory, so a runtime can + keep serving the version it last had of them. +- **Anything else still refuses the workspace.** An error outside the agents' folders, or one the + reduced workspace still has (a teammate naming a left-out agent), keeps the whole workspace refused. +- **Without the flag nothing changes.** `pact show`'s exit code is how a session's door and a sealed + plan decide that a document loads, and neither may lose an agent quietly, so isolation is asked for. +- **The author is still told.** `pact check ` refuses the tree. `pact check ` for a clean + agent beside broken ones now says `loader/other-agents-left-out` (a runtime that leaves them out runs + this agent) instead of `loader/the-workspace-around-it-is-broken`. + ## 2026-11 — PACT Core, first phase: actors and functions the checker reads, and a workspace that changes while it runs (16 November) The seven primitives were a way of classifying PACT (`docs/97`: "a mental model, not a syntax"). diff --git a/README.md b/README.md index d589ddb..e87b550 100644 --- a/README.md +++ b/README.md @@ -1001,11 +1001,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5310 tests (1719 Rust + 3591 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5312 tests (1721 Rust + 3591 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5310 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5312 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/crates/pact-cli/src/main.rs b/crates/pact-cli/src/main.rs index c22bb8b..6392269 100644 --- a/crates/pact-cli/src/main.rs +++ b/crates/pact-cli/src/main.rs @@ -16,7 +16,7 @@ mod suites; use anyhow::{Result, bail}; use camino::{Utf8Path, Utf8PathBuf}; -use pact_diag::Diagnostics; +use pact_diag::{Diagnostics, Severity}; use pact_loader::Loader; const USAGE: &str = "\ @@ -62,6 +62,13 @@ OPTIONS: of the run, the same way. --field For `pact fill`: `instructions` (the default) or `description`. + --leave-out-broken-agents + For `pact show` and `pact discover`: when every problem + of a workspace lies in some agents' own folders, leave + those agents out and serve the rest, naming each one + left out. For a runtime serving a tree that is edited + while it runs. Without it, one problem anywhere + refuses the whole workspace. --unsafe-spec Validate against $PACT_SPEC instead of the specification compiled into this binary. Development builds only, and the source is named on every run that uses it. @@ -84,6 +91,7 @@ const OPTIONS: &[&str] = &[ "--run-inputs", "--remembered", "--field", + "--leave-out-broken-agents", ]; /// The options that are followed by a value, which is theirs and not a path. @@ -357,6 +365,11 @@ fn run() -> Result { // `loader/never-offered` used to offer a fix that was itself refused when // typed, and this would have made that a wall. let deny_warnings = args.iter().any(|a| a == "--deny-warnings"); + // For a runtime serving a tree that is edited while it runs: one agent's bad + // edit leaves that agent out instead of the whole workspace. Opt-in, because + // `pact show`'s exit code is also how a session's door and a sealed plan + // decide that a document loads, and neither may lose an agent quietly. + let leave_out_broken = args.iter().any(|a| a == "--leave-out-broken-agents"); // Where this operator's agents really answer, for `pact card`. // // `agent_card` always took a base and the CLI always passed the literal @@ -411,9 +424,9 @@ fn run() -> Result { match cmd { "about" => about(), "check" => check(&path, quiet, unsafe_spec, deny_warnings), - "show" => show(&path, unsafe_spec), + "show" => show(&path, unsafe_spec, leave_out_broken), "waits" => waits_cmd(&path, unsafe_spec), - "discover" => discover_cmd(&path, unsafe_spec), + "discover" => discover_cmd(&path, unsafe_spec, leave_out_broken), "write" => write_cmd(&path), "build" => { let to = args @@ -1557,6 +1570,21 @@ fn durability_matches_what_this_tree_does( fn validate_as( path: &Utf8PathBuf, unsafe_spec: bool, +) -> Result<( + Option, + Diagnostics, + Vec, + &'static str, +)> { + validate_leaving_out(path, unsafe_spec, &[]) +} + +/// [`validate_as`], with the named agents taken out of the document before +/// anything else reads it, as though their folders were not there. +fn validate_leaving_out( + path: &Utf8PathBuf, + unsafe_spec: bool, + left_out: &[String], ) -> Result<( Option, Diagnostics, @@ -1566,6 +1594,17 @@ fn validate_as( &'static str, )> { let (mut node, mut diags) = load(path)?; + if !left_out.is_empty() + && let Some(agents) = node + .as_mut() + .and_then(|root| root.as_map_mut()) + .and_then(|root| root.get_mut("agents")) + .and_then(|entry| entry.node.as_map_mut()) + { + for key in left_out { + agents.shift_remove(key); + } + } // `based-on:` is resolved BEFORE the schema sees anything (G11). A derived // entry has to be validated as the thing it becomes: validating it as @@ -2057,6 +2096,77 @@ fn validate( Ok((node, diags, substituted)) } +/// A workspace as a runtime reads it: whole when it loads, or without the +/// agents whose own folders hold every error, when the rest then loads. +struct Isolated { + node: Option, + diags: Diagnostics, + /// Each agent left out, with how many errors its own files hold. + left_out: Vec<(String, usize)>, +} + +/// Validate `path` the way `show` and `discover` serve it to a runtime that +/// passed `--leave-out-broken-agents`; without it (`leave_out` false) the +/// workspace is validated whole, as every other command validates it. +/// +/// An error in one agent's own files (`agents//…`) used to withdraw the +/// whole workspace, so editing one agent online took every other agent of the +/// workspace out of service with it. When every error lies in the folders of +/// some agents, those agents are left out, the rest is validated again without +/// them, and it is served if it then loads, with each left-out agent's +/// problems printed as warnings. An error anywhere else, or one the reduced +/// document still has (a teammate naming a left-out agent), keeps the whole +/// workspace refused, as before. `pact check` still refuses the tree. +fn isolate(path: &Utf8PathBuf, unsafe_spec: bool, leave_out: bool) -> Result { + let (node, diags, _, _) = validate_as(path, unsafe_spec)?; + let whole = |node, diags| Isolated { node, diags, left_out: Vec::new() }; + if !leave_out || !diags.has_errors() || !path.is_dir() { + return Ok(whole(node, diags)); + } + let mut counts: Vec<(String, usize)> = Vec::new(); + for d in diags.items().iter().filter(|d| d.severity == Severity::Error) { + let file = d.span.file.as_str(); + let relative = file + .strip_prefix(path.as_str()) + .unwrap_or(file) + .trim_start_matches("./") + .trim_start_matches('/'); + let key = relative + .strip_prefix("agents/") + .and_then(|rest| rest.split('/').next()) + .filter(|key| !key.is_empty() && path.join("agents").join(key).is_dir()); + let Some(key) = key else { + return Ok(whole(node, diags)); + }; + match counts.iter_mut().find(|(k, _)| k == key) { + Some((_, n)) => *n += 1, + None => counts.push((key.to_string(), 1)), + } + } + counts.sort(); + let keys: Vec = counts.iter().map(|(k, _)| k.clone()).collect(); + let (reduced, mut kept, _, _) = validate_leaving_out(path, unsafe_spec, &keys)?; + if kept.has_errors() || reduced.is_none() { + return Ok(whole(node, diags)); + } + kept.borrow_sources_from(&diags); + for (key, n) in &counts { + kept.push(pact_diag::Diagnostic::warning( + "loader/agent-left-out", + pact_diag::Span::in_folder(path.join("agents").join(key), "agent.yaml"), + format!( + "Agent '{key}' is left out until its {n} problem(s) are fixed; the workspace's other agents load without it." + ), + format!("Run `pact check {path}` and fix what it prints under `agents/{key}/`."), + )); + } + for mut d in diags.items().iter().filter(|d| d.severity == Severity::Error).cloned() { + d.severity = Severity::Warning; + kept.push(d); + } + Ok(Isolated { node: reduced, diags: kept, left_out: counts }) +} + /// The folder a path names, which is the path itself when it is one. /// /// `pact check` accepts a single file as well as a directory, and every question @@ -2635,8 +2745,9 @@ fn check_in_context( // Printing only the named folder's problems is right — it is the question // that was asked — but staying SILENT about the rest was the same false `OK` // C13 was about, reached a different way. `pact discover` refuses a workspace - // it cannot load whole (`skipping : N problem(s)`, then `[]`), so an - // agent in a tree with an error anywhere in it cannot be run by any runtime, + // it cannot load whole (`skipping : N problem(s)`, then `[]`), and with + // `--leave-out-broken-agents` one whose errors are not all in some agents' + // own folders, so an agent in such a tree cannot be run by any runtime, // however clean its own folder is. Measured, before this: // // ```text @@ -2652,7 +2763,30 @@ fn check_in_context( // be answering for somebody else's file; `--deny-warnings` still fails, and // `pact check ` is named so the next step is one command away. let elsewhere = whole.error_count() - mine.error_count(); - if elsewhere > 0 { + // When every problem elsewhere lies in other agents' own folders, `show` and + // `discover` with `--leave-out-broken-agents` leave those agents out and + // serve this one (see [`isolate`]). + let key = here.file_name().map(str::to_string); + let runs_without_them = elsewhere > 0 + && mine.error_count() == 0 + && isolate(root, unsafe_spec, true).is_ok_and(|served| { + !served.diags.has_errors() && !served.left_out.iter().any(|(k, _)| Some(k) == key.as_ref()) + }); + if runs_without_them { + let start = + pact_loader::firstfile::to_start(root, &pact_loader::policy::Policy::default(), root); + mine.push(pact_diag::Diagnostic::warning( + "loader/other-agents-left-out", + pact_diag::Span::in_folder(root.as_str(), start.as_str()), + format!( + "'{root}' — the workspace this agent is in — has {elsewhere} \ + problem(s) in other agents' folders: a runtime that reads it \ + with `--leave-out-broken-agents` runs this agent without them; \ + any other skips the whole workspace until they are fixed." + ), + format!("Run `pact check {root}` to see them, and fix those too."), + )); + } else if elsewhere > 0 { let start = pact_loader::firstfile::to_start(root, &pact_loader::policy::Policy::default(), root); mine.push(pact_diag::Diagnostic::warning( @@ -2661,7 +2795,8 @@ fn check_in_context( format!( "'{root}' — the workspace this agent is in — has {elsewhere} \ problem(s) in other files, so nothing in this tree can run yet: \ - `pact discover` skips a workspace it cannot load whole." + `pact discover` skips a workspace whose problems are not all in \ + some agents' own folders." ), format!("Run `pact check {root}` to see them, and fix those too."), )); @@ -2792,7 +2927,7 @@ fn described_with_a_hole( /// D2/AC-6.1: the runtime finds agents by walking the tree. No registration /// step, no build artifact, no `.pact/` dependency. -fn discover_cmd(path: &Utf8PathBuf, unsafe_spec: bool) -> Result { +fn discover_cmd(path: &Utf8PathBuf, unsafe_spec: bool, leave_out_broken: bool) -> Result { let roots = discover::find_workspaces(path, 6); let mut out = Vec::new(); for root in &roots { @@ -2800,8 +2935,8 @@ fn discover_cmd(path: &Utf8PathBuf, unsafe_spec: bool) -> Result { // schema sees anything (G11), so the inventory publishes each agent as // the thing it becomes. The bare re-load this replaces published // `model: null` and `limits: null` for every derived agent (D-5). - let (node, mut d, _) = match validate(root, unsafe_spec) { - Ok(v) => v, + let (node, mut d, left_out) = match isolate(root, unsafe_spec, leave_out_broken) { + Ok(Isolated { node, diags, left_out }) => (node, diags, left_out), Err(e) => { // Never silent: a folder this walk found and then could not // read is named, the way every other skip here is. @@ -2839,10 +2974,26 @@ fn discover_cmd(path: &Utf8PathBuf, unsafe_spec: bool) -> Result { if !d.is_empty() { eprint!("{}", d.render()); } - out.push(discover::inventory(&discover::Found { + let mut inventory = discover::inventory(&discover::Found { root: root.clone(), document, - })); + }); + // The agents this workspace holds and does not serve until their own + // problems are fixed, so a runtime can keep what it last served of them. + if !left_out.is_empty() + && let Some(object) = inventory.as_object_mut() + { + object.insert( + "left-out".into(), + serde_json::Value::Array( + left_out + .iter() + .map(|(key, problems)| serde_json::json!({ "id": format!("pact:{key}"), "problems": problems })) + .collect(), + ), + ); + } + out.push(inventory); } out!( "{}", @@ -3193,11 +3344,11 @@ fn build_cmd(path: &Utf8PathBuf, named: bool, to: Option) -> Result Ok(0) } -fn show(path: &Utf8PathBuf, unsafe_spec: bool) -> Result { +fn show(path: &Utf8PathBuf, unsafe_spec: bool, leave_out_broken: bool) -> Result { // Validated, for the reason `waits_cmd` gives: every adapter in this // repository reads its document through `pact show`, so a tree that only // `check` refuses is a tree every adapter accepts. - let (node, mut diags, _) = validate(path, unsafe_spec)?; + let Isolated { node, mut diags, .. } = isolate(path, unsafe_spec, leave_out_broken)?; if diags.has_errors() { eprint!("{}", diags.render()); return Ok(1); diff --git a/crates/pact-cli/tests/one_broken_agent_leaves_its_workspace_served.rs b/crates/pact-cli/tests/one_broken_agent_leaves_its_workspace_served.rs new file mode 100644 index 0000000..d244d9b --- /dev/null +++ b/crates/pact-cli/tests/one_broken_agent_leaves_its_workspace_served.rs @@ -0,0 +1,96 @@ +//! With `--leave-out-broken-agents`, an error in one agent's own folder leaves +//! that agent out of what `pact show` and `pact discover` serve, not the whole +//! workspace; without it, nothing changes. +//! +//! Editing an agent online used to withdraw every agent of its workspace the +//! moment the edit was invalid: a runtime serving the tree read `discover`'s +//! `skipping .: 1 problem(s)` and `show`'s exit 1, and stopped serving agents +//! nobody had touched. `pact check` still refuses the tree, so an author is +//! told; a runtime that asks keeps serving what is still valid, and hears which +//! agents were left out. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +const LEAVE_OUT: &str = "--leave-out-broken-agents"; + +fn pact() -> Command { + Command::new(env!("CARGO_BIN_EXE_pact")) +} + +/// A workspace with a valid `clerk` and an `auditor` whose limit is below the floor. +fn tree(name: &str, clerk_team: bool) -> PathBuf { + let root = std::env::temp_dir().join(format!("pact-one-broken-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let write = |path: &str, text: &str| { + let p = root.join(path); + std::fs::create_dir_all(p.parent().unwrap()).unwrap(); + std::fs::write(p, text).unwrap(); + }; + write("workspace.yaml", "name: desk\ndescription: A desk of two agents.\n"); + let team = if clerk_team { "team:\n auditor: reads the clerk's decisions\n" } else { "" }; + write( + "agents/clerk/agent.yaml", + &format!("description: Decides tickets.\ninstructions: Decide each ticket.\n{team}"), + ); + write( + "agents/auditor/agent.yaml", + "description: Audits decisions.\ninstructions: Read the decisions.\nlimits:\n steps-at-most: -1\n when-it-runs-out: answer-with-what-it-has\n", + ); + root +} + +fn run(args: &[&str], root: &Path) -> (i32, String, String) { + let out = pact().args(args).arg(root).output().unwrap(); + ( + out.status.code().unwrap_or(-1), + String::from_utf8_lossy(&out.stdout).into_owned(), + String::from_utf8_lossy(&out.stderr).into_owned(), + ) +} + +#[test] +fn the_broken_agent_is_left_out_and_its_sibling_is_served() { + let root = tree("sibling", false); + let (code, stdout, stderr) = run(&["discover", LEAVE_OUT], &root); + assert_eq!(code, 0, "{stderr}"); + let inventory: serde_json::Value = serde_json::from_str(&stdout).unwrap(); + let ids: Vec<&str> = inventory[0]["agents"].as_array().unwrap().iter().map(|a| a["id"].as_str().unwrap()).collect(); + assert_eq!(ids, ["pact:clerk"], "{stdout}"); + assert_eq!(inventory[0]["left-out"], serde_json::json!([{ "id": "pact:auditor", "problems": 1 }])); + assert!(stderr.contains("Agent 'auditor' is left out until its 1 problem(s) are fixed"), "{stderr}"); + assert!(stderr.contains("loader/agent-left-out") && stderr.contains("schema/below-the-floor"), "{stderr}"); + + let (code, stdout, stderr) = run(&["show", LEAVE_OUT], &root); + assert_eq!(code, 0, "{stderr}"); + let document: serde_json::Value = serde_json::from_str(&stdout).unwrap(); + let agents: Vec<&String> = document["agents"].as_object().unwrap().keys().collect(); + assert_eq!(agents, ["clerk"], "{stdout}"); + + // Without the flag the workspace is refused whole, as a session's door and a + // sealed plan need it to be. + let (code, stdout, stderr) = run(&["discover"], &root); + assert_eq!((code, stdout.trim()), (0, "[]"), "{stderr}"); + assert!(stderr.contains("skipping"), "{stderr}"); + let (code, _, _) = run(&["show"], &root); + assert_eq!(code, 1); + + // The author is still refused, and checking the clean agent says what a runtime does. + let (code, stdout, _) = run(&["check"], &root.join("agents/clerk")); + assert_eq!(code, 0, "{stdout}"); + assert!(stdout.contains("loader/other-agents-left-out"), "{stdout}"); + let (code, stdout, _) = run(&["check"], &root); + assert_eq!(code, 1, "{stdout}"); + assert!(stdout.contains("schema/below-the-floor"), "{stdout}"); +} + +#[test] +fn a_teammate_that_names_the_broken_agent_keeps_the_whole_workspace_refused() { + let root = tree("teammate", true); + let (code, stdout, stderr) = run(&["discover", LEAVE_OUT], &root); + assert_eq!(code, 0, "{stderr}"); + assert_eq!(stdout.trim(), "[]", "{stdout}"); + assert!(stderr.contains("problem(s)"), "{stderr}"); + let (code, _, _) = run(&["show", LEAVE_OUT], &root); + assert_eq!(code, 1); +} diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index 4b85429..73e4606 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,7 +80,7 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1719 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Rust | 5 crates, **1721 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | | Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3591 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | diff --git a/site-docs/index.md b/site-docs/index.md index 97ff0d6..b4f6ae2 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -47,7 +47,7 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,310** — 1719 Rust, 3591 adapter | +| Tests | **5,312** — 1721 Rust, 3591 adapter | | Schema kinds | **69** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | diff --git a/site-docs/reference/cli.md b/site-docs/reference/cli.md index 3e91a51..a66ac32 100644 --- a/site-docs/reference/cli.md +++ b/site-docs/reference/cli.md @@ -73,6 +73,12 @@ The loaded document as JSON — what every adapter actually receives. Useful for seeing what the [Expansion Rule](../concepts/expansion-rule.md) produced from your tree, and what a `based-on:` inherited. +With `--leave-out-broken-agents`, a workspace whose problems all lie in some +agents' own folders (`agents//…`) is printed without those agents, and +each one left out is named on stderr (`loader/agent-left-out`). Without it, one +problem anywhere exits 1, which is how a session's door and a sealed plan know +a document loads. + ## `pact waits` Every point where a run can stop and wait for a person, with the deadline a @@ -107,6 +113,15 @@ for a runtime deciding whether to take the tree: `boundary` is `open` (no `listed`, and `allows` is the parts written. An absent line and an empty list are opposite answers, and `allows` is empty for both: read `boundary`. +A runtime serving a tree that is edited while it runs passes +`--leave-out-broken-agents`, so that one agent's bad edit does not take the +workspace's other agents out of service. A workspace whose problems all lie in +some agents' own folders is then listed without those agents, with +`"left-out": [{ "id": "pact:", "problems": N }]` naming each one, and the +runtime can keep serving the version it last had of them. A problem anywhere +else, or one the workspace still has without them (a teammate naming a +left-out agent), skips the workspace as before. + ```console $ pact discover tests/trees/workflows-16-trial-booking [{ …, "digest": "sha256:…", diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 82ee84b..de22ba0 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,9 +12,9 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1719** | `cargo test --workspace` | +| Rust tests | **1721** | `cargo test --workspace` | | Adapter tests | **3591** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,310** | both of the above | +| Total | **5,312** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` |