From 25592b575e23b6f118c99f3adb4099efa01654f8 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 29 Oct 2026 16:37:25 +0800 Subject: [PATCH 1/2] feat: ship behavior-review with guided goals and actionable recommendations Signed-off-by: Kent Huang --- README.md | 34 ++++-- docs/architecture.md | 61 +++++++--- e2e/README.md | 20 ++-- ...tion-handoff-journey-first-architecture.md | 4 +- .../skills/behavior-review}/SKILL.md | 113 +++++++++++------- .../behavior-review}/fixtures/synthetic.json | 0 .../behavior-review}/references/execution.md | 89 ++++++++++---- .../behavior-review}/references/replay.md | 4 +- .../skills/behavior-review}/scripts/replay.py | 2 +- ...totype-test.py => behavior-review-test.py} | 5 +- 10 files changed, 225 insertions(+), 107 deletions(-) rename {.agents/skills/verification-handoff-prototype => plugin/skills/behavior-review}/SKILL.md (55%) rename {.agents/skills/verification-handoff-prototype => plugin/skills/behavior-review}/fixtures/synthetic.json (100%) rename {.agents/skills/verification-handoff-prototype => plugin/skills/behavior-review}/references/execution.md (57%) rename {.agents/skills/verification-handoff-prototype => plugin/skills/behavior-review}/references/replay.md (96%) rename {.agents/skills/verification-handoff-prototype => plugin/skills/behavior-review}/scripts/replay.py (99%) rename tests/{verification-handoff-prototype-test.py => behavior-review-test.py} (98%) diff --git a/README.md b/README.md index 97c16ff..ec797e1 100644 --- a/README.md +++ b/README.md @@ -90,22 +90,42 @@ codex plugin add behavior-diff@spacedock ## Use Behavior Diff -1. Edit one instruction file. Keep the change uncommitted. -2. Ask Claude Code or Codex to run Behavior Diff on the change. -3. Review the report before you commit the instruction. +Use the shipped [`behavior-review` skill](plugin/skills/behavior-review/SKILL.md) +for a guided review of an instruction change or PR through a practical adoption +recommendation. It owns the journey; the underlying `behavior-diff` skill owns +comparison preparation and assessment, and the existing runner executes trials. -For example: +In Claude Code: ```text -Run behavior diff on my AGENTS.md change. +/behavior-diff:behavior-review Review my AGENTS.md change and recommend whether to adopt it. +``` + +In Codex: + +```text +$behavior-review Review my AGENTS.md change and recommend whether to adopt it. ``` -In Claude Code, you can also run: +Supply the change or PR and any context you permit the agent to inspect. The +agent reviews the change, follows the comparison through completion, and leads +its first final answer with what to do and why. Decision-changing risks stay +visible; detailed evidence and methodology stay in linked evidence. It reuses +existing authority and asks once for missing bounded run approval. Paid model +calls and external effects still require their own applicable authority; a +recommendation never authorizes commit, push, merge, release, or publication. + +For a comparison without the guided review journey, keep an instruction edit +uncommitted and ask: ```text -/behavior-diff +Run behavior diff on my AGENTS.md change. ``` +Or invoke `/behavior-diff:behavior-diff` in Claude Code or `$behavior-diff` in +Codex. Explicit PR or historical Git inputs are also supported; they do not +require an uncommitted edit. + Behavior Diff finds the changed instruction file and works out what the change is meant to do and what it must preserve. From that intent, the agent designs a neutral task and, when needed, coherent ordinary and edge-case test data, plus diff --git a/docs/architecture.md b/docs/architecture.md index 4c09725..76054e4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,7 +18,13 @@ scripts handle execution, evidence, and reporting. +----------------------------+----------------------------+ v +---------------------------------------------------------+ -| Skill preparation (LLM; reference recipe) | +| behavior-review: guided review journey (optional entry) | +| Review context -> reuse authority / one bounded proposal| +| Delegate comparison to sibling behavior-diff skill | ++----------------------------+----------------------------+ + v ++---------------------------------------------------------+ +| behavior-diff preparation (LLM; reference recipe) | | Change intent + preservation -> capabilities/premises | | -> neutral task + ordinary/bounded edge data + private | | case criteria; reuse sufficient project data | @@ -59,6 +65,13 @@ scripts handle execution, evidence, and reporting. | reached / missed / unknown; intent supported / | | contradicted / insufficient evidence; run health apart; | | report agreement or gap disclosed; user decides | ++---------------------------------------------------------+ + v ++---------------------------------------------------------+ +| behavior-review: first final answer (guided entry) | +| Action + reason first; decision-changing risks visible | +| Detailed evidence and methodology linked; no adoption | +| action or external effect implied | +---------------------------------------------------------+ ``` @@ -1023,29 +1036,33 @@ synthetic local emulators only. An explicit publication-policy update and review of actual outgoing content remain required before uploading any private session; blinding and private repository visibility are not publication permission. -## Local verification handoff prototype +## Guided behavior review -The repository-local [maintainer skill](../.agents/skills/verification-handoff-prototype/SKILL.md) -owns the journey, the canonical Behavior Diff skill owns the comparison, and the -existing runner owns execution. These are responsibilities, not new processes -or an additional coordinator. +The shipped [`behavior-review` skill](../plugin/skills/behavior-review/SKILL.md) +is the guided user entry. It owns the journey, the sibling `behavior-diff` +skill owns the comparison, and the existing runner owns execution. These are +responsibilities, not new processes or an additional coordinator. Direct +`behavior-diff` requests still enter comparison preparation without the review +journey. ```text Owner's question + selected change + existing authority - -> handoff: review context and delegate comparison to Behavior Diff - -> Behavior Diff: resolve inputs, feasibility, and missing run terms - -> handoff: one proposal only for missing authority or material decisions + -> behavior-review: review context and delegate comparison to behavior-diff + -> behavior-diff: resolve inputs, feasibility, and missing run terms + -> behavior-review: one proposal for missing authority or material decisions -> Behavior Diff: prepare -> freeze -> verify -> existing runner: fresh trials -> recorded evidence -> existing posttrial path - -> Behavior Diff: assess reached cases, findings, limits, and execution health - -> handoff: practical recommendation + adoption risk (no merge) + -> behavior-diff: assess reached cases, findings, limits, and execution health + -> behavior-review: action-first recommendation + adoption risk (no merge) ``` -The handoff follows delegated work through completion without owner reminders, +Behavior review follows delegated work through completion without owner reminders, but does not maintain another scenario recipe, launch command, or behavioral -assessment. It uses one resolved Behavior Diff version even when reviewing a -different repository. Existing frozen input records, run handles, status readers, -and evidence remain authoritative; this introduces no new handoff schema or +assessment. It resolves `behavior-diff` from the same installed plugin's sibling +skill directory, including when reviewing a different repository, rather than +from a maintainer `.agents` directory or the original source checkout. +Existing frozen input records, run handles, status readers, and evidence remain +authoritative; this introduces no new handoff schema or automatic execution-resume promise. Ordinary comparisons use the existing runner's supported host path. The optional @@ -1058,6 +1075,11 @@ without prior discussion or outcomes. Required safety or evidence boundaries cannot be waived merely to complete a journey. The handoff reuses existing authorization and requests only missing terms. +When the goal is ambiguous, it first inspects the change, then offers 2–4 +outcome-based options with a recommendation in the same proposal as the run terms. +Clear goals skip this menu. Explicit approval accepts the stated goal and covered +run terms; selecting a goal alone does not authorize execution. A changed goal +requires only uncovered terms to be approved, not a new stage-by-stage consent loop. Paid runs, changed provider/data exposure, and expanded scope need bounded authority. No global configuration writes or PR operations are implied. An execution failure stops the affected work, not safe non-model diagnosis or @@ -1066,10 +1088,11 @@ recovery; the handoff carries it through within authority or explains the remaining decision. It does not build a replacement launcher. The agreed question determines whether a simulation is sufficient or a live -service is required. The final answer explains what to do with the change, why, -and what remains unverified. Detailed evidence stays in private links; risks -that affect the decision remain visible. A useful null result can complete the -journey; missing required evidence or failed execution cannot. +service is required. The first final answer leads with what to do with the +change and why, followed by decision-changing risk and remaining required +evidence gaps. Non-decision-changing methodology and detailed evidence stay in +private links. A useful null result can complete the journey; missing required +evidence or failed execution cannot. Explicit replay-only requests use saved evidence without new trials. Optional `replay.py` owns private snapshots, correction invalidation, confirmed revisions diff --git a/e2e/README.md b/e2e/README.md index f43104c..3da52a9 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -30,20 +30,24 @@ Keep private sources, trials, reports, and temporary comparisons outside this checkout. No model-backed verification runs in CI. Synthetic fixtures remain the deterministic test set and supplemental edge-case examples. -## Local verification handoff prototype +## Behavior review and replay checks -For the isolated, conversational handoff-to-decision replay, explicitly load -[the maintainer skill source](../.agents/skills/verification-handoff-prototype/SKILL.md) -in Claude Code or Codex, then supply the change, permitted context and case root. -OMP can load that same path; this is not an installed-plugin or runtime-adapter claim. -The host operates the helper; the owner should not have to choose evidence IDs. +The shipped [`behavior-review` skill](../plugin/skills/behavior-review/SKILL.md) +is the guided review-to-recommendation entry. In Claude Code invoke +`/behavior-diff:behavior-review`; in Codex invoke `$behavior-review`, then supply +the selected change or PR and permitted context. It delegates comparison to the +sibling `behavior-diff` skill in the same installed plugin. No source-checkout +maintainer skill is needed. Live comparisons still require bounded model-cost +approval; replay-only requests do not run trials or external actions. -Python 3.10+ is sufficient. The skill includes the one-command helper invocation +For the conversational fixed-evidence replay, also supply the private case root. +The host operates the helper; the owner should not have to choose evidence IDs. +Python 3.10+ is sufficient for replay. The skill includes the helper invocation and `fixtures/synthetic.json` for an offline, invented demonstration. Run its deterministic behavior checks separately with: ```sh -python3 tests/verification-handoff-prototype-test.py +python3 tests/behavior-review-test.py ``` Use a mode-0700 directory outside all Git trees for mode-0600 session/packet files. diff --git a/plans/2026-10-29-verification-handoff-journey-first-architecture.md b/plans/2026-10-29-verification-handoff-journey-first-architecture.md index 719f471..b821a32 100644 --- a/plans/2026-10-29-verification-handoff-journey-first-architecture.md +++ b/plans/2026-10-29-verification-handoff-journey-first-architecture.md @@ -495,8 +495,8 @@ No release, retirement of live mode, or reduction of host support is implied. ## References - [Current architecture](../docs/architecture.md) -- [Verification handoff skill](../.agents/skills/verification-handoff-prototype/SKILL.md) -- [Current handoff execution instructions](../.agents/skills/verification-handoff-prototype/references/execution.md) +- [Behavior review skill](../plugin/skills/behavior-review/SKILL.md) +- [Current handoff execution instructions](../plugin/skills/behavior-review/references/execution.md) - [Canonical Behavior Diff skill](../plugin/skills/behavior-diff/SKILL.md) - [Canonical preparation and trusted launch recipe](../plugin/skills/behavior-diff/references/scenario-and-test-data.md) - [Existing trial adapter](../plugin/skills/behavior-diff/scripts/run-trial.sh) diff --git a/.agents/skills/verification-handoff-prototype/SKILL.md b/plugin/skills/behavior-review/SKILL.md similarity index 55% rename from .agents/skills/verification-handoff-prototype/SKILL.md rename to plugin/skills/behavior-review/SKILL.md index 17b6008..ee9603e 100644 --- a/.agents/skills/verification-handoff-prototype/SKILL.md +++ b/plugin/skills/behavior-review/SKILL.md @@ -1,15 +1,16 @@ --- -name: verification-handoff-prototype -description: Carries an instruction change from one owner-approved handoff proposal through review and Behavior Diff verification to a decision-ready recommendation. Use when the owner wants to hand off verification of a change or PR without coordinating the agent's next steps. +name: behavior-review +description: Reviews an instruction change or PR and carries one bounded owner-approved verification through Behavior Diff to an action-first recommendation. Use when the owner wants to know whether to merge, revise, or hold a behavior change without coordinating the verification steps. --- -# Verification handoff prototype +# Behavior Review Own verification through a decision-ready result. Success means less owner -coordination, not completion of a checklist. The handoff owns the journey; +coordination, not completion of a checklist. Behavior Review owns the journey; Behavior Diff owns the comparison; the existing runner owns execution. -Use the same workflow on Claude Code and Codex. OMP may load this source explicitly. -Resolve bundled files relative to this loaded skill, never the target repository. +Use the same workflow on Claude Code and Codex. Other hosts may load this skill. +Resolve bundled files relative to this loaded skill's physical directory, never +the target repository or an original source checkout. ## Ownership and authority @@ -51,11 +52,26 @@ subject. Explain what benefit was observed, what risk remains, and whether that risk changes the recommendation. The owner must not need to open a report to understand the decision. -Use short, natural paragraphs. Translate retain/revise/hold into a concrete action: -"keep these instructions", "change this rule", or "wait for evidence about this -behavior". Name what is on hold; a failed test is not a reason by itself to reject -the PR. Distinguish intended benefits from demonstrated benefits. Do not turn -missing evidence into reassurance. +Use short, natural paragraphs. Name the adoption decision and the concrete action +the owner should take; "keep the instructions" alone does not answer whether to +merge a PR. A failed test is not a reason by itself to reject the PR. Distinguish +intended benefits from demonstrated benefits, and do not turn missing evidence +into reassurance. + +The first post-verification conclusion must give the action without waiting for +the owner to ask: **"Recommend merging "**, **"Recommend revising +to "**, or **"Hold : "**. For a change without a PR, name the equivalent keep/revise/hold +decision for that change. Give the reason immediately, then the observed benefit, +preserved safeguards and any uncertainty that changes the decision. Do not lead +with a verification-status label or defer the recommendation to a follow-up. + +State supported confidence in the owner's actual safeguards: name which checks, +approval stops or other protections remained intact in the inspected evidence. +Do not promise guarantees or turn a missing observation into reassurance. Do not +append default "only simulation" or "not E2E" boilerplate. Explain a method limit +in the answer when its practical consequence changes the decision; otherwise put +the accurate scope and methodology in linked evidence. Keep commands, trace formats, sandbox details, source mappings and diagnostic history in linked evidence unless they change the owner's decision. Explain @@ -70,22 +86,32 @@ Humanizer guidance in embedded mode without installing it or changing evidence. 1. **Understand the decision without model calls.** Resolve and load the canonical `behavior-diff` skill using the source-selection rule in [execution handoff](references/execution.md). Inspect the selected diff, - relevant unchanged rules and available discussion. Give Behavior Diff the - owner's question and source context; use its method to resolve inputs, - requirements, capabilities, tested scope and claim limits. Do not require the - owner to assemble an evidence packet or confirm facts the repository answers. - By default, answer whether the change helps in the supported current - environment; do not ask the owner to choose a technical comparison mode. -2. **Settle authority once.** If needed, send one proposal with the intended - outcomes, preserved requirements, tested scope, important limits and missing - run terms, with explicit/inferred sources and a recommendation. Behavior Diff - supplies comparison-specific terms; communicate them, not a design exercise. - Skip renewed consent for terms already authorized. Keep study questions out. - A goal correction requires reassessing affected evidence and resending only - changed terms. + relevant unchanged rules and available discussion before asking about the goal. + If the request leaves materially different verification goals open, offer 2–4 + concrete, outcome-based options derived from that change. Mark one recommended + and explain why; usually check both the intended new behavior and preservation + of existing safeguards. Allow the owner to name a different concern. Do not + ask only "What do you want to verify?" or present model, fixture or execution + mode choices as goals. If the goal is already clear, skip the options. + Give Behavior Diff the proposed or explicit question and source context; use + its method to resolve inputs, capabilities, tested scope and run terms. Do not + require the owner to assemble evidence or confirm repository-provided facts. +2. **Settle goal and authority together.** Put any goal options and their + recommendation in the same handoff proposal as the applicable bounded run + terms, following [execution handoff](references/execution.md#handoff-proposal). + State which option each set of terms covers. Explicitly say that "approve" + accepts the recommended goal and its stated run terms; choosing a goal alone + is not run authorization. Do not add separate scope and execution approval + rounds when one proposal can settle both. Reuse existing authorization and + skip renewed consent for covered terms. A different goal requires reassessing + affected scope and resending only terms not already authorized. Keep study + questions out; the owner chooses the outcome, not the comparison design. 3. **Review, then prepare verification.** Inspect instructions and saved evidence. - Review completion is not verification completion. For changed agent behavior, - continue directly through Behavior Diff's canonical preparation and preflight. + For a PR, independently gather the relevant normal code-review findings, CI + results and mergeability through read-only repository/forge tools; do not ask + the owner to assemble them. Review completion is not verification completion. + For changed agent behavior, continue directly through Behavior Diff's canonical + preparation and preflight. Existing evidence can satisfy the question only when revisions, conditions and coverage match; explain that conclusion. Report a material ambiguity without editing the source or silently testing a repaired version. @@ -102,23 +128,30 @@ Humanizer guidance in embedded mode without installing it or changing evidence. supported route is work to do, not owner homework. Ask again only for a material goal correction, unavailable fact or authority beyond approved terms. Never rerun merely to obtain a difference. -6. **Return a decision-ready result.** Follow the explanation guidance above. - Recommend retain, revise or hold in ordinary language, naming the change and - the reason. Explain what improved versus Before, what stayed intact, and what - remains uncertain in terms of the owner's workflow. Say what adopting now - would leave unverified and whether to change the PR, wait, or leave it alone - while verification continues. Put file mappings and report corrections in the - linked evidence; bring any correction that changes the recommendation into - the answer. Equal results do not prove a rule unnecessary, and simulated - behavior does not prove real-service reliability. +6. **Return a decision-ready result.** Give the action-first conclusion above in + the first post-verification answer, not only after an adoption question. + Explain what improved versus Before, what stayed intact, and what remains + uncertain in terms of the owner's workflow. Apply the merge-readiness rule + below before recommending merge. Put file mappings, detailed methodology and + report corrections in linked evidence; surface any correction or uncertainty + that changes the recommendation. Equal results do not prove a rule unnecessary, + and evidence about agent decisions does not establish unobserved service behavior. ## Adoption questions -Merge is the owner's decision, not a verification outcome. When asked whether to -merge, restate the retain/revise/hold result and name the merge-readiness checks -that did and did not run, such as code review, CI and mergeability. A summary of -intended changes is not a code review. Do not upgrade the recommendation under a -leading question. Do not merge. +Recommend merging a PR only when the behavior evidence supports adoption and the +relevant normal code review, current CI results and mergeability have been checked +for the reviewed head. Independently obtain these read-only, including unresolved +material review findings and required checks; a summary of intended changes is not +a code review, and missing, pending or skipped required checks are not passes. +If a required check is unavailable or inconclusive, hold the merge recommendation +and name that blocker while preserving any supported behavioral findings. Refresh +readiness if the head changes; do not apply evidence to an unreviewed revision. + +Merge remains the owner's decision. A recommendation is not execution +authorization: do not merge, publish, adopt or edit the source under this skill. +When asked later whether to merge, use the same evidence standard; do not upgrade +the recommendation merely because the question leads toward approval. When the owner asks whether the journey needed them, answer from the session: list each owner input and whether it was necessary, without agreeing to a framing diff --git a/.agents/skills/verification-handoff-prototype/fixtures/synthetic.json b/plugin/skills/behavior-review/fixtures/synthetic.json similarity index 100% rename from .agents/skills/verification-handoff-prototype/fixtures/synthetic.json rename to plugin/skills/behavior-review/fixtures/synthetic.json diff --git a/.agents/skills/verification-handoff-prototype/references/execution.md b/plugin/skills/behavior-review/references/execution.md similarity index 57% rename from .agents/skills/verification-handoff-prototype/references/execution.md rename to plugin/skills/behavior-review/references/execution.md index 3d21fcb..0b6e725 100644 --- a/.agents/skills/verification-handoff-prototype/references/execution.md +++ b/plugin/skills/behavior-review/references/execution.md @@ -1,36 +1,70 @@ # Execution handoff The host owns this work. Do not expose helper setup as owner homework. Resolve -this handoff skill's physical location before choosing Behavior Diff. If it -belongs to a Behavior Diff source checkout, load that checkout's canonical -`plugin/skills/behavior-diff` source, even when the reviewed project is elsewhere. -Otherwise load the discovered installed Behavior Diff skill. Retain its exact -base and version for the whole comparison; never silently mix versions. -Missing tooling is a concrete prerequisite to resolve or report, not permission -to invent a substitute runner. +Behavior Review's physical installed skill directory, following any symlink to +its actual location. Load exactly `../behavior-diff/SKILL.md` relative to that +directory: the comparison skill is a sibling in the same installed plugin. +Retain its exact physical base and version for the whole comparison. Never resolve +it from the target repository, `.agents`, the original source checkout or an older +plugin cache, and never silently mix versions. A missing sibling is a concrete +installation prerequisite to resolve or report, not permission to search for a +different copy or invent a substitute runner. ## Handoff proposal -Explain what the owner will learn before listing run terms. Use one message -shaped like this synthetic example; replace every placeholder with checked facts: +Inspect the change first. When the goal is ambiguous, offer 2–4 distinct outcomes +tailored to it, with a recommended option and a short reason. Do not require +technical test design from the owner. Skip the options for an already clear goal. +Combine this clarification with the run proposal, not a separate questionnaire. + +For an invented PR adding archive checksum checks, the options could be: + +1. **New checks and existing safeguards (recommended):** does the agent verify + the archive while preserving recovery and approval before adoption? +2. **New checks:** does the agent detect a checksum mismatch and avoid accepting + a damaged archive? +3. **Existing safeguards:** do recovery and human approval still work after + adding checksum verification? + +Explain why the recommendation fits the actual change, and let the owner name +another concern. These example options are not a fixed menu for every PR. + +Then state run terms covering the recommended option. If all offered options +can share the same bounded terms, explicitly say so; otherwise identify the +differences before seeking approval. Do not promise unknown terms to avoid an +extra question. Shape the combined message like this synthetic example, replacing +every placeholder with checked facts: ```text -I recommend checking whether this change helps the agent -without . I'll compare the old and new instructions using -. This will tell us , but won't establish -. + +I recommend