From 6fbf63a11d1d706750b7ffc94d668fc76b147672 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 29 Oct 2026 00:01:59 +0800 Subject: [PATCH 1/2] Fix verification handoff routing and owner approval flow Signed-off-by: Kent Huang --- .../verification-handoff-prototype/SKILL.md | 104 +++++++++++------ .../references/execution.md | 109 ++++++++++++++---- docs/architecture.md | 24 ++-- 3 files changed, 165 insertions(+), 72 deletions(-) diff --git a/.agents/skills/verification-handoff-prototype/SKILL.md b/.agents/skills/verification-handoff-prototype/SKILL.md index a838b50..fbfa0f9 100644 --- a/.agents/skills/verification-handoff-prototype/SKILL.md +++ b/.agents/skills/verification-handoff-prototype/SKILL.md @@ -1,6 +1,6 @@ --- name: verification-handoff-prototype -description: Carries an instruction change from owner-confirmed goals through review and authorized Behavior Diff verification to a decision-ready recommendation. Use when the owner wants to hand off verification of a change or PR without coordinating the agent's next steps. +description: Carries an instruction change from one owner-approved handoff proposal through review and Behavior Diff verification to a decision-ready recommendation. Use when the owner wants to hand off verification of a change or PR without coordinating the agent's next steps. --- # Verification handoff prototype @@ -13,16 +13,26 @@ Resolve bundled files relative to this loaded skill, never the target repository ## Ownership and authority +- The owner has three touchpoints: the request, one handoff proposal, and the + adoption decision. Skip the proposal when the request already confirms the + question and authorizes every run term. Any other owner input is an exception. - Own discovery, planning, tool selection, preflight, execution, bounded recovery and interpretation. Never ask the owner to select evidence IDs, design tests, - debug the harness or remind you to take the next agreed step. -- Ask only for material goal corrections, unavailable domain facts, or missing - authority. Each interruption states the concrete decision, recommendation and - consequence. Continue useful independent work while awaiting that answer. -- Before paid trials, obtain approval covering purpose, trial count, model/provider, - cost estimate or explicit budget, and outgoing data. Reuse approval within that - scope; changes to scope or budget need new approval. Editing this skill is not - run approval. Do not send private evidence to a new provider without consent. + debug the harness, choose a fallback model, ask what a report means or remind + you to take the next agreed step. +- Ask outside the proposal only for material goal corrections, unavailable domain + facts, or authority beyond approved terms. Each interruption states the concrete + decision, recommendation and consequence. Never end a turn by asking what to do + next or by giving the owner a phrase to type. +- Model-backed preparation, trials and assessment need approval of these terms: + purpose, trials per side, retry allowance, preparation/trial/assessment models + with named fallbacks, provider and account, cost estimate or quota boundary, and + outgoing data. An approval given before the terms were stated covers only the + terms the owner named. Put the remaining terms in the proposal and wait for + approval before any model-backed step. Within approved terms, continue without + further prompts. Switch only to a named fallback; changes outside the terms need + new approval. Editing this skill is not run approval. Do not send private + evidence to a new provider without consent. - No source edits, global configuration writes, publication, adoption or merge are implied. Accepting findings or answering a study question records a decision only. A separate explicit operational request must name the action and target; @@ -33,45 +43,63 @@ Resolve bundled files relative to this loaded skill, never the target repository ## Workflow -1. **Understand the change.** Inspect the selected diff, relevant unchanged rules - and available discussion. Pin revisions and source paths; prepare historical - inputs through Behavior Diff's supported provenance and isolation procedure. - Do not require the owner to assemble an evidence packet. -2. **Confirm the question once.** Propose concise intended outcomes, preserved - requirements and bounded scope with explicit/inferred sources. Distinguish - added rule value from reliability in the owner's current environment. Reuse - prior confirmation; ask only about material uncertainty. Keep study questions - out of this gate. Goal corrections require reassessing affected evidence. +1. **Understand the change and route it without model calls.** Load the + available `behavior-diff` skill. Inspect the selected diff, relevant unchanged + rules and available discussion. Pin revisions and source paths. Then settle the + route with deterministic checks: fixture fit, file mapping, host + authentication, model availability and isolation, as + [execution handoff](references/execution.md) describes. Do not require the + owner to assemble an evidence packet or confirm a route the repository answers. +2. **Send one handoff proposal.** In one message, propose the intended outcomes, + preserved requirements, tested scope and route, plus the run terms above, with + explicit/inferred sources and a recommendation. Distinguish added rule value + from reliability in the owner's current environment. Keep study questions out. + Reuse confirmation and authorization already given. A goal correction requires + reassessing affected evidence and resending only the changed terms. 3. **Review, then prepare verification.** Inspect instructions and saved evidence. Review completion is not verification completion. For changed agent behavior, - proceed directly to preparing a Behavior Diff comparison; do not wait for - “now run Behavior Diff.” Existing evidence can satisfy the question only when - revisions, conditions and coverage match; explain that conclusion. Report a - material ambiguity without editing the source or silently testing a repaired - version. Preserve the agreed question unless the owner explicitly changes it. -4. **Preflight before spending.** Load the available `behavior-diff` skill and - follow [execution handoff](references/execution.md). Own fixture preparation, - host compatibility, scenario reach and confound checks. Present any missing - run authorization as one concrete bounded proposal, not an open-ended menu. -5. **Run and assess.** Once preflight and authorization are satisfied, launch and - follow through to raw actions, returns and artifacts. Separate setup failures, - missed cases and unavailable assessment from behavioral failures. Recover - within the approved boundary; never rerun merely to obtain a difference. -6. **Return a decision-ready result.** Lead with retain/revise/hold and why. State - what changed versus Before, what stayed intact, what remains unknown, and - whether additional instructions showed added value. Link inspectable evidence + proceed directly to preparing the Behavior Diff comparison. Existing evidence + can satisfy the question only when revisions, conditions and coverage match; + explain that conclusion. Report a material ambiguity without editing the source + or silently testing a repaired version. Keep the agreed question. +4. **Preflight before spending.** Follow the execution handoff. Own fixture + preparation, host compatibility, isolation, scenario reach and confound checks. + A blocker with an in-scope route is work to do, not a stopping point. Ask again + only when preflight needs terms outside the approval. +5. **Run and assess.** After approval, launch and follow through to raw actions, + returns and artifacts. Monitor status after each launch and handle failed runs + as the execution handoff describes. Separate setup failures, missed cases and + unavailable assessment from behavioral failures. Never rerun merely to obtain + a difference. +6. **Return a decision-ready result.** Lead with retain, revise or hold and why. + State what changed versus Before, what stayed intact, what remains unknown, and + whether additional instructions showed added value. Disclose how files were + mapped and any generated-report claim you corrected. Link inspectable evidence after the conclusion. Equal outcomes mean no observed added benefit under these conditions, not proof that a rule is universally unnecessary. Simulated behavior does not prove real-service reliability. Do not wait for the owner - to ask what the report means. Do not merge. + to ask what the report means. + +## Adoption questions + +Merge is the owner's decision, not a verification outcome. When asked whether to +merge, restate the retain/revise/hold result and name the merge-readiness checks +that did and did not run, such as code review, CI and mergeability. A summary of +intended changes is not a code review. Do not upgrade the recommendation under a +leading question. Do not merge. + +When the owner asks whether the journey needed them, answer from the session: +list each owner input and whether it was necessary, without agreeing to a framing +the record contradicts. ## Completion and blockers Finish with the bounded question answered, or an evidence-backed blocker naming what failed, what was tried, what remains unverified and the smallest recommended -resolution. Missing capability is not permission to fake a worker or shrink scope. -An intermediate review, helper result or successful process exit is not completion. -Keep the exact run handle and private continuation context for interrupted work. +resolution. Stop only when no in-scope route exists. Missing capability is not +permission to fake a worker or shrink scope. An intermediate review, helper +result or successful process exit is not completion. Keep the exact run handle +and private continuation context for interrupted work. After delivery, optional feedback may ask what coordination the owner still had to do. Do not require an interview to finish. Distinguish necessary owner choices diff --git a/.agents/skills/verification-handoff-prototype/references/execution.md b/.agents/skills/verification-handoff-prototype/references/execution.md index 3b13615..fcf1a2b 100644 --- a/.agents/skills/verification-handoff-prototype/references/execution.md +++ b/.agents/skills/verification-handoff-prototype/references/execution.md @@ -7,6 +7,29 @@ source; elsewhere use the discovered installed skill. Never silently mix version Missing tooling is a concrete prerequisite to resolve or report, not permission to invent a substitute runner. +## Handoff proposal + +Settle the route before writing the proposal, so it does not stop on a fixture +question the repository already answers. Use one message shaped like this +synthetic example: + +```text +Question: does the change make the agent while keeping +? Route: ordinary-agent Before/After comparison of + against a local simulated . Not tested: . +Run: 3 Before + 3 After trials, at most one startup retry. Models: +preparation and trials , or if is rate limited +or unavailable; assessment , no fallback approved. +Provider/account: . Cost/quota: . +Outgoing data: pinned instruction files and fixture data to ; no new +provider. +Recommendation: approve as proposed. +``` + +Name a fallback for each model role, or state that none is approved. A budget +alone never authorizes a different model. + ## Prepare a meaningful comparison - Follow Behavior Diff's historical source isolation, purpose derivation, scenario @@ -27,17 +50,25 @@ to invent a substitute runner. ## Route by the behavior that must execute Distinguish an application's runbook decisions from actual workflow-engine state -or inter-agent handoffs. A mention of a worker or gate is not evidence that a Go -source project is needed. When Behavior Diff directs you to its Spacedock -reference, inspect the rule and fixture prerequisites before invoking its builder. -Do not apply a source-repository fixture to an unrelated application repository. - -If the loaded skill requires a fixture that cannot represent the requested -behavior, report that specific contract/capability mismatch with a recommended -resolution. Do not silently ignore the downstream contract or replace a real -worker handoff with a scripted reply. A general decision scenario is valid only -when it answers the agreed question; narrowing away a real handoff requires owner -approval. Do not ask the owner to diagnose the mismatch. +or inter-agent handoffs. Decide from the target repository, not from words in its +instructions: a workflow engine is under test only when the repository contains +its source, configuration or state, or the owner says it runs one. A mention of a +worker, gate or Spacedock is not evidence that a Go project is needed. When +Behavior Diff directs you to its Spacedock reference, inspect the rule and fixture +prerequisites before invoking its builder. Do not apply a source-repository +fixture to an unrelated application repository. + +If the repository does not run the engine, the ordinary-agent decision comparison +is the correct route, not a narrowed scope; propose it without first reporting a +blocker. If the repository does run it and the loaded skill's fixture cannot +represent the requested behavior, report that contract/capability mismatch with a +recommended resolution. Do not replace a real worker handoff with a scripted +reply; narrowing away a real handoff requires owner approval. Do not ask the owner +to diagnose the mismatch. + +A multi-file change must keep its file mapping. If the runner accepts only one +target, a combined file is a disclosed test construction: state it in the +proposal and result, and do not present it as the repository layout. ## Check compatibility and confounds before launch @@ -57,6 +88,18 @@ boundary and verify it with harmless probes. Disabling injection alone is not filesystem isolation. Any model-backed probe belongs in the approved run budget. Do not claim complete isolation if a relevant channel cannot be checked. +Run the fresh historical preparer as a host subprocess with its own isolated +context. Use an in-session subagent only when the host verifiably gives it no +injected session memory. If Behavior Diff provides no isolation for the host, use +a disposable launcher in the private run root, for example a wrapper first on +`PATH` that applies process-local flags disabling memory and plugins and denies +reads of memory stores and the original checkout. Prefer the host's existing +sign-in. If a private host home is unavoidable, disclose it in the proposal, keep +any credential copy mode 0600 inside the mode-0700 run root, never print it, and +delete it when the run ends or is interrupted. Verify the launcher with harmless +non-model probes before model use, keep it with the evidence, and record it in +the result. Never change the owner's global host settings instead. + Follow Behavior Diff's preflight and frozen-handoff helpers rather than rebuilding their validators. Simulation checks prove the fixture's mechanics, not that a real website or workflow engine works. Run a real-service check only with its @@ -64,19 +107,32 @@ specific authorization; do not replace the agreed question with a simulation. ## Authorization, execution and recovery -This handoff requires the bounded live-run approval in SKILL.md before entering -Behavior Diff's automatic-launch step. Do not treat its normal no-extra-prompt -rule as overriding this boundary. Once approved, carry execution through without -another stage-by-stage permission loop. A study answer or “accept limitations” -is never run or merge authorization. +Owner approval of the handoff proposal, or a request that already authorizes +every term, supplies the bounded run approval required before Behavior Diff's +automatic-launch step. Sending the proposal is not approval. Do not treat Behavior +Diff's normal no-extra-prompt rule as overriding this boundary. Once approved, +carry execution through without another stage-by-stage permission loop. A study +answer or “accept limitations” is never run or merge authorization. Keep the printed run handle, frozen inputs, exact host/model settings and failed -attempts. Diagnose tooling failures from their outputs. Repair routine local -setup within the authorization, without editing the target change or global -settings. Retry only when the approved budget includes it and the cause is -understood; otherwise propose one bounded recovery. Do not retune after a null -result. A changed scenario or isolation condition is a new comparison, not a -replacement result. Stop repeated unsuccessful recovery with a concrete blocker. +attempts. In the launch message, say that the runner opens its report by itself +when the run ends. Check status right after launch and keep monitoring while the +run or any trial is live or uncertain (`pending`, `preparing`, `running`, +`normalizing`, `uncertain`); zero completed trials in those states is normal, not +a failure. When the run ends or its runner is lost (`owner_lost`) with any trial +not `finished`, tell the owner at once, even if the report is already open, and +name each failed phase or trial state that status reports. Say that no model ran +only when startup errors and empty native traces show it; a failure during +normalization can follow model output. Preserve that run, diagnose its +outputs, and relaunch only within the retry allowance. Repair routine local setup +without editing the target change or global settings. On rate limiting or model +unavailability, retry once, then switch to the named fallback and record the +switch; never idle on repeated failures. A fallback switch relaunches the whole +comparison on one model, does not consume the startup retry, and is a new +comparison rather than a mix of models. Without a named fallback, send one +bounded proposal. Do not retune after a null result. A changed scenario or +isolation condition is a new comparison, not a replacement result. +Stop repeated unsuccessful recovery with a concrete blocker. ## Interpretation and return @@ -84,13 +140,16 @@ Audit raw actions, returns and artifacts for each case, including minority resul If automated assessment is rejected, retain the report and distinguish a documented raw-evidence assessment from the unavailable automated result. Never re-label it as an automatic pass. Distinguish rule effects from pre-existing instructions, -shared memory, task hints and environment failures. +shared memory, task hints and environment failures. Check the generated headline +against raw decisions; a wording difference is not a changed outcome. Correct it +in your assessment without editing the report. Deliver the recommendation without requiring “what does the report mean?”: what question was tested; added benefit observed or not demonstrated; regressions and preserved behavior; decisive evidence; unresolved scope and its decision impact. A process change without an outcome change is not automatically an improvement. -Recommend retain/revise/hold at the tested scope. Do not infer universal redundancy -from a few equal trials or promise the next real run will succeed. Finish the +Recommend retain/revise/hold at the tested scope; never merge, and do not turn +that result into a merge recommendation. Do not infer universal redundancy from +a few equal trials or promise the next real run will succeed. Finish the verification handoff before offering optional process feedback; never operate the PR in response to that feedback. diff --git a/docs/architecture.md b/docs/architecture.md index f962ce6..ac61582 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -981,20 +981,26 @@ owns the verification handoff, separate from the installed plugin: ```text Selected change + project context - -> host proposes goals -> owner confirms/corrects material scope + -> no-model route checks (fixture fit, file mapping, host, models, isolation) + -> one handoff proposal: goals + route + run terms with named model fallbacks + (skipped when the request already authorizes them) -> instruction and saved-evidence review -> Behavior Diff preparation + compatibility/isolation preflight - -> bounded live-run approval if missing -> execute + audit evidence - -> decision-ready recommendation -> owner adoption decision (no automatic merge) + -> execute, check status, recover within terms -> audit evidence + -> retain/revise/hold recommendation -> owner adoption decision (no merge) ``` +The owner's touchpoints are the request, one proposal and the adoption decision. The host owns planning, preparation, bounded recovery and interpretation, without -requiring owner reminders between stages. It delegates comparison mechanics to -Behavior Diff's existing contracts, not a second runner. It checks fixture fit, -scenario reach and environment confounds before launch; unsupported capabilities -remain blockers rather than silently reduced scope. Paid runs and private-data -handoffs need explicit bounded authority. Global configuration is not a write target. -Evaluation responses never authorize PR operations. +requiring owner reminders between stages. It decides whether a workflow engine is +under test from the target repository, not instruction wording. It normally runs +the fresh preparer as an isolated host subprocess, using a disposable private +launcher when Behavior Diff has no host isolation. It delegates comparison +mechanics to Behavior Diff's existing contracts, not a second runner. Unsupported +capabilities remain blockers rather than silently reduced scope. Paid runs and +private-data handoffs need explicit bounded authority; model switches are limited +to named fallbacks. Global configuration is not a write target. Evaluation +responses and merge questions never authorize PR operations. Explicit replay-only requests use saved evidence without new trials. Optional `replay.py` owns private snapshots, correction invalidation, confirmed revisions From 95c72fb8484a7d361a66a443ef97dbaf9d8ff44f Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 29 Oct 2026 00:04:53 +0800 Subject: [PATCH 2/2] Align both Behavior Diff skills with behavior-based fixture routing Signed-off-by: Kent Huang --- plugin/skills/behavior-diff-live/SKILL.md | 12 ++++++++---- plugin/skills/behavior-diff/SKILL.md | 12 ++++++++---- .../skills/behavior-diff/references/spacedock-duo.md | 10 ++++++---- 3 files changed, 22 insertions(+), 12 deletions(-) diff --git a/plugin/skills/behavior-diff-live/SKILL.md b/plugin/skills/behavior-diff-live/SKILL.md index 4a3d774..8ad487c 100644 --- a/plugin/skills/behavior-diff-live/SKILL.md +++ b/plugin/skills/behavior-diff-live/SKILL.md @@ -18,11 +18,15 @@ one sample. Report what happened, never say "consistently", and treat the decision diff as a sketch until the headless 3+3 confirms it. -**Spacedock workflow rule?** If the changed file is a spacedock workflow -doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO / -ensign / gate), use Spacedock fixtures. +**Spacedock behavior under test?** Spacedock fixtures are isolated before/after test repos. The real Spacedock -binary creates their workflow state. Do not create this state by editing files. +binary creates their workflow state; do not create this state by editing files. +Use these fixtures only when the requested comparison exercises actual Spacedock +role decisions, engine state or worker handoffs, supported by repository +configuration/state or an explicit user requirement. Mentions of Spacedock, FO, +ensign, gate or dispatch alone are not sufficient. Ordinary application-runbook +decisions use ordinary-agent scenarios without requiring Spacedock or Go. +Never simulate away a real worker handoff. Before designing the run, read `references/spacedock-duo.md` in the sibling `behavior-diff` skill's directory (both skills install together). It chooses the single-role or diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index e67042c..3587bbd 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -59,11 +59,15 @@ agent. Your job is to prepare `--file` and `--task` well. Runs land under `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/`. -**Spacedock workflow rule?** If the changed file is a spacedock workflow -doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO / -ensign / gate), use Spacedock fixtures. +**Spacedock behavior under test?** Spacedock fixtures are isolated before/after test repos. The real Spacedock -binary creates their workflow state. Do not create this state by editing files. +binary creates their workflow state; do not create this state by editing files. +Use these fixtures only when the requested comparison exercises actual Spacedock +role decisions, engine state or worker handoffs, supported by repository +configuration/state or an explicit user requirement. Mentions of Spacedock, FO, +ensign, gate or dispatch alone are not sufficient. Ordinary application-runbook +decisions use ordinary-agent scenarios without requiring Spacedock or Go. +Never simulate away a real worker handoff. Before designing the run, read `references/spacedock-duo.md` inside this skill's base directory. It chooses the single-role or two-agent path. Create the fixtures with diff --git a/plugin/skills/behavior-diff/references/spacedock-duo.md b/plugin/skills/behavior-diff/references/spacedock-duo.md index 9c0b1da..3367864 100644 --- a/plugin/skills/behavior-diff/references/spacedock-duo.md +++ b/plugin/skills/behavior-diff/references/spacedock-duo.md @@ -1,9 +1,11 @@ # Spacedock workflow rules — shared module for the behavior-diff skills -Load this whenever the change under test is a spacedock workflow rule: -the changed file is a workflow doc (e.g. `docs/dev/README.md`) in a repo -containing `cmd/spacedock`, or the user says spacedock / FO / first -officer / ensign / gate / dispatch. +Load this when the requested comparison exercises actual Spacedock role +decisions, engine state or worker handoffs, supported by repository +configuration/state or an explicit user requirement. Mentions of Spacedock, +FO, first officer, ensign, gate or dispatch alone are not sufficient. +Ordinary application-runbook decisions use ordinary-agent scenarios and do +not require Spacedock or Go. Never simulate away a real worker handoff. ## 1. Decide the shape: single-role or handoff