From 26d7b92645423bea6c5273d6d0c30b757a5616e8 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 29 Oct 2026 08:56:35 +0800 Subject: [PATCH] Make verification handoff explain decisions and own bounded recovery Signed-off-by: Kent Huang --- .../verification-handoff-prototype/SKILL.md | 59 +++++++++---- .../references/execution.md | 83 ++++++++++++++----- docs/architecture.md | 14 +++- 3 files changed, 117 insertions(+), 39 deletions(-) diff --git a/.agents/skills/verification-handoff-prototype/SKILL.md b/.agents/skills/verification-handoff-prototype/SKILL.md index fbfa0f9..41d8550 100644 --- a/.agents/skills/verification-handoff-prototype/SKILL.md +++ b/.agents/skills/verification-handoff-prototype/SKILL.md @@ -41,6 +41,29 @@ Resolve bundled files relative to this loaded skill, never the target repository or unrelated studies unless explicitly requested. Do not persist raw owner responses by default. Familiarity and reused evidence must be disclosed. +## Explain the decision in ordinary language + +Write for the owner of the workflow, not the maintainer of the test tool. Apply +this to proposals, progress messages, blockers and the final answer. Lead with +what you recommend doing to the change and why, using the owner's task as the +subject. Explain what benefit was observed, what risk remains, and whether that +risk changes the recommendation. The owner must not need to open a report to +understand the decision. + +Use short, natural paragraphs. Translate retain/revise/hold into a concrete action: +"keep these instructions", "change this rule", or "wait for evidence about this +behavior". Name what is on hold; a failed test is not a reason by itself to reject +the PR. Distinguish intended benefits from demonstrated benefits. Do not turn +missing evidence into reassurance. + +Keep commands, trace formats, sandbox details, source mappings and diagnostic +history in linked evidence unless they change the owner's decision. Explain +technical limits by their practical consequence. Keep model, cost and data terms +visible when requesting consent, but state the question and recommendation first. +Progress messages should say what changed, whether the result is affected, and +what you are doing next; do not narrate every debugging step. Use the available +Humanizer guidance in embedded mode without installing it or changing evidence. + ## Workflow 1. **Understand the change and route it without model calls.** Load the @@ -71,14 +94,15 @@ Resolve bundled files relative to this loaded skill, never the target repository as the execution handoff describes. Separate setup failures, missed cases and unavailable assessment from behavioral failures. Never rerun merely to obtain a difference. -6. **Return a decision-ready result.** Lead with retain, revise or hold and why. - State what changed versus Before, what stayed intact, what remains unknown, and - whether additional instructions showed added value. Disclose how files were - mapped and any generated-report claim you corrected. Link inspectable evidence - after the conclusion. Equal outcomes mean no observed added benefit under - these conditions, not proof that a rule is universally unnecessary. Simulated - behavior does not prove real-service reliability. Do not wait for the owner - to ask what the report means. +6. **Return a decision-ready result.** Follow the explanation guidance above. + Recommend retain, revise or hold in ordinary language, naming the change and + the reason. Explain what improved versus Before, what stayed intact, and what + remains uncertain in terms of the owner's workflow. Say what adopting now + would leave unverified and whether to change the PR, wait, or leave it alone + while verification continues. Put file mappings and report corrections in the + linked evidence; bring any correction that changes the recommendation into + the answer. Equal results do not prove a rule unnecessary, and simulated + behavior does not prove real-service reliability. ## Adoption questions @@ -94,12 +118,19 @@ the record contradicts. ## Completion and blockers -Finish with the bounded question answered, or an evidence-backed blocker naming -what failed, what was tried, what remains unverified and the smallest recommended -resolution. Stop only when no in-scope route exists. Missing capability is not -permission to fake a worker or shrink scope. An intermediate review, helper -result or successful process exit is not completion. Keep the exact run handle -and private continuation context for interrupted work. +Finish with the bounded question answered, or an evidence-backed blocker after +the execution handoff's recovery steps. A failed attempt or exhausted model-call +allowance stops further paid execution, not safe non-model diagnosis or useful +independent review. Finish that work before returning. Respect an explicit +owner instruction to stop all work. + +Do not end with "fix the runner" as owner homework. Explain what the failure means +for the change, what remains unknown, what you tried, and the specific permission, +capability or unavailable fact needed next. When only new run authority is +missing, recommend a bounded check and explain what repair makes it worth trying. +Keep the run handle and private continuation context. Never bypass isolation, +invent evidence, silently narrow scope, or repeat unsuccessful attempts merely +to avoid returning a legitimate blocker. After delivery, optional feedback may ask what coordination the owner still had to do. Do not require an interview to finish. Distinguish necessary owner choices diff --git a/.agents/skills/verification-handoff-prototype/references/execution.md b/.agents/skills/verification-handoff-prototype/references/execution.md index fcf1a2b..57efd3b 100644 --- a/.agents/skills/verification-handoff-prototype/references/execution.md +++ b/.agents/skills/verification-handoff-prototype/references/execution.md @@ -9,22 +9,21 @@ to invent a substitute runner. ## Handoff proposal -Settle the route before writing the proposal, so it does not stop on a fixture -question the repository already answers. Use one message shaped like this -synthetic example: +Explain what the owner will learn before listing run terms. Use one message +shaped like this synthetic example; replace every placeholder with checked facts: ```text -Question: does the change make the agent while keeping -? Route: ordinary-agent Before/After comparison of - against a local simulated . Not tested: . -Run: 3 Before + 3 After trials, at most one startup retry. Models: -preparation and trials , or if is rate limited -or unavailable; assessment , no fallback approved. -Provider/account: . Cost/quota: . -Outgoing data: pinned instruction files and fixture data to ; no new -provider. -Recommendation: approve as proposed. +I recommend checking whether this change helps the agent +without . I'll compare the old and new instructions using +. This will tell us , but won't establish +. + +I'll run three attempts with each version, using , +with , and . +This uses with . It sends + to ; results stay . +Recovery is limited to . Do you approve this comparison? ``` Name a fallback for each model role, or state that none is approved. A budget @@ -70,6 +69,15 @@ A multi-file change must keep its file mapping. If the runner accepts only one target, a combined file is a disclosed test construction: state it in the proposal and result, and do not present it as the repository layout. +Choose capabilities from the question, not from a previous failed setup. For a +question about agent decisions, a disclosed local service simulation may be +sufficient; real image generation is not automatically a prerequisite for testing +whether the agent requests a cover and waits for approval. If the question includes +image quality, actual saving, live services or real worker continuation, preserve +those requirements. After scope is approved, changing the evidence method in a way +that loses a required claim needs owner approval. Complete independent checks +where possible without presenting partial coverage as the full comparison. + ## Check compatibility and confounds before launch Inspect effective subprocess settings, not just the interactive session's model. @@ -100,6 +108,18 @@ delete it when the run ends or is interrupted. Verify the launcher with harmless non-model probes before model use, keep it with the evidence, and record it in the result. Never change the owner's global host settings instead. +Before a paid probe, check the actual launcher path with non-model probes: +authentication availability, writing and reading in the trial workspace, and +denial of reads from excluded sources. A copy from the outer shell alone does not +verify the agent's execution path. Check for nested sandbox restrictions before +launch; do not simply disable protections to make the probe pass. An alternative +isolation configuration needs equivalent read/write boundaries verified first. +Check whether the host records the actions needed by the frozen criteria. +File existence can establish an artifact, not that the agent inspected it. +If the host cannot exercise an execution path without a model, mark that path +unverified and include its smallest necessary probe in the approval. Do not +claim that an outer-shell check proves the inner path works. + Follow Behavior Diff's preflight and frozen-handoff helpers rather than rebuilding their validators. Simulation checks prove the fixture's mechanics, not that a real website or workflow engine works. Run a real-service check only with its @@ -132,7 +152,25 @@ comparison on one model, does not consume the startup retry, and is a new comparison rather than a mix of models. Without a named fallback, send one bounded proposal. Do not retune after a null result. A changed scenario or isolation condition is a new comparison, not a replacement result. -Stop repeated unsuccessful recovery with a concrete blocker. +When an attempt fails, separate the affected capability from the rest of the +question. Inspect the failed phase and native evidence; continue safe non-model +diagnosis and independent review within the existing authority. Before another +paid attempt, make a concrete in-scope local repair or state a distinct diagnostic +hypothesis and what the probe would distinguish. In either case, require remaining +authorization and perform applicable non-model checks first. Preserve original +failures and record exactly what changed. Do not repeat an unchanged failed probe. + +A missing action record blocks claims about that action, not every possible +finding. Inspect supported recording and artifact sources without treating an +agent's final answer as proof of execution. If required evidence cannot be +captured, stop that comparison rather than spending the remaining trial budget +on unassessable runs. Report available findings as partial. + +Exhausted probe/retry authority does not require the owner to diagnose the setup. +Finish permitted non-model work, then request only the missing authority with +the repair, expected information and bounded cost. If no safe repair or supported +route exists, explain the remaining capability gap and its decision consequence. +Do not run indefinitely, weaken isolation, or change the approved question. ## Interpretation and return @@ -144,12 +182,11 @@ shared memory, task hints and environment failures. Check the generated headline against raw decisions; a wording difference is not a changed outcome. Correct it in your assessment without editing the report. -Deliver the recommendation without requiring “what does the report mean?”: -what question was tested; added benefit observed or not demonstrated; regressions -and preserved behavior; decisive evidence; unresolved scope and its decision impact. +Use SKILL.md's ordinary-language decision guidance for the owner-facing answer. +Keep the full goal-by-goal audit, source mapping, minority results, report +corrections and recovery record in linked private evidence. State any limitation +that affects adoption in the answer itself. Recommend retain/revise/hold at the +tested scope; never merge or turn that result into a merge recommendation. A process change without an outcome change is not automatically an improvement. -Recommend retain/revise/hold at the tested scope; never merge, and do not turn -that result into a merge recommendation. Do not infer universal redundancy from -a few equal trials or promise the next real run will succeed. Finish the -verification handoff before offering optional process feedback; never operate the -PR in response to that feedback. +Do not infer universal redundancy from equal trials or promise live reliability. +Finish the handoff before offering optional process feedback. diff --git a/docs/architecture.md b/docs/architecture.md index ac61582..a535f7d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -986,8 +986,8 @@ Selected change + project context (skipped when the request already authorizes them) -> instruction and saved-evidence review -> Behavior Diff preparation + compatibility/isolation preflight - -> execute, check status, recover within terms -> audit evidence - -> retain/revise/hold recommendation -> owner adoption decision (no merge) + -> execute -> diagnose failures + continue independent checks within authority + -> audit evidence -> plain-language recommendation + adoption risk (no merge) ``` The owner's touchpoints are the request, one proposal and the adoption decision. @@ -1002,6 +1002,16 @@ private-data handoffs need explicit bounded authority; model switches are limite to named fallbacks. Global configuration is not a write target. Evaluation responses and merge questions never authorize PR operations. +Failed attempts stop the affected execution, not safe non-model recovery or +independent review. The host verifies launcher boundaries before paid probes, +requires a repair or distinct hypothesis before retries, and keeps required +evidence gaps explicit. Exhausted run authority triggers a bounded proposal only +after permitted diagnosis is complete. Decision simulations do not automatically +require live service capabilities; the approved question controls that choice. +Owner-facing messages explain what to do with the change, why, and what remains +unverified. Technical diagnosis and detailed evidence stay in private links; +decision-relevant risks and consent terms remain visible. + Explicit replay-only requests use saved evidence without new trials. Optional `replay.py` owns private snapshots, correction invalidation, confirmed revisions and read-only root-scoped citations for that subset. Its fixed-evidence mode is