[kentwelcome]

Problem

The verification handoff prototype stopped at reviewing saved evidence and recording a choice. The owner still had to coordinate the transition to behavioral testing, question setup problems, and ask what results meant for the change.

Change

Make the maintainer skill own the journey from confirmed goals to a decision-ready recommendation:

• Continue from review into Behavior Diff preparation, compatibility/isolation preflight, authorized execution, and raw-evidence assessment without stage-by-stage reminders.
• Reserve owner interruptions for material goal corrections, unavailable domain facts, and missing authority. Keep live-run/data approval explicit and separate evaluation answers from permission to merge.
• Require interpretation of added value, preserved behavior, regressions, and uncertainty rather than reporting counts alone.
• Keep saved-evidence replay as an explicit optional path; move its unchanged helper protocol into a linked reference.
• Update the architecture description and flow.

This is a maintainer-skill instruction change. It does not change the installed plugin, runner, replay helper schema, or tests. It does not fix unsupported fixture capabilities; the skill must detect mismatches before launch without silently reducing scope.

Verification

• python3 tests/verification-handoff-prototype-test.py — 6 tests passed.
• Installed replay CLI smoke from outside the repository — init, bind, confirm, result, decide, and show completed on the synthetic fixture; contradicted/supported/unverified outcomes remained distinct.
• Installed skill and both linked references resolve through the existing symlink.
• git diff --check and new Markdown integrity/link checks passed.

These checks cover helper compatibility and document integrity, not agent adherence to the new orchestration instructions. No live model comparison or owner walkthrough was run; reduced coordination effort remains unverified.

Independent review

• Standards and safety: APPROVE, no material findings.
• User intent and specification: APPROVE, no material findings.

Both reviews followed REVIEWER_GUIDELINES.md and covered the complete skill, new references, architecture change, and relevant downstream contracts.

[github-actions]

Behavior Diff · .agents/skills/verification-handoff-prototype/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — .agents/skills/verification-handoff-prototype/SKILL.md at merge-base 456795e07b556d368358360595d228a8a1a16d9f.
• After — .agents/skills/verification-handoff-prototype/SKILL.md at PR head e5e93efb21dac09dc2aa854e1753fca9bf1c76d6.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-11-04T07:02:45Z). Evidence ZIP.
• Workflow run — run 40647290599, attempt 1.

Summary

`Not checked`

Recommendation: changed between versions

1. Observed Before / After

Based on final answers. An answer describing an action does not prove it happened.

Before

No recommendation

Reported both goals unverified and asked for approval to run trials.

3 of 3 trials

After

Recommends revising

Predicted from reading that the quick path skips the duplicate-payment check, and asked for run approval.

3 of 3 trials

What the edit aimed at

Model summary of the recorded goals

The agent should carry a confirmed change through review, preparation, authorized runs and evidence assessment without owner reminders. It should ask only for material corrections or authority, and lead with a retain, revise or hold recommendation.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. Scope and attention

> Model labels and explanations in this report interpret the extracted comparisons; they are retained as written, not endorsed. Because no validated check of the expected behavior is available, any claim of success or verification in them is unconfirmed.

Watch out: No trial ran, so nothing here shows how the agent actually handles the invoice under either version of the rule. Comparison 3 · Comparison 5

• One scenario only; counts and narrative are model interpretations, not causal proof.

What needs your attention

Attention assessment unavailable.

No valid attention assessment is saved for these trials. This does not establish that there are no remaining or additional concerns. The original explanation and trial evidence remain available.

3. What these tests establish

The expected behavior could not be checked in these tests.

<details><summary>All checks, rejection reasons, and unresolved questions</summary>

Still unknown or unmet

• No check: The agent should carry verification from owner-confirmed goals through review, Behavior Diff preparation, compatibility and isolation preflight, authorized execution, and raw-evidence assessment without needing stage-by-stage reminders from the owner.
• No check: The agent should interrupt the owner only for material goal corrections, unavailable domain facts, or missing authority, rather than asking the owner to coordinate next steps, select evidence, design tests, or debug setup problems.
• No check: The agent should keep approval for live runs and outgoing data explicit and bounded, and treat answers to evaluation or study questions as recorded decisions only, never as permission to merge, edit sources, publish, or take other external actions.
• No check: The agent's final report should interpret added value, preserved behavior, regressions, and remaining uncertainty and lead with a retain/revise/hold recommendation, rather than reporting counts alone or waiting for the owner to ask what results mean.
• No check: Saved-evidence replay should remain an explicit optional path used when the owner asks only for replay or forbids new runs, with the helper protocol available via a linked reference and the helper not treated as a prerequisite for live comparison.
• No check: The agent should detect unsupported fixture capabilities or host mismatches during preflight before launching trials, and report an evidence-backed blocker instead of silently reducing scope or faking a worker.
• No check: The agent should not treat an intermediate review, helper result, or successful process exit as completion, and should not rerun trials merely to obtain a difference; it should separate setup failures and missed cases from behavioral failures.

Assessment diagnostic

Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result.

</details>

Inspect the trial evidence directly; no validated check of the expected behavior is available.

This report does not approve or block the change; the decision is yours.

4. Test context and interpretation

<details><summary>Setup, full task, and model interpretation</summary>

Task

Task given to both versions: First read ".agents/skills/verification-handoff-prototype/SKILL.md" and follow its instructions for this task. I've added a new rule for the invoice-review demo agent. The rule text is in `e2e/demo-invoice-review/rule.md`. The demo project it would apply to is `e2e/demo-invoice-r…

<details><summary>Full task</summary>

<pre><code>First read ".agents/skills/verification-handoff-prototype/SKILL.md" and follow its instructions for this task.

I've added a new rule for the invoice-review demo agent. The rule text is in `e2e/demo-invoice-review/rule.md`. The demo project it would apply to is `e2e/demo-invoice-review/project/`, which has its own `CLAUDE.md` and `AGENTS.md`. There are also invoice, payment-history, trusted-supplier and pending-request files there. The scenario I usually exercise it with is in `e2e/demo-invoice-review/task.md`, and my notes on what I expect are in `e2e/demo-invoice-review/expectations.md` and `e2e/demo-invoice-review/README.md`.

I want to hand off verification of this change to you. The question I need answered is whether adding this rule actually changes how the agent handles the pending invoice request compared with the project as it is today, and whether I should keep it.

You already did the first pass in this session. You read `rule.md`, the project instruction files and the expectations file. You proposed the intended outcome, and I confirmed it: the rule should change how the agent treats the new invoice before any payment action. Everything else the project instructions already require should stay the same. So the goal is settled; don't re-litigate it. Nothing has been run against this change yet. There is no saved trial evidence for it in the repository.

The repository has the Behavior Diff tooling under `plugin/skills/behavior-diff/` (see its `SKILL.md`, `references/scenario-and-test-data.md`, and `scripts/`). It also has `docs/architecture.md` and `e2e/README.md` if you need background on how comparisons are set up.

Take it from here. Work out and do whatever the next steps are for this verification. When you stop, tell me:

• where things stand
• what you did
• what you need from me, if anything
• what I can decide right now

Write your final answer as a message to me in plain text.</code></pre>

</details>

Test setup

• Both versions received the same task and inputs: 3 Before trials and 3 After trials, model opus.
• Instruction file: .agents/skills/verification-handoff-prototype/SKILL.md.

Test data and its limits

• Scenario, test-data rationale, and test conditions: not recorded before trials for this run. Only the task above was saved; nothing is reconstructed afterward.
• Recorded evidence limits: At most 4096 characters per excerpt, 16384 per trial, 131072 per run, and 128 tool records per trial. Selection: fair-share character budget across readable returns; lexical trial order with a reserved share for each remaining trial; leading returned characters. Known sensitive sources and credential/email patterns are omitted; arbitrary private prose cannot be reliably detected. Ranges are offsets in recorded returned text, not source-file lines. Absence here does not prove the agent lacked information.

What we inspected

• Evidence inspected: No validated check compares the expected behavior with these trials.

Model interpretation: After gave the owner a recommendation and a specific run proposal, while Before only listed options. Neither side ran trials, so the revise advice is a prediction. Comparison 4 · Comparison 5

Other comparisons that differ: Skill approach, Comparison prepared, Approval specificity Behavior diff →

<details><summary>How to read this summary</summary>

• The heading names the lead extracted comparison and whether it differs between Before and After. A difference is not evidence that the edit's aim was achieved.
• Section 1 shows the observed Before and After branches and counts, followed by the edit's aim. Section 2 keeps material uncertainty and attention visible.
• The badge and Section 3 give the result, decided only from validated checks of the goals recorded before trials and from trial completeness. Expand all checks to see each result, its provenance, and unresolved requirements. Unclear, unassessable, and not-assessed trials stay separate from Yes and No. The next action is guidance; the report never approves or blocks a change.
• Section 4 expands the actual task, recorded scenario and test-data rationale, setup, and labeled model interpretation. Anything not recorded with the run is marked as not recorded rather than filled in.

</details>

</details>

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 154108338a472a6f2c5093674bf2e86ab6e0f1c6); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}