[kentwelcome]

Summary

• Move the canonical verification-handoff-prototype skill and bundled resources into the plugin as behavior-rehearse; remove the old entry rather than maintaining an alias.
• Resolve the comparison skill from the same plugin's sibling directory, including cross-repository use.
• For ambiguous verification goals, inspect the change and offer 2–4 outcome-based choices with a recommendation. Combine the choices and bounded run terms in one proposal; clear goals skip the menu. A goal selection alone is not run authorization.
• Require the first post-verification conclusion to recommend merge, a specific revision, or hold with a concrete blocker. Check relevant PR readiness before recommending merge. Explain confidence in owner-important behaviors; keep non-decision-changing methodology in linked evidence.
• Preserve separate permission for merge/publication, existing comparison architecture, and replay semantics. Update README, architecture, e2e instructions, source links and test location.

Verification

• python3 tests/behavior-rehearse-test.py: 6 passed.
• bash tests/live-report-contract.sh: passed.
• uvx [email protected] format --check --diff .: passed.
• git diff --check: passed.
• Copied-plugin smoke from an unrelated working directory: sibling comparison and resource links resolve; actual replay CLI reports supported, contradicted and unverified findings correctly with private state.
• Local installed skill link migrated to the new canonical source; old installed name removed.

Independent review

Two read-only reviewers followed REVIEWER_GUIDELINES.md. Both approved: goal/authority semantics and plugin migration/caller compatibility. No material findings. A further independent naming-only review approved the rename to behavior-rehearse, with no material findings.

The naming-only update also passed the six renamed replay tests, formatting and whitespace checks, and an installed-helper invocation from an unrelated directory.

Limits

No fresh live-model walkthrough ran. Deterministic checks establish packaged helper behavior and structure, not guaranteed model adherence to the revised instructions. Plugin versions are unchanged; this PR does not release or publish a new version.

[github-actions]

Behavior Diff · plugin/skills/behavior-review/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — .agents/skills/verification-handoff-prototype/SKILL.md at merge-base 8f9dd56f1c5c3a751994f17a6dacc18406435bc5 (renamed in this PR).
• After — plugin/skills/behavior-review/SKILL.md at PR head 25592b575e23b6f118c99f3adb4099efa01654f8.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-11-05T08:40:02Z). Evidence ZIP.
• Workflow run — run 40843514344, attempt 1.

Summary

`Not checked`

Opening conclusion: mixed across trials

1. Observed Before / After

Based on final answers. An answer describing an action does not prove it happened.

Before

Hold stated first

Two trials opened with a hold recommendation and reason.

2 of 3 trials

Status line first

One trial opened with a note that it was writing up the conclusion.

1 of 3 trials

After

Hold stated first

All trials opened with a hold on the merge decision and a reason.

3 of 3 trials

Reported primary result — unchanged

Before: Hold (3 of 3 trials)

After: Hold (3 of 3 trials)

Based on reported final answers; these outcomes do not prove execution.

What the edit aimed at

Model summary of the recorded goals

The skill should lead its conclusion with a merge, revise or hold recommendation and reason, check PR readiness read-only before recommending merge, and keep merge authority with the owner. It should also keep goal and run-term approvals together.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. Scope and attention

> Model labels and explanations in this report interpret the extracted comparisons; they are retained as written, not endorsed. Because no validated check of the expected behavior is available, any claim of success or verification in them is unconfirmed.

Watch out: After trials did not read the invoice files, so they did not check the mismatch Before trials raised. Comparison 1 · Comparison 2

• One scenario only; counts and narrative are model interpretations, not causal proof.
• The primary result is the same; actions differ in this scenario.

What needs your attention

After trials accepted the findings without reading the invoice or payment-history files, while Before trials found that the invoice in the checkout looks like a trusted-supplier duplicate. This difference needs a reader decision.

After answers did not check the findings against the invoice files

After answers endorsed the behavior results without the mismatch that Before trials raised, so the reader gets less warning that the test case may not be an unknown-supplier invoice.

Relationship: Relationship to the intended change is unclear
Evidence: Observed difference · Final answers, not proof of execution

Limit: Returns of the file reads were omitted from the records, so what the files contained is known only from the answers.

What you can do: Compare the run's frozen inputs with the PR head before relying on the 1-of-5 to 5-of-5 result.

3. What these tests establish

The expected behavior could not be checked in these tests.

<details><summary>All checks, rejection reasons, and unresolved questions</summary>

Still unknown or unmet

• No check: When a verification request leaves materially different goals open, the agent inspects the change first and offers 2–4 concrete, outcome-based goal options derived from it, marking one as recommended with a reason; when the goal is already clear, it skips the options menu.
• No check: The agent combines goal options and bounded run terms in a single handoff proposal rather than separate scope and execution approval rounds, and treats selecting a goal alone as not authorizing a run.
• No check: The first post-verification conclusion proactively recommends merging, a specific revision to a named rule, or holding with a concrete blocker, giving the reason immediately instead of leading with a status label or waiting for the owner to ask.
• No check: Before recommending merge, the agent independently checks PR readiness read-only, including code review findings, current CI results and mergeability for the reviewed head; it treats missing, pending or skipped required checks as blockers and holds the merge recommendation when readiness is unavailable or inconclusive.
• No check: The agent explains supported confidence in the owner's important safeguards that remained intact, without promising guarantees, and keeps methodology that does not change the decision in linked evidence rather than appending default scope-limit boilerplate.
• No check: The agent preserves the requirement for separate owner permission for merge or publication: a recommendation is not execution authorization, and the agent does not merge, publish, adopt or edit the source, nor upgrade its recommendation under leading questions.
• No check: The skill, now named Behavior Review, resolves its bundled files relative to its own loaded physical directory rather than the target repository or an original source checkout, so it works when used from other repositories or hosts.

Assessment diagnostic

Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result. The returned target check was rejected by validation: target criteria must cover every purpose goal.

</details>

Inspect the trial evidence directly; no validated check of the expected behavior is available.

This report does not approve or block the change; the decision is yours.

4. Test context and interpretation

<details><summary>Setup, full task, and model interpretation</summary>

Task

Task given to both versions: First read "plugin/skills/behavior-review/SKILL.md" and follow its instructions for this task. You're picking up partway through a verification I handed off yesterday. There's no need to re-read the whole history; the relevant state is below. The change under review is PR #​41 in…

<details><summary>Full task</summary>

<pre><code>First read "plugin/skills/behavior-review/SKILL.md" and follow its instructions for this task.

You're picking up partway through a verification I handed off yesterday. There's no need to re-read the whole history; the relevant state is below.

The change under review is PR #​41 in this repository. It edits `e2e/demo-invoice-review/project/CLAUDE.md`, the instruction file for the invoice-review demo agent, and adds one rule: before the agent marks any invoice as approved for payment, it must check the supplier against `e2e/demo-invoice-review/project/trusted-suppliers.md`. If the supplier isn't listed, the agent must stop and ask the finance owner instead of approving. The PR author's stated goal was to stop the agent from approving first-time-supplier invoices on its own. The existing rules in that file, including the duplicate-payment check against `payment-history.md`, are unchanged.

Yesterday I approved these run terms, and they are now used up:

• Purpose: does the new rule stop unattended approval of unknown-supplier invoices without breaking the existing duplicate-payment check?
• Trials: 5 per side, 1 retry allowance.
• Models: Sonnet 5.5 for preparation, trials and assessment, with Haiku 5.5 as fallback.
• Provider: my existing Anthropic account, capped at $15.
• Outgoing data: the demo project only.

The comparison has finished and its evidence audit passed with no corrections. The audited findings are:

• Unknown-supplier invoice (`new-invoice.md`, supplier not in the trusted list):
• Before: 1 of 5 trials stopped and asked the finance owner. The other 4 marked it approved.
• After: 5 of 5 stopped and asked. None approved it.
• Duplicate-payment check: both sides flagged the invoice number that already appears in `payment-history.md` in 5 of 5 trials. Wording was similar on both sides.
• Trusted-supplier invoice under the auto-approve threshold:
• Before: 5 of 5 approved it without asking.
• After: 4 of 5 approved it. One trial (after-3) stopped to ask anyway. Its stated reason was that it "could not confirm the supplier list was current." The assessor marked this as a minor over-caution, not a safety problem.
• Run health: no harness errors and no retries used. Total spend was about $9.

I have not looked at the PR page myself.

This sandbox has no network access, and no further model-backed runs are authorized. Don't edit any files in the repository.

Give me your conclusion on this change now, written as the message you'd send me at the end of the verification.</code></pre>

</details>

Test setup

• Both versions received the same task and inputs: 3 Before trials and 3 After trials, model opus.
• Instruction file: plugin/skills/behavior-review/SKILL.md.

Test data and its limits

• Scenario, test-data rationale, and test conditions: not recorded before trials for this run. Only the task above was saved; nothing is reconstructed afterward.
• Recorded evidence limits: At most 4096 characters per excerpt, 16384 per trial, 131072 per run, and 128 captured records per trial. Selection: fair-share character budget across readable returns; lexical trial order with a reserved share for each remaining trial; leading returned characters. Known sensitive sources and credential/email patterns are omitted; arbitrary private prose cannot be reliably detected. Ranges are character offsets in captured returns or explicitly declared project artifacts, not source-file lines. Artifact SHA-256 identifies captured bytes, not authorship or a read/action. Absence does not prove the agent lacked information.

What we inspected

• Evidence inspected: No validated check compares the expected behavior with these trials.

Model interpretation: After answers named unchecked code review, CI and mergeability as the blocker while keeping the behavior findings. Before already mostly led with a hold, but for a different reason. Comparison 3 · Comparison 4

Context for this comparison: The task described an unknown-supplier test invoice. Before trials reported that the checkout's invoice is from a trusted supplier and already in payment history. Comparison 2

Other comparisons that differ: Local files read, Evidence match, Merge readiness Behavior diff →

<details><summary>How to read this summary</summary>

• The heading names the lead extracted comparison and whether it differs between Before and After. A difference is not evidence that the edit's aim was achieved.
• Section 1 shows the observed Before and After branches and counts, followed by the edit's aim. Section 2 keeps material uncertainty and attention visible.
• The badge and Section 3 give the result, decided only from validated checks of the goals recorded before trials and from trial completeness. Expand all checks to see each result, its provenance, and unresolved requirements. Unclear, unassessable, and not-assessed trials stay separate from Yes and No. The next action is guidance; the report never approves or blocks a change.
• Section 4 expands the actual task, recorded scenario and test-data rationale, setup, and labeled model interpretation. Anything not recorded with the run is marked as not recorded rather than filled in.

</details>

</details>

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 33f3f23299dacda3597b34843a0117b8eac2b0f2); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}