[kentwelcome]

Problem

An explicit committed-PR comparison could stop at git status: the skill required an uncommitted instruction edit before selecting inputs, then described historical support as not adding a committed-edit invocation mode. The no-edit boundary could also be read as prohibiting exact historical snapshots. The existing --before-file mechanics already support this comparison.

Change

• Resolve requested PR/Git revisions first, explicit Before/After files second, and local edits last. A clean checkout is valid for historical inputs.
• Pin revisions and document PR merge-base/head and ordinary commit parent/head defaults, with material ambiguities left for clarification.
• Permit byte-preserving historical copies in private scratch space while prohibiting source changes or invented instruction edits.
• Explain historical preparation, common project context, companion-file coverage, and real prerequisites in the existing reference; update the architecture flow.

No runner, CLI, manifest version, or approval policy changes. No private session content is included.

Verification

• python3 tests/shared-data-test.py: 29 tests run, 28 passed, 1 skipped (requires a case-insensitive filesystem).
• python3 tests/source-provenance-test.py: 9 passed.
• Historical-input CLI smoke: exact committed Before/After snapshots prepared and verified; unrelated source edits remained untouched and did not enter the stage. No model calls.
• git diff --check: passed.

These checks establish input preparation and existing mechanics. A live agent evaluation of the revised wording was not run; agent adherence is not claimed as proven.

Independent review

• Standards/safety: APPROVE, no material findings.
• Requirements/specification: APPROVE, no material findings.

Both read-only reviewers followed REVIEWER_GUIDELINES.md and checked the complete change against the underlying preparation and runner contracts.

[github-actions]

Behavior Diff · plugin/skills/behavior-diff/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — plugin/skills/behavior-diff/SKILL.md at merge-base 8a561f93514912b57f0d6edb6e66fb6b49ef55f4.
• After — plugin/skills/behavior-diff/SKILL.md at PR head b561e35b373d6f878f4c7bb515209123bdba01fa.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-11-04T07:37:05Z). Evidence ZIP.
• Workflow run — run 40650948950, attempt 1.

Summary

`Not checked`

Final outcome: unchanged between versions

1. Observed Before / After

Based on final answers. An answer describing an action does not prove it happened.

Before

Stopped, asked for input

Selected no Before version and asked for the original file or history.

3 of 3 trials

After

Stopped, asked for input

Selected no Before version and asked for the original file or history.

3 of 3 trials

What the edit aimed at

Model summary of the recorded goals

When asked to compare a PR or Git revisions, the agent should pin exact Before and After versions and proceed even with a clean working tree. It should not alter the checkout or the change under test, and should keep the local-edit flow.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. Scope and attention

> Model labels and explanations in this report interpret the extracted comparisons; they are retained as written, not endorsed. Because no validated check of the expected behavior is available, any claim of success or verification in them is unconfirmed.

Watch out: After trials read the historical reference but all still stopped; results say nothing about handling a real parent commit. Comparison 2 · Comparison 4

• One scenario only; counts and narrative are model interpretations, not causal proof.
• The primary result is the same; actions differ in this scenario.
• Matching observations in this scenario do not prove that the instruction edit has no effect.

What needs your attention

No additional change needing your decision was found in these trials.

No finding was selected for your attention in these trials. This is not a guarantee that the target is met or that real use has no additional concerns.

3. What these tests establish

The expected behavior could not be checked in these tests.

<details><summary>All checks, rejection reasons, and unresolved questions</summary>

Still unknown or unmet

• No check: When the user explicitly requests a comparison of a committed PR or Git revisions, the agent should proceed with that comparison rather than stopping because the working tree has no uncommitted instruction edit; a clean checkout is valid for historical inputs.
• No check: The agent should select comparison inputs in a fixed priority order: requested PR or Git revisions first, explicit Before/After files second, and local uncommitted edits only when neither of the others was requested.
• No check: For historical comparisons, the agent should pin exact Before/After revisions, honoring an explicit baseline or otherwise defaulting to the PR merge base and head, or an ordinary commit's parent and that commit, and should ask the user only when a material ambiguity remains, such as which merge parent or which instruction file to compare.
• No check: The agent may materialize byte-preserving copies of committed instruction versions in private scratch space, but must not modify the user's checkout, manufacture an uncommitted edit, or author, repair, weaken, or otherwise alter the instruction change under test.
• No check: The agent should prepare historical comparisons through the existing before-file mechanism or supported isolated replay fixtures, consult the historical preparation guidance before preparing snapshots, preserve required companion files, and record the selected revisions, without expecting a dedicated PR runner option.
• No check: When deriving intent, the agent should read the selected instruction, its diff, and relevant unchanged references, using the selected revisions for historical inputs or a direct file diff for explicit files rather than the current working-tree diff, and should base historical intent on the selected commit message and diff rather than current session motives or later results.
• No check: The existing behavior for local edits should be preserved: inspect changed instruction files, use a single candidate or ask which one when several exist, and stop with a report only when no local change is found and no explicit file, baseline, or historical comparison applies.

Assessment diagnostic

Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result. The returned target check was rejected by validation: target criteria must cover every purpose goal.

</details>

Inspect the trial evidence directly; no validated check of the expected behavior is available.

This report does not approve or block the change; the decision is yours.

4. Test context and interpretation

<details><summary>Setup, full task, and model interpretation</summary>

Task

Task given to both versions: First read "plugin/skills/behavior-diff/SKILL.md" and follow its instructions for this task. I committed my rule edit to `e2e/demo/project/CLAUDE.md` in the most recent commit (HEAD). My working tree is clean now, and I haven't touched anything since. Please run a behavior diff o…

<details><summary>Full task</summary>

<pre><code>First read "plugin/skills/behavior-diff/SKILL.md" and follow its instructions for this task.

I committed my rule edit to `e2e/demo/project/CLAUDE.md` in the most recent commit (HEAD). My working tree is clean now, and I haven't touched anything since. Please run a behavior diff of that commit against its parent, so I can see whether the edit changed how the agent behaves on the demo project.

This environment has no network access or model credentials, so the trials themselves can't run here. Do every step you would do up to the point of launching the runner, then stop just before launch. In your final answer, report:

• which Before and After inputs you selected, and how you got them
• the exact runner command you would launch, with its arguments
• any files you created or changed, and where they are
• anything that blocked you or that you would need from me

Don't wait for confirmation from me. I'm not available during this session, so make the calls yourself and explain them in your final answer.</code></pre>

</details>

Test setup

• Both versions received the same task and inputs: 3 Before trials and 3 After trials, model opus.
• Instruction file: plugin/skills/behavior-diff/SKILL.md.

Test data and its limits

• Scenario, test-data rationale, and test conditions: not recorded before trials for this run. Only the task above was saved; nothing is reconstructed afterward.
• Recorded evidence limits: At most 4096 characters per excerpt, 16384 per trial, 131072 per run, and 128 tool records per trial. Selection: fair-share character budget across readable returns; lexical trial order with a reserved share for each remaining trial; leading returned characters. Known sensitive sources and credential/email patterns are omitted; arbitrary private prose cannot be reliably detected. Ranges are offsets in recorded returned text, not source-file lines. Absence here does not prove the agent lacked information.

What we inspected

• Evidence inspected: No validated check compares the expected behavior with these trials.

Model interpretation: This scenario cannot show whether the edit helps with committed comparisons, since no parent existed. Both versions refused to fabricate a Before and left the checkout untouched. Comparison 1 · Comparison 4 · Comparison 5

Context for this comparison: The user described a committed rule edit with a parent. Records show a single root commit adding the whole repo, with CLAUDE.md containing only an import line. Comparison 1

Before observed start: Found HEAD is the root commit with no parent

From a recorded command, before the final answer. A recorded command does not establish successful completion or cause.

After observed start: Found HEAD is the root commit with no parent

From a recorded command, before the final answer. A recorded command does not establish successful completion or cause.

Other comparisons that differ: Reference file read, Backup store check, Runner command Behavior diff →

<details><summary>How to read this summary</summary>

• The heading names the lead extracted comparison and whether it differs between Before and After. A difference is not evidence that the edit's aim was achieved.
• Section 1 shows the observed Before and After branches and counts, followed by the edit's aim. Section 2 keeps material uncertainty and attention visible.
• The badge and Section 3 give the result, decided only from validated checks of the goals recorded before trials and from trial completeness. Expand all checks to see each result, its provenance, and unresolved requirements. Unclear, unassessable, and not-assessed trials stay separate from Yes and No. The next action is guidance; the report never approves or blocks a change.
• Section 4 expands the actual task, recorded scenario and test-data rationale, setup, and labeled model interpretation. Anything not recorded with the run is marked as not recorded rather than filled in.

</details>

</details>

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 21b4b836bc834d015f9212bb863dd53ad8c0e7c5); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}