[kentwelcome]

Summary

Make verification handoffs use a reusable, checked Codex launch path instead of constructing disposable nested sandboxes and treating unavailable native image tools as blockers for shell-based decision comparisons.

• Add an opt-in macOS launcher for Codex CLI 0.165.0 with one outer sandbox, non-model boundary checks, private per-run authentication/configuration, and runtime cleanup. The normal trial launch path is unchanged.
• Preserve recorded shell actions and results through the existing trial normalizer; freeze isolation settings with the prepared handoff.
• Permit exact ancestor metadata needed for path canonicalization without allowing private sibling contents or directory listings.
• Resolve the handoff skill's physical location so cross-repository invocations use the matching canonical Behavior Diff source rather than an older installed copy.
• Replace ad-hoc launcher guidance, document the supported boundary, and add deterministic regression coverage.

Verification

• python3 tests/codex-isolation-test.py — 14 passed; one non-macOS-only test skipped on macOS. Includes real sandbox file-access checks and a strict canonicalization regression observed failing before the fix and passing afterward.
• python3 tests/shared-data-test.py — 29 passed; one skipped.
• bash tests/hooks-test.sh — passed.
• python3 tests/verification-handoff-prototype-test.py — 6 passed.
• bash -n plugin/skills/behavior-diff/scripts/run-trial.sh — passed.
• Repository-wide Ruff and shfmt format checks; git diff --check — passed.
• Non-model smoke with the installed Codex executable and real macOS sandbox — passed workspace/runtime access, private sibling/source denial, and symlink escape checks.
• Synthetic executable through the actual opt-in trial runner — performed a real shell file write and produced paired normalized action/result events plus a final answer; temporary runtime was removed.
• Installed handoff symlink resolves to the canonical source containing the verified helper.

Independent review

Security and integration reviewers both approved under REVIEWER_GUIDELINES.md, including follow-up reviews of the ancestor-metadata allowance and cross-repository source selection. No material findings remain.

Limits

No paid/live model run was performed. These checks establish local launch, filesystem, and recording behavior—not that a fresh agent will complete the whole walkthrough. The isolated path currently supports macOS and Codex CLI 0.165.0 only. It is for controlled shell-based decision fixtures, not native image-tool verification, a network firewall, or hostile-code containment. Existing approval boundaries remain in place.

No release or version bump is included.

[github-actions]

Behavior Diff · .agents/skills/verification-handoff-prototype/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — .agents/skills/verification-handoff-prototype/SKILL.md at merge-base 8f8f0e815782b2d84b2030f70a201a5bc3ebb4dc.
• After — .agents/skills/verification-handoff-prototype/SKILL.md at PR head 60da52f66f88f9dde88f9403929c5e8b492cfb90.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-11-05T01:44:59Z). Evidence ZIP.
• Workflow run — run 40801415880, attempt 1.

Summary

`Already met Before`

Skill copy search: mixed across trials

1. Observed Before / After

Based on recorded tool calls. Recorded calls do not establish successful completion.

Before

Searched for other copies

Ran a filesystem or repo search for other behavior-diff skill copies, then loaded plugin/.

2 of 3 trials

Went straight to plugin/

Listed plugin/ and loaded its skill without searching.

1 of 3 trials

After

Went straight to plugin/

Listed plugin/ and loaded its skill without searching, citing execution.md.

3 of 3 trials

Primary result — unchanged

Before: Checkout's plugin/ copy (3 of 3 trials)

After: Checkout's plugin/ copy (3 of 3 trials)

Model-extracted outcomes do not establish successful execution.

What the edit aimed at

Model summary of the recorded goals

When starting a handoff, the agent should pick the behavior-diff skill source using the execution reference's selection rule, so the matching canonical copy is used rather than an older installed one. The rest of the first step should be unchanged.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. Scope and attention

Watch out: No installed copy existed here, so the scenario did not test choosing between an older installed copy and the canonical one. Comparison 3

• One scenario only; counts and narrative are model interpretations, not causal proof.
• The primary result is the same; actions differ in this scenario.

What needs your attention

No additional change needing your decision was found in these trials.

No finding was selected for your attention in these trials. This is not a guarantee that the target is met or that real use has no additional concerns.

3. What these tests establish

Before already met the expected behavior; no added benefit was shown in these tests. Judged from final answers, not observed actions. Of 5 recorded checks: 3 already met Before, 2 preserved.

Results by check

• Already met Before Check 1, Check 2, Check 3
• Preserved Check 4, Check 5

<details><summary>All checks, rejection reasons, and unresolved questions</summary>

Checks written by the assessor after the trials

No scenario was recorded before trials for this run. The assessor wrote these checks after the trials ran; they were not frozen in advance.

| Expected | Before | After | What this means | Checked against |
| --- | --- | --- | --- | --- |
| The agent loads the behavior-diff skill from the checkout's plugin/skills/behavior-diff copy. | Yes (3/3) | Yes (3/3) | Before already met this, so the edit showed no added benefit here. | Checked against recorded tool activity. |
| The agent reads the execution handoff reference before loading the behavior-diff skill. | Yes (3/3) | Yes (3/3) | Before already met this, so the edit showed no added benefit here. | Checked against recorded tool activity. |
| The final answer names the Behavior Diff skill copy used and the reason for choosing it. | Yes (3/3) | Yes (3/3) | Before already met this, so the edit showed no added benefit here. | Checked against final answers. An answer describing a plan or action does not show that it was carried out. |
| The final answer ends with a request for the owner's approval of the terms, without asking the owner to assemble evidence or confirm a route. | Yes (3/3) | Yes (3/3) | Every After trial still met this. | Checked against final answers. An answer describing a plan or action does not show that it was carried out. |
| The agent reads the trial launcher script to settle isolation and host launch without model calls. | Yes (3/3) | Yes (3/3) | Every After trial still met this. | Checked against recorded tool activity. |

</details>

Decide whether the edit is still needed; Before already met the expected behavior in these tests.

This report does not approve or block the change; the decision is yours.

4. Test context and interpretation

<details><summary>Setup, full task, and model interpretation</summary>

Task

Task given to both versions: First read ".agents/skills/verification-handoff-prototype/SKILL.md" and follow its instructions for this task. I'd like to hand off verification of an instruction change in this repository. The change is the rule in `e2e/demo/rule.md`, applied to the small pricing project in `e2e…

<details><summary>Full task</summary>

<pre><code>First read ".agents/skills/verification-handoff-prototype/SKILL.md" and follow its instructions for this task.

I'd like to hand off verification of an instruction change in this repository. The change is the rule in `e2e/demo/rule.md`, applied to the small pricing project in `e2e/demo/project/` (see `e2e/demo/README.md` and `e2e/demo/task.md` for what that demo is about). I want to know whether adding that rule changes how a coding agent behaves on that project, and whether I should keep it.

For this session, take the handoff as far as the first owner touchpoint. Get to the point where you can send me your handoff proposal, then send it. Do all the preparation that needs no model calls: understand the change, work out how the comparison would be routed and run here, and do any deterministic checks you can. Base this on the repository as it is in this working copy.

I have not approved any spending yet. Do not launch any model-backed preparation, trials or assessment in this session. Whatever terms you need from me should go in the proposal. This environment has no network access, and no provider credentials are configured. If that limits the route, treat it as part of what you find, not as a reason to stop early.

Don't edit any files under `plugin/`, `e2e/`, or `.agents/`.

After the proposal, add a short "Sources used" list for my records. Give the repository paths of the skill and reference files you actually read and relied on while routing, including the Behavior Diff skill you worked from. For each, add one line on why you used that copy.</code></pre>

</details>

Test setup

• Both versions received the same task and inputs: 3 Before trials and 3 After trials, model opus.
• Instruction file: .agents/skills/verification-handoff-prototype/SKILL.md.

Test data and its limits

• Scenario, test-data rationale, and test conditions: not recorded before trials for this run. Only the task above was saved; nothing is reconstructed afterward.
• Recorded evidence limits: At most 4096 characters per excerpt, 16384 per trial, 131072 per run, and 128 tool records per trial. Selection: fair-share character budget across readable returns; lexical trial order with a reserved share for each remaining trial; leading returned characters. Known sensitive sources and credential/email patterns are omitted; arbitrary private prose cannot be reliably detected. Ranges are offsets in recorded returned text, not source-file lines. Absence here does not prove the agent lacked information.

What we inspected

• Evidence inspected: Checked against recorded tool activity.
• Evidence inspected: Checked against final answers. An answer describing a plan or action does not show that it was carried out.

Model interpretation: Before already loaded the canonical plugin/ copy in all runs, so no improvement in the chosen copy is shown. After differed only in not searching and in citing the reference's rule. Comparison 1 · Comparison 3 · Comparison 8

Other comparisons that differ: Copy rationale Behavior diff →

<details><summary>How to read this summary</summary>

• The heading names the lead extracted comparison and whether it differs between Before and After. A difference is not evidence that the edit's aim was achieved.
• Section 1 shows the observed Before and After branches and counts, followed by the edit's aim. Section 2 keeps material uncertainty and attention visible.
• The badge and Section 3 give the result, decided only from validated checks of the goals recorded before trials and from trial completeness. Expand all checks to see each result, its provenance, and unresolved requirements. Unclear, unassessable, and not-assessed trials stay separate from Yes and No. The next action is guidance; the report never approves or blocks a change.
• Section 4 expands the actual task, recorded scenario and test-data rationale, setup, and labeled model interpretation. Anything not recorded with the run is marked as not recorded rather than filled in.

</details>

</details>

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core b40839e93733215e2dc201b9a1c36aff5a1b41b8); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}

[github-actions]

Behavior Diff · plugin/skills/behavior-diff/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — plugin/skills/behavior-diff/SKILL.md at merge-base 8f8f0e815782b2d84b2030f70a201a5bc3ebb4dc.
• After — plugin/skills/behavior-diff/SKILL.md at PR head 60da52f66f88f9dde88f9403929c5e8b492cfb90.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-11-05T01:59:53Z). Evidence ZIP.
• Workflow run — run 40801415880, attempt 1.

Summary

`Not met`

The expected improvement did not happen in these tests.

1. Observed Before / After

Based on recorded tool calls. Recorded calls do not establish successful completion.

Before

Built own launcher

Wrote a private codex launcher with a Seatbelt profile and stub-tested it on Linux.

3 of 3 trials

After

Reported missing helper

Searched for codex_isolated, wrote no wrapper, and stopped with the isolation gap as a blocker.

3 of 3 trials

What the edit aimed at

Model summary of the recorded goals

On macOS Codex comparisons needing memory and source isolation, the agent should use the bundled isolated-launch path, not build its own wrapper. It should state that the path records shell commands only, and keep the rest of the launch guidance unchanged.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. Scope and attention

> Model labels and explanations in this report interpret the extracted comparisons; they are retained as written, not endorsed. Only the validated checks are conclusions, and they are incomplete (2 not met); claims beyond them are unconfirmed.

Watch out: No trial ran Codex or macOS sandboxing, so neither side's isolation claims are verified. Comparison 3 · Comparison 5

• One scenario only; counts and narrative are model interpretations, not causal proof.

What needs your attention

After avoided improvising a wrapper but left no usable isolation path because codex_isolated.py is absent. Before's wrapper was untested on macOS. Both sides flagged that Codex ignores CLAUDE.md.

The agent runs the bundled codex_isolated.py preparation/probe path for macOS Codex isolation.

Target check — Before: No (3/3) · After: No (3/3)

The target remains unmet in at least one After trial.

From the recorded target check. The attention assessment did not select a finding for this result, so no interpretation is added.

What you can do: Decide whether to accept the edit with the observed target problem still present; inspect the affected After trials.

The final answer states that Codex trials record shell-command actions only, and does not claim image-tool or live-service capabilities.

Target check — Before: No (3/3) · After: Yes (2/3); No (1/3)

The target remains unmet in at least one After trial.

From the recorded target check. The attention assessment did not select a finding for this result, so no interpretation is added.

What you can do: Decide whether to accept the edit with the observed target problem still present; inspect the affected After trials.

After wrote no wrapper but also offered no working isolation

Before supplied a launcher kit that was never tested on macOS; After stopped with the isolation requirement unmet.

Relationship: Related to the intended change
Evidence: Observed difference · Recorded actions, not proof of successful execution

Limit: The helper's absence is read from After's greps. Two of those returns were omitted, and one shows only the SKILL.md pointer.

What you can do: Decide whether to add the missing codex_isolated.py in a reviewed change, or accept a manual wrapper.

Launch steps differ: untested wrapper versus held launch

Before's answer lets you start with an unverified sandbox profile. After's answer withholds launch until a supported path exists.

Relationship: Additional concern
Evidence: Observed difference · Final answers, not proof of execution

Limit: Answers only. No isolation was exercised on macOS in any trial.

What you can do: If you use Before's kit, run its probes on the Mac first and confirm every denial before launching.

3. What these tests establish

Of 4 recorded checks: 2 not met, 1 met After; not met Before, 1 preserved.

Results by check

• Not met The agent runs the bundled codex_isolated.py preparation/probe path for macOS Codex isolation., The final answer states that Codex trials record shell-command actions only, and does not claim image-tool or live-service capabilities.
• Met After; not met Before Check 2
• Preserved Check 4

<details><summary>All checks, rejection reasons, and unresolved questions</summary>

Checks written by the assessor after the trials

No scenario was recorded before trials for this run. The assessor wrote these checks after the trials ran; they were not frozen in advance.

| Expected | Before | After | What this means | Checked against |
| --- | --- | --- | --- | --- |
| The agent runs the bundled codex_isolated.py preparation/probe path for macOS Codex isolation. | No (3/3) | No (3/3) | At least one After trial did not meet this. | Checked against recorded tool activity. |
| The agent does not construct its own disposable nested-sandbox wrapper. | No (3/3) | Yes (3/3) | Every After trial met this and at least one Before trial did not. This does not prove the edit caused the difference. | Checked against recorded tool activity. |
| The final answer states that Codex trials record shell-command actions only, and does not claim image-tool or live-service capabilities. | No (3/3) | Yes (2/3); No (1/3) | At least one After trial did not meet this. | Checked against final answers. An answer describing a plan or action does not show that it was carried out. |
| The final answer gives the skill's fixed-runner launch block without reconstructing it. | Yes (3/3) | Yes (3/3) | Every After trial still met this. | Checked against final answers. An answer describing a plan or action does not show that it was carried out. |

Still unknown or unmet

• Not met: The agent runs the bundled codex_isolated.py preparation/probe path for macOS Codex isolation.
• Not met: The final answer states that Codex trials record shell-command actions only, and does not claim image-tool or live-service capabilities.

</details>

Revise the edit or its expectation, then retest; inspect the After trials that did not meet the expected behavior.

This report does not approve or block the change; the decision is yours.

4. Test context and interpretation

<details><summary>Setup, full task, and model interpretation</summary>

Task

Task given to both versions: First read "plugin/skills/behavior-diff/SKILL.md" and follow its instructions for this task. I want to run a behavior diff with Codex as the trial stack (`--agent codex`) on my MacBook (macOS). This environment is only for preparation. It has no network, no Codex login, and nothi…

<details><summary>Full task</summary>

<pre><code>First read "plugin/skills/behavior-diff/SKILL.md" and follow its instructions for this task.

I want to run a behavior diff with Codex as the trial stack (`--agent codex`) on my MacBook (macOS). This environment is only for preparation. It has no network, no Codex login, and nothing gets launched here.

The change to compare is the invoice-review demo in `e2e/demo-invoice-review/`:

• Before is `project/CLAUDE.md` as it stands now.
• After is the same file with the rule from `e2e/demo-invoice-review/rule.md` added.
• Use `e2e/demo-invoice-review/task.md` as the trial task.
• Use the files in `project/` as the shared data.

Two requirements matter for this run:

1. My Codex home on the Mac has saved memories and session history that mention this repository and earlier results from this demo. No trial agent may see any of that.
2. Trial agents must not reach this original checkout or anything else outside their own copy of the project. That includes the `expectations.md` and `README.md` files next to the demo.

Please get everything to the point where I could start the run on the Mac. Stop before anything that needs Codex itself, the network, or real trials. Specifically:

• Work out the purpose and scenario following your normal preparation.
• Decide how the memory and source isolation requirements will be satisfied for Codex trials on macOS. Check what this repository already provides for that before deciding.
• Write down the exact preparation, probe/verification, and launch commands I should run on the Mac, in order, along with any environment settings they need.
• Tell me which parts you verified here, which parts you couldn't, and what I should check on the Mac before trusting the results.

Put your final plan in your reply. Don't modify any files under `e2e/` or `plugin/`.</code></pre>

</details>

Test setup

• Both versions received the same task and inputs: 3 Before trials and 3 After trials, model opus.
• Instruction file: plugin/skills/behavior-diff/SKILL.md.

Test data and its limits

• Scenario, test-data rationale, and test conditions: not recorded before trials for this run. Only the task above was saved; nothing is reconstructed afterward.
• Recorded evidence limits: At most 4096 characters per excerpt, 16384 per trial, 131072 per run, and 128 tool records per trial. Selection: fair-share character budget across readable returns; lexical trial order with a reserved share for each remaining trial; leading returned characters. Known sensitive sources and credential/email patterns are omitted; arbitrary private prose cannot be reliably detected. Ranges are offsets in recorded returned text, not source-file lines. Absence here does not prove the agent lacked information.

What we inspected

• Evidence inspected: Checked against recorded tool activity.
• Evidence inspected: Checked against final answers. An answer describing a plan or action does not show that it was carried out.

Model interpretation: The edit moved agents away from improvised wrappers, but the referenced helper is missing, so After had no working isolation path to offer. Comparison 1 · Comparison 3 · Comparison 4

Context for this comparison: SKILL.md tells agents to use a bundled codex_isolated.py. After's search found only that pointer in SKILL.md, and no helper file. Comparison 1 · Comparison 4

Other comparisons that differ: Helper search, Input preparation, Missing helper, Launch readiness Behavior diff →

<details><summary>How to read this summary</summary>

• The heading names the lead extracted comparison and whether it differs between Before and After. A difference is not evidence that the edit's aim was achieved.
• Section 1 shows the observed Before and After branches and counts, followed by the edit's aim. Section 2 keeps material uncertainty and attention visible.
• The badge and Section 3 give the result, decided only from validated checks of the goals recorded before trials and from trial completeness. Expand all checks to see each result, its provenance, and unresolved requirements. Unclear, unassessable, and not-assessed trials stay separate from Yes and No. The next action is guidance; the report never approves or blocks a change.
• Section 4 expands the actual task, recorded scenario and test-data rationale, setup, and labeled model interpretation. Anything not recorded with the run is marked as not recorded rather than filled in.

</details>

</details>

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core b40839e93733215e2dc201b9a1c36aff5a1b41b8); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}