Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .agents/skills/verification-handoff-prototype/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,12 @@ Humanizer guidance in embedded mode without installing it or changing evidence.

## Workflow

1. **Understand the change and route it without model calls.** Load the
available `behavior-diff` skill. Inspect the selected diff, relevant unchanged
rules and available discussion. Pin revisions and source paths. Then settle the
route with deterministic checks: fixture fit, file mapping, host
authentication, model availability and isolation, as
1. **Understand the change and route it without model calls.** Resolve and load
the `behavior-diff` skill using the source-selection rule in
[execution handoff](references/execution.md). Inspect the selected diff,
relevant unchanged rules and available discussion. Pin revisions and source
paths. Then settle the route with deterministic checks: fixture fit, file
mapping, host authentication, model availability and isolation, as
[execution handoff](references/execution.md) describes. Do not require the
owner to assemble an evidence packet or confirm a route the repository answers.
2. **Send one handoff proposal.** In one message, propose the intended outcomes,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# Execution handoff

The host owns this work. Do not expose helper setup as owner homework. Load the
available Behavior Diff skill once and retain its exact base and version. In a
Behavior Diff development checkout, use its canonical `plugin/skills/behavior-diff`
source; elsewhere use the discovered installed skill. Never silently mix versions.
The host owns this work. Do not expose helper setup as owner homework. Resolve
this handoff skill's physical location before choosing Behavior Diff. If it
belongs to a Behavior Diff source checkout, load that checkout's canonical
`plugin/skills/behavior-diff` source, even when the reviewed project is elsewhere.
Otherwise load the discovered installed Behavior Diff skill. Retain its exact
base and version for the whole comparison; never silently mix versions.
Missing tooling is a concrete prerequisite to resolve or report, not permission
to invent a substitute runner.

Expand Down Expand Up @@ -98,27 +100,34 @@ Do not claim complete isolation if a relevant channel cannot be checked.

Run the fresh historical preparer as a host subprocess with its own isolated
context. Use an in-session subagent only when the host verifiably gives it no
injected session memory. If Behavior Diff provides no isolation for the host, use
a disposable launcher in the private run root, for example a wrapper first on
`PATH` that applies process-local flags disabling memory and plugins and denies
reads of memory stores and the original checkout. Prefer the host's existing
sign-in. If a private host home is unavoidable, disclose it in the proposal, keep
any credential copy mode 0600 inside the mode-0700 run root, never print it, and
delete it when the run ends or is interrupted. Verify the launcher with harmless
non-model probes before model use, keep it with the evidence, and record it in
the result. Never change the owner's global host settings instead.

Before a paid probe, check the actual launcher path with non-model probes:
authentication availability, writing and reading in the trial workspace, and
denial of reads from excluded sources. A copy from the outer shell alone does not
verify the agent's execution path. Check for nested sandbox restrictions before
launch; do not simply disable protections to make the probe pass. An alternative
isolation configuration needs equivalent read/write boundaries verified first.
Check whether the host records the actions needed by the frozen criteria.
File existence can establish an artifact, not that the agent inspected it.
If the host cannot exercise an execution path without a model, mark that path
unverified and include its smallest necessary probe in the approval. Do not
claim that an outer-shell check proves the inner path works.
injected session memory. For macOS Codex decision comparisons, use the bundled
`codex_isolated.py` launcher described in Behavior Diff's
`references/scenario-and-test-data.md`. Do not reuse an old private launcher or
write a new wrapper. Use one selected Behavior Diff version for preparation and
trials; an older installation without the helper is a version prerequisite, not
evidence that the repository needs Go or live image generation.

The helper uses one outer sandbox and checks its file-access boundary before
starting Codex. It isolates the host home and temporarily copies existing sign-in
credentials, so disclose that in the proposal. It cleans up the temporary runtime
on completion or a handled interruption. Never print credentials or change global
host settings. If the host cannot start the helper's sandbox, preserve its error;
use the host's explicit permission mechanism when allowed, not an unconfined
fallback. A sandbox failure is about that execution environment, not the PR.

The supported evidence path is Codex shell commands, command returns and workspace
artifacts. Use local adapters for approved decision simulations. Missing native
image-tool events do not establish that shell actions cannot be recorded.
Do not require real artwork saving for a decision-only question. If the approved
question requires actual image generation, inspection or a real service, those
capabilities remain separate preflight requirements; never silently simulate them.
File existence establishes an artifact, not that the agent inspected it.

On other hosts, use their documented isolation rather than claiming this macOS
helper applies. If no supported path can preserve the required boundary, report
that limitation and finish independent checks. A model-backed capability probe
still requires explicit authorization; non-model checks do not prove agent
compliance or live-service reliability.

Follow Behavior Diff's preflight and frozen-handoff helpers rather than rebuilding
their validators. Simulation checks prove the fixture's mechanics, not that a
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ jobs:
- name: Check shared-data preparation boundaries
run: python3 tests/shared-data-test.py

- name: Check isolated Codex launcher without models
run: python3 tests/codex-isolation-test.py

- name: Check optional Jev helper without network
run: python3 tests/jev-test.py

Expand Down
27 changes: 20 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,15 @@ Normal trial tools, the primary model and reasoning settings, trial-local storag
and native trace normalization are unchanged; user configuration is not edited.
Claude, Codex, and Pi do not use this OMP advisor subsystem.

Codex trials opt into `codex_isolated.py` when
`BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT` is set; `BEHAVIOR_DIFF_CODEX_DENY_READ`
names an additional source checkout to exclude. This macOS-only path validates
its outer sandbox before model execution and avoids adding a second sandbox
inside it. Each process gets a private disposable host home; workspace writes
remain allowed while sibling private data is denied. Normal Codex launches are
unchanged when the opt-in is absent. This path targets controlled shell-based
decision fixtures, not native image tooling or network isolation.

The trace contains paired tool actions and recorded returns plus a final answer.
Claude's stream is retained; Codex command output and Pi/OMP tool results keep
their call identities during normalization. Raw host logs and stderr remain
Expand Down Expand Up @@ -994,12 +1003,16 @@ The owner's touchpoints are the request, one proposal and the adoption decision.
The host owns planning, preparation, bounded recovery and interpretation, without
requiring owner reminders between stages. It decides whether a workflow engine is
under test from the target repository, not instruction wording. It normally runs
the fresh preparer as an isolated host subprocess, using a disposable private
launcher when Behavior Diff has no host isolation. It delegates comparison
mechanics to Behavior Diff's existing contracts, not a second runner. Unsupported
capabilities remain blockers rather than silently reduced scope. Paid runs and
private-data handoffs need explicit bounded authority; model switches are limited
to named fallbacks. Global configuration is not a write target. Evaluation
the fresh preparer as an isolated host subprocess. On macOS, Codex preparation
and opt-in trials use the bundled `codex_isolated.py` launcher rather than a
per-session wrapper. It verifies an outer filesystem sandbox before disabling
Codex's nested sandbox, isolates the temporary host home, and cleans up copied
sign-in data. `run-trial.sh` still normalizes recorded shell actions; assessment
uses its existing host path. The frozen handoff pins the isolation environment
and shipped helper bytes. Unsupported capabilities remain blockers rather than
silently reduced scope. Paid runs and private-data handoffs need explicit bounded
authority; model switches are limited to named fallbacks. Global configuration
is not a write target. Evaluation
responses and merge questions never authorize PR operations.

Failed attempts stop the affected execution, not safe non-model recovery or
Expand All @@ -1017,7 +1030,7 @@ Explicit replay-only requests use saved evidence without new trials. Optional
and read-only root-scoped citations for that subset. Its fixed-evidence mode is
not a fresh-run status. Unchanged support survives correction; missing/changed
citations become gaps. Private packets stay outside Git trees; synthetic fixtures
exercise helper mechanics only. No helper schema or production runner changed.
exercise helper mechanics only. The replay schema is unchanged.
Owner effort reduction requires an actual walkthrough, not deterministic checks;
feedback is optional after delivery. Historical experimental purpose stays frozen.

Expand Down
6 changes: 6 additions & 0 deletions plugin/skills/behavior-diff/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,12 @@ the single-role or two-agent path. Create the fixtures with
reconstruct validators, inventories, or commands. Spacedock fixture delivery
remains on its own reference path.

For macOS Codex comparisons requiring memory and source isolation, use the
reference's bundled `codex_isolated.py` preparation/probe path and opt-in
trial environment. Do not invent another nested-sandbox wrapper. This path
records shell-command actions for local decision fixtures; it does not
establish native image-tool or live-service capabilities.

Preserve the current host and model choices above. Keep three trials per
side; select `fast:true` only for an explicit current request for `fast`,
`--fast`, `two runs`, or `one trial per side`. Never put purpose, scenario,
Expand Down
69 changes: 68 additions & 1 deletion plugin/skills/behavior-diff/references/scenario-and-test-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,71 @@ If `prepare` exits with a nonzero status, the helper removes the incomplete
stage. Stop. Correct the inputs, then prepare a new stage path. Do not edit
files in a stage by hand.

## Isolated Codex decision comparisons on macOS

For a comparison that requires isolation from the owner's memory and checkout,
use `scripts/codex_isolated.py` rather than building a private launcher. This is
an opt-in execution path for preparation and trials, not a new scenario generator
or evidence assessor. It supports shell-command actions and workspace artifacts;
it does not establish native image-tool recording or live-service reliability.
The supported host is Codex CLI `0.165.0` on macOS. The probe rejects other
versions rather than assuming that isolation flags still have the same meaning.
This version check is not a live-model execution check.

Create a private scratch root outside the original checkout. Keep the preparation
workspace, staged project, run output and private assessment notes under that
root in separate directories. A process may read its own workspace and temporary
runtime, but not sibling notes or trial workspaces. Put only permitted historical
inputs into the preparer's workspace. Do not give it prior outcomes or owner
answers.

Before model use, run the non-model boundary check:

```bash
python3 "$SKILL_DIR/scripts/codex_isolated.py" probe \
--workspace "$PREPARATION_WORKSPACE" --root "$PRIVATE_ROOT" \
--deny-read "$ORIGINAL_PROJECT"
```

After bounded model/data approval, invoke a fresh preparer through the same helper:

```bash
python3 "$SKILL_DIR/scripts/codex_isolated.py" exec \
--workspace "$PREPARATION_WORKSPACE" --root "$PRIVATE_ROOT" \
--deny-read "$ORIGINAL_PROJECT" --model "$PREPARATION_MODEL" \
< "$PRIVATE/preparation-task.txt" > "$PRIVATE/preparation-raw.jsonl"
```

The prompt is read by the launching process; trial-visible files remain confined
to the workspace. Disclose that the helper temporarily copies the existing Codex
sign-in into a private per-process home. It removes that runtime on completion
or a handled interruption. It never writes the user's global configuration.

Set these variables before freezing, verifying and launching the comparison:

```bash
export BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT="$PRIVATE_ROOT"
export BEHAVIOR_DIFF_CODEX_DENY_READ="$ORIGINAL_PROJECT"
```

The standard fixed-runner invocation below is unchanged. `run-trial.sh` selects
the helper for Codex trials when the isolation root is set. The frozen handoff
records both settings and rejects changes before launch. Assessment remains on
its existing host path and receives the permitted recorded evidence.

The helper verifies one outer macOS sandbox, then disables Codex's inner sandbox
only inside that boundary to avoid nested-sandbox failures. If this check fails,
no model starts. Do not bypass a denied sandbox launch; obtain any necessary host
permission or report the unavailable execution capability. This is filesystem
isolation for controlled decision fixtures, not a network firewall or a guarantee
against hostile code. Do not put credentials or real external actions in fixtures.
Native image generation and real worker handoffs need their own supported paths.

The boundary check does not prove that a model uses tools correctly or that its
events are complete. Audit native `command_execution` events and normalized
tool results after authorized trials; do not treat final-answer claims as actions.


## Verify, then run

Verification immediately before launch compares the complete handoff:
Expand Down Expand Up @@ -508,4 +573,6 @@ directory that the runner prints with `status.py`. To list stage runs with
- Live services, credentials, writable external systems, and multi-file
instruction versions are not delivered.
- The helper does not detect secrets. The caller reviews every shared file.
- No path provides a filesystem sandbox. See [Freeze discipline](#freeze-discipline).
- Staging and freezing alone do not provide a filesystem sandbox. The optional
[isolated Codex path](#isolated-codex-decision-comparisons-on-macos) adds the
documented boundary. See also [Freeze discipline](#freeze-discipline).
Loading
Loading