diff --git a/.agents/skills/verification-handoff-prototype/SKILL.md b/.agents/skills/verification-handoff-prototype/SKILL.md index 41d8550..101e7eb 100644 --- a/.agents/skills/verification-handoff-prototype/SKILL.md +++ b/.agents/skills/verification-handoff-prototype/SKILL.md @@ -66,11 +66,12 @@ Humanizer guidance in embedded mode without installing it or changing evidence. ## Workflow -1. **Understand the change and route it without model calls.** Load the - available `behavior-diff` skill. Inspect the selected diff, relevant unchanged - rules and available discussion. Pin revisions and source paths. Then settle the - route with deterministic checks: fixture fit, file mapping, host - authentication, model availability and isolation, as +1. **Understand the change and route it without model calls.** Resolve and load + the `behavior-diff` skill using the source-selection rule in + [execution handoff](references/execution.md). Inspect the selected diff, + relevant unchanged rules and available discussion. Pin revisions and source + paths. Then settle the route with deterministic checks: fixture fit, file + mapping, host authentication, model availability and isolation, as [execution handoff](references/execution.md) describes. Do not require the owner to assemble an evidence packet or confirm a route the repository answers. 2. **Send one handoff proposal.** In one message, propose the intended outcomes, diff --git a/.agents/skills/verification-handoff-prototype/references/execution.md b/.agents/skills/verification-handoff-prototype/references/execution.md index 57efd3b..e7c87fa 100644 --- a/.agents/skills/verification-handoff-prototype/references/execution.md +++ b/.agents/skills/verification-handoff-prototype/references/execution.md @@ -1,9 +1,11 @@ # Execution handoff -The host owns this work. Do not expose helper setup as owner homework. Load the -available Behavior Diff skill once and retain its exact base and version. In a -Behavior Diff development checkout, use its canonical `plugin/skills/behavior-diff` -source; elsewhere use the discovered installed skill. Never silently mix versions. +The host owns this work. Do not expose helper setup as owner homework. Resolve +this handoff skill's physical location before choosing Behavior Diff. If it +belongs to a Behavior Diff source checkout, load that checkout's canonical +`plugin/skills/behavior-diff` source, even when the reviewed project is elsewhere. +Otherwise load the discovered installed Behavior Diff skill. Retain its exact +base and version for the whole comparison; never silently mix versions. Missing tooling is a concrete prerequisite to resolve or report, not permission to invent a substitute runner. @@ -98,27 +100,34 @@ Do not claim complete isolation if a relevant channel cannot be checked. Run the fresh historical preparer as a host subprocess with its own isolated context. Use an in-session subagent only when the host verifiably gives it no -injected session memory. If Behavior Diff provides no isolation for the host, use -a disposable launcher in the private run root, for example a wrapper first on -`PATH` that applies process-local flags disabling memory and plugins and denies -reads of memory stores and the original checkout. Prefer the host's existing -sign-in. If a private host home is unavoidable, disclose it in the proposal, keep -any credential copy mode 0600 inside the mode-0700 run root, never print it, and -delete it when the run ends or is interrupted. Verify the launcher with harmless -non-model probes before model use, keep it with the evidence, and record it in -the result. Never change the owner's global host settings instead. - -Before a paid probe, check the actual launcher path with non-model probes: -authentication availability, writing and reading in the trial workspace, and -denial of reads from excluded sources. A copy from the outer shell alone does not -verify the agent's execution path. Check for nested sandbox restrictions before -launch; do not simply disable protections to make the probe pass. An alternative -isolation configuration needs equivalent read/write boundaries verified first. -Check whether the host records the actions needed by the frozen criteria. -File existence can establish an artifact, not that the agent inspected it. -If the host cannot exercise an execution path without a model, mark that path -unverified and include its smallest necessary probe in the approval. Do not -claim that an outer-shell check proves the inner path works. +injected session memory. For macOS Codex decision comparisons, use the bundled +`codex_isolated.py` launcher described in Behavior Diff's +`references/scenario-and-test-data.md`. Do not reuse an old private launcher or +write a new wrapper. Use one selected Behavior Diff version for preparation and +trials; an older installation without the helper is a version prerequisite, not +evidence that the repository needs Go or live image generation. + +The helper uses one outer sandbox and checks its file-access boundary before +starting Codex. It isolates the host home and temporarily copies existing sign-in +credentials, so disclose that in the proposal. It cleans up the temporary runtime +on completion or a handled interruption. Never print credentials or change global +host settings. If the host cannot start the helper's sandbox, preserve its error; +use the host's explicit permission mechanism when allowed, not an unconfined +fallback. A sandbox failure is about that execution environment, not the PR. + +The supported evidence path is Codex shell commands, command returns and workspace +artifacts. Use local adapters for approved decision simulations. Missing native +image-tool events do not establish that shell actions cannot be recorded. +Do not require real artwork saving for a decision-only question. If the approved +question requires actual image generation, inspection or a real service, those +capabilities remain separate preflight requirements; never silently simulate them. +File existence establishes an artifact, not that the agent inspected it. + +On other hosts, use their documented isolation rather than claiming this macOS +helper applies. If no supported path can preserve the required boundary, report +that limitation and finish independent checks. A model-backed capability probe +still requires explicit authorization; non-model checks do not prove agent +compliance or live-service reliability. Follow Behavior Diff's preflight and frozen-handoff helpers rather than rebuilding their validators. Simulation checks prove the fixture's mechanics, not that a diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3aba067..49370ea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -109,6 +109,9 @@ jobs: - name: Check shared-data preparation boundaries run: python3 tests/shared-data-test.py + - name: Check isolated Codex launcher without models + run: python3 tests/codex-isolation-test.py + - name: Check optional Jev helper without network run: python3 tests/jev-test.py diff --git a/docs/architecture.md b/docs/architecture.md index a535f7d..60dd666 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -272,6 +272,15 @@ Normal trial tools, the primary model and reasoning settings, trial-local storag and native trace normalization are unchanged; user configuration is not edited. Claude, Codex, and Pi do not use this OMP advisor subsystem. +Codex trials opt into `codex_isolated.py` when +`BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT` is set; `BEHAVIOR_DIFF_CODEX_DENY_READ` +names an additional source checkout to exclude. This macOS-only path validates +its outer sandbox before model execution and avoids adding a second sandbox +inside it. Each process gets a private disposable host home; workspace writes +remain allowed while sibling private data is denied. Normal Codex launches are +unchanged when the opt-in is absent. This path targets controlled shell-based +decision fixtures, not native image tooling or network isolation. + The trace contains paired tool actions and recorded returns plus a final answer. Claude's stream is retained; Codex command output and Pi/OMP tool results keep their call identities during normalization. Raw host logs and stderr remain @@ -994,12 +1003,16 @@ The owner's touchpoints are the request, one proposal and the adoption decision. The host owns planning, preparation, bounded recovery and interpretation, without requiring owner reminders between stages. It decides whether a workflow engine is under test from the target repository, not instruction wording. It normally runs -the fresh preparer as an isolated host subprocess, using a disposable private -launcher when Behavior Diff has no host isolation. It delegates comparison -mechanics to Behavior Diff's existing contracts, not a second runner. Unsupported -capabilities remain blockers rather than silently reduced scope. Paid runs and -private-data handoffs need explicit bounded authority; model switches are limited -to named fallbacks. Global configuration is not a write target. Evaluation +the fresh preparer as an isolated host subprocess. On macOS, Codex preparation +and opt-in trials use the bundled `codex_isolated.py` launcher rather than a +per-session wrapper. It verifies an outer filesystem sandbox before disabling +Codex's nested sandbox, isolates the temporary host home, and cleans up copied +sign-in data. `run-trial.sh` still normalizes recorded shell actions; assessment +uses its existing host path. The frozen handoff pins the isolation environment +and shipped helper bytes. Unsupported capabilities remain blockers rather than +silently reduced scope. Paid runs and private-data handoffs need explicit bounded +authority; model switches are limited to named fallbacks. Global configuration +is not a write target. Evaluation responses and merge questions never authorize PR operations. Failed attempts stop the affected execution, not safe non-model recovery or @@ -1017,7 +1030,7 @@ Explicit replay-only requests use saved evidence without new trials. Optional and read-only root-scoped citations for that subset. Its fixed-evidence mode is not a fresh-run status. Unchanged support survives correction; missing/changed citations become gaps. Private packets stay outside Git trees; synthetic fixtures -exercise helper mechanics only. No helper schema or production runner changed. +exercise helper mechanics only. The replay schema is unchanged. Owner effort reduction requires an actual walkthrough, not deterministic checks; feedback is optional after delivery. Historical experimental purpose stays frozen. diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index 3587bbd..7212256 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -255,6 +255,12 @@ the single-role or two-agent path. Create the fixtures with reconstruct validators, inventories, or commands. Spacedock fixture delivery remains on its own reference path. + For macOS Codex comparisons requiring memory and source isolation, use the + reference's bundled `codex_isolated.py` preparation/probe path and opt-in + trial environment. Do not invent another nested-sandbox wrapper. This path + records shell-command actions for local decision fixtures; it does not + establish native image-tool or live-service capabilities. + Preserve the current host and model choices above. Keep three trials per side; select `fast:true` only for an explicit current request for `fast`, `--fast`, `two runs`, or `one trial per side`. Never put purpose, scenario, diff --git a/plugin/skills/behavior-diff/references/scenario-and-test-data.md b/plugin/skills/behavior-diff/references/scenario-and-test-data.md index 15a1442..ec59de5 100644 --- a/plugin/skills/behavior-diff/references/scenario-and-test-data.md +++ b/plugin/skills/behavior-diff/references/scenario-and-test-data.md @@ -417,6 +417,71 @@ If `prepare` exits with a nonzero status, the helper removes the incomplete stage. Stop. Correct the inputs, then prepare a new stage path. Do not edit files in a stage by hand. +## Isolated Codex decision comparisons on macOS + +For a comparison that requires isolation from the owner's memory and checkout, +use `scripts/codex_isolated.py` rather than building a private launcher. This is +an opt-in execution path for preparation and trials, not a new scenario generator +or evidence assessor. It supports shell-command actions and workspace artifacts; +it does not establish native image-tool recording or live-service reliability. +The supported host is Codex CLI `0.165.0` on macOS. The probe rejects other +versions rather than assuming that isolation flags still have the same meaning. +This version check is not a live-model execution check. + +Create a private scratch root outside the original checkout. Keep the preparation +workspace, staged project, run output and private assessment notes under that +root in separate directories. A process may read its own workspace and temporary +runtime, but not sibling notes or trial workspaces. Put only permitted historical +inputs into the preparer's workspace. Do not give it prior outcomes or owner +answers. + +Before model use, run the non-model boundary check: + +```bash +python3 "$SKILL_DIR/scripts/codex_isolated.py" probe \ + --workspace "$PREPARATION_WORKSPACE" --root "$PRIVATE_ROOT" \ + --deny-read "$ORIGINAL_PROJECT" +``` + +After bounded model/data approval, invoke a fresh preparer through the same helper: + +```bash +python3 "$SKILL_DIR/scripts/codex_isolated.py" exec \ + --workspace "$PREPARATION_WORKSPACE" --root "$PRIVATE_ROOT" \ + --deny-read "$ORIGINAL_PROJECT" --model "$PREPARATION_MODEL" \ + < "$PRIVATE/preparation-task.txt" > "$PRIVATE/preparation-raw.jsonl" +``` + +The prompt is read by the launching process; trial-visible files remain confined +to the workspace. Disclose that the helper temporarily copies the existing Codex +sign-in into a private per-process home. It removes that runtime on completion +or a handled interruption. It never writes the user's global configuration. + +Set these variables before freezing, verifying and launching the comparison: + +```bash +export BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT="$PRIVATE_ROOT" +export BEHAVIOR_DIFF_CODEX_DENY_READ="$ORIGINAL_PROJECT" +``` + +The standard fixed-runner invocation below is unchanged. `run-trial.sh` selects +the helper for Codex trials when the isolation root is set. The frozen handoff +records both settings and rejects changes before launch. Assessment remains on +its existing host path and receives the permitted recorded evidence. + +The helper verifies one outer macOS sandbox, then disables Codex's inner sandbox +only inside that boundary to avoid nested-sandbox failures. If this check fails, +no model starts. Do not bypass a denied sandbox launch; obtain any necessary host +permission or report the unavailable execution capability. This is filesystem +isolation for controlled decision fixtures, not a network firewall or a guarantee +against hostile code. Do not put credentials or real external actions in fixtures. +Native image generation and real worker handoffs need their own supported paths. + +The boundary check does not prove that a model uses tools correctly or that its +events are complete. Audit native `command_execution` events and normalized +tool results after authorized trials; do not treat final-answer claims as actions. + + ## Verify, then run Verification immediately before launch compares the complete handoff: @@ -508,4 +573,6 @@ directory that the runner prints with `status.py`. To list stage runs with - Live services, credentials, writable external systems, and multi-file instruction versions are not delivered. - The helper does not detect secrets. The caller reviews every shared file. -- No path provides a filesystem sandbox. See [Freeze discipline](#freeze-discipline). +- Staging and freezing alone do not provide a filesystem sandbox. The optional + [isolated Codex path](#isolated-codex-decision-comparisons-on-macos) adds the + documented boundary. See also [Freeze discipline](#freeze-discipline). diff --git a/plugin/skills/behavior-diff/scripts/codex_isolated.py b/plugin/skills/behavior-diff/scripts/codex_isolated.py new file mode 100755 index 0000000..1de63f2 --- /dev/null +++ b/plugin/skills/behavior-diff/scripts/codex_isolated.py @@ -0,0 +1,375 @@ +#!/usr/bin/env python3 +"""Run Codex under one macOS filesystem sandbox, without a nested seatbelt. + +This is filesystem containment, not network or complete adversarial isolation. +Only exec reads credentials or starts a model; probe performs local access checks. +""" + +import argparse +import json +import os +from pathlib import Path +import shutil +import signal +import stat +import subprocess +import sys +import tempfile + + +class IsolationError(Exception): + """A prerequisite for isolated execution could not be established.""" + + +def contains(parent, child): + return child == parent or parent in child.parents + + +def canonical(value, label): + path = Path(value).expanduser() + if not path.is_absolute(): + raise IsolationError(f"{label} must be an absolute path") + # macOS's system /tmp and /var aliases are legitimate; caller-controlled + # symlinks are not, including symlinked ancestors of a missing root. + current = Path(path.anchor) + for part in path.parts[1:]: + current = current / part + if current.is_symlink() and current not in (Path("/tmp"), Path("/var")): + raise IsolationError(f"{label} must not contain symlinks") + return path.resolve() + + +def validate_paths(workspace, root, deny_read, codex): + workspace = canonical(workspace, "workspace") + root = canonical(root, "root") + home = Path.home().resolve() + denied = [canonical(value, "deny-read") for value in deny_read] + if not workspace.is_dir(): + raise IsolationError("workspace must be an existing directory") + if contains(workspace, home): + raise IsolationError("workspace must not expose the original home directory") + if root == Path("/") or root == home or contains(workspace, root): + raise IsolationError( + "root must be outside workspace and must not be home or filesystem root" + ) + for path in denied: + if not path.exists(): + raise IsolationError("deny-read paths must exist") + if contains(path, workspace) or contains(workspace, path): + raise IsolationError("deny-read paths must not overlap workspace") + if contains(path, root) or contains(root, path): + raise IsolationError("deny-read paths must not overlap private root") + if not contains(root, workspace): + raise IsolationError("workspace must be inside the private root") + executable = shutil.which(codex) + if executable is None: + raise IsolationError( + "Codex executable is unavailable; install Codex or pass --codex" + ) + executable = Path(executable).resolve() + if not executable.is_file() or not os.access(executable, os.X_OK): + raise IsolationError("Codex executable must be an executable file") + runtime_parent = executable.parent + if runtime_parent == home or contains(runtime_parent, home): + raise IsolationError("Codex runtime parent would expose the original home") + configured_home = ( + Path(os.environ.get("CODEX_HOME", str(home / ".codex"))).expanduser().resolve() + ) + if contains(workspace, configured_home) or contains(configured_home, workspace): + raise IsolationError("workspace must not expose original authentication data") + if contains(runtime_parent, configured_home): + raise IsolationError( + "Codex runtime parent would expose original authentication data" + ) + if contains(runtime_parent, root) or contains(root, runtime_parent): + raise IsolationError("Codex runtime must be outside private root") + if contains(runtime_parent, workspace) or contains(workspace, runtime_parent): + raise IsolationError("Codex runtime must be outside workspace") + if any(contains(runtime_parent, p) or contains(p, runtime_parent) for p in denied): + raise IsolationError("Codex runtime parent overlaps protected source data") + denied.append(configured_home) + if not root.exists(): + # Do not create an uncontrolled chain of parents or chmod existing data. + root.mkdir(mode=0o700) + info = root.stat() + if ( + not stat.S_ISDIR(info.st_mode) + or info.st_uid != os.getuid() + or info.st_mode & 0o077 + ): + raise IsolationError( + "root permissions must be 0700 and ownership must match this user" + ) + return workspace, root, home, denied, executable + + +def sandbox_profile(workspace, root, runtime, home, denied, codex_parent): + def subpath(path): + return "(subpath " + json.dumps(str(path)) + ")" + + protected = [home, root, *denied] + reads = [workspace, runtime, codex_parent] + # Canonicalization needs ancestor metadata, not directory contents. Keep + # sibling files and directory listings denied by using exact literals only. + ancestors = sorted({str(parent) for path in reads for parent in path.parents}) + return "\n".join( + [ + "(version 1)", + "(allow default)", + "(deny file-read* " + " ".join(map(subpath, protected)) + ")", + "(allow file-read* " + " ".join(map(subpath, reads)) + ")", + "(allow file-read-metadata " + + " ".join("(literal " + json.dumps(path) + ")" for path in ancestors) + + ")", + "(deny file-write*)", + "(allow file-write* " + + subpath(workspace) + + " " + + subpath(runtime) + + ' (literal "/dev/null") (literal "/dev/tty"))', + "", + ] + ) + + +def isolated_environment(runtime): + # Deliberately omit API keys, inherited Codex settings, shell startup files, + # dynamic loader overrides, package-manager hooks, and plugin configuration. + env = { + key: os.environ[key] for key in ("LANG", "LC_ALL", "TERM") if key in os.environ + } + env.update( + { + "PATH": "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin", + "HOME": str(runtime / "home"), + "CODEX_HOME": str(runtime / "codex-home"), + "TMPDIR": str(runtime / "tmp"), + "SHELL": "/bin/sh", + "BEHAVIOR_DIFF_TRIAL": "1", + "BEHAVIOR_DIFF_HOME": str(runtime / "behavior-diff-home"), + } + ) + return env + + +def sandbox_command(profile, command): + return ["/usr/bin/sandbox-exec", "-f", str(profile), *command] + + +def preflight(workspace, root, runtime, home, denied, executable, profile, env): + canary_dir = Path(tempfile.mkdtemp(prefix=".codex-canary-", dir=root)) + link = workspace / (canary_dir.name + "-link") + checks = [] + try: + canary = canary_dir / "private" + canary.write_text("synthetic isolation canary\n") + link.symlink_to(canary) + # All operations run under the exact profile used for the model process. + script = """set -eu +printf probe > "$4.write" +/bin/cat "$4.write" >/dev/null +/bin/rm "$4.write" +printf probe > "$2/.runtime-probe" +/bin/cat "$2/.runtime-probe" >/dev/null +/bin/rm "$2/.runtime-probe" +if /bin/cat "$3" >/dev/null 2>&1; then exit 41; fi +if printf leak > "$3" 2>/dev/null; then exit 42; fi +if /bin/cat "$4" >/dev/null 2>&1; then exit 43; fi +shift 4 +for source in "$@"; do + if [ -d "$source" ]; then + if /bin/ls "$source" >/dev/null 2>&1; then exit 44; fi + else + if /bin/cat "$source" >/dev/null 2>&1; then exit 45; fi + fi +done +""" + result = subprocess.run( + sandbox_command( + profile, + [ + "/bin/sh", + "-c", + script, + "probe", + str(workspace), + str(runtime), + str(canary), + str(link), + *map(str, denied), + ], + ), + cwd=workspace, + env=env, + capture_output=True, + timeout=20, + ) + if result.returncode: + raise IsolationError( + "outer sandbox access probe failed (including nested sandbox denial); no model started" + ) + checks.extend( + [ + "workspace-read-write", + "runtime-read-write", + "private-sibling-denied", + "symlink-escape-denied", + ] + ) + if denied: + checks.append("protected-source-denied") + result = subprocess.run( + sandbox_command(profile, [str(executable), "--version"]), + cwd=workspace, + env=env, + capture_output=True, + timeout=20, + ) + if result.returncode or result.stdout.strip() != b"codex-cli 0.165.0": + raise IsolationError( + "isolated Codex must report codex-cli 0.165.0; no model started" + ) + checks.append("codex-runtime-version") + return checks + finally: + link.unlink(missing_ok=True) + Path(str(link) + ".write").unlink(missing_ok=True) + shutil.rmtree(canary_dir) + + +def copy_auth(runtime): + configured = Path( + os.environ.get("CODEX_HOME", str(Path.home() / ".codex")) + ).expanduser() + auth = configured / "auth.json" + if not auth.is_file(): + raise IsolationError( + "Codex auth.json is unavailable in the original CODEX_HOME; authenticate before isolated exec" + ) + target = runtime / "codex-home" / "auth.json" + # Credentials live only in the per-call private runtime, never in evidence. + with auth.open("rb") as source, target.open("xb") as destination: + os.chmod(target, 0o600) + shutil.copyfileobj(source, destination) + + +def run(args): + workspace, root, home, denied, executable = validate_paths( + args.workspace, + args.root, + args.deny_read, + args.codex, + ) + if sys.platform != "darwin": + raise IsolationError("isolated Codex requires macOS sandbox-exec") + if not os.access("/usr/bin/sandbox-exec", os.X_OK): + raise IsolationError( + "macOS sandbox-exec is unavailable; refusing unsandboxed execution" + ) + with tempfile.TemporaryDirectory(prefix=".codex-runtime-", dir=root) as temporary: + runtime = Path(temporary) + for name in ("home", "codex-home", "tmp", "behavior-diff-home"): + (runtime / name).mkdir(mode=0o700) + profile = runtime / "isolation.sb" + profile.write_text( + sandbox_profile(workspace, root, runtime, home, denied, executable.parent) + ) + env = isolated_environment(runtime) + checks = preflight( + workspace, root, runtime, home, denied, executable, profile, env + ) + if args.command == "probe": + print(json.dumps({"status": "pass", "checks": checks})) + return 0 + copy_auth(runtime) + command = [ + str(executable), + "--no-daemon", + "exec", + "--ignore-user-config", + "--ignore-rules", + "--ephemeral", + "--skip-git-repo-check", + "--json", + "--sandbox", + "danger-full-access", + "--model", + args.model, + ] + for setting in ( + "features.memories=false", + "features.plugins=false", + "features.apps=false", + "features.hooks=false", + "features.chronicle=false", + "features.daemon_auto_start=false", + "features.skill_search=false", + "features.shell_snapshot=false", + "features.multi_agent=false", + "features.multi_agent_v2=false", + "features.image_generation=false", + "features.view_image=false", + "features.computer_use=false", + "features.browser_use=false", + "features.browser_use_external=false", + "features.browser_use_full_cdp_access=false", + "features.in_app_browser=false", + "features.skip_host_skill_discovery=true", + 'web_search="disabled"', + ): + command.extend(["-c", setting]) + command.append("-") + child = subprocess.Popen( + sandbox_command(profile, command), + cwd=workspace, + env=env, + start_new_session=True, + ) + try: + return child.wait() + finally: + # Kill descendants before deleting auth, including on interruption. + try: + os.killpg(child.pid, signal.SIGKILL) + except ProcessLookupError: + pass + child.wait() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + subparsers = parser.add_subparsers(dest="command", required=True) + for name in ("probe", "exec"): + command = subparsers.add_parser(name) + command.add_argument("--workspace", required=True) + command.add_argument("--root", required=True) + command.add_argument("--deny-read", action="append", default=[]) + command.add_argument("--codex", default="codex") + if name == "exec": + command.add_argument("--model", required=True) + args = parser.parse_args() + if args.command == "exec" and ( + not args.model.strip() or args.model != args.model.strip() + ): + parser.error("--model must be an exact nonempty model ID") + + def interrupted(signum, frame): + raise IsolationError("interrupted; private runtime is being removed") + + for signum in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP): + signal.signal(signum, interrupted) + try: + return run(args) + except (IsolationError, OSError, subprocess.SubprocessError) as error: + # Do not include arbitrary subprocess output or credential paths. + message = ( + str(error) if isinstance(error, IsolationError) else type(error).__name__ + ) + print("codex-isolated: " + message, file=sys.stderr) + if args.command == "probe": + print(json.dumps({"status": "failure", "checks": [], "error": message})) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugin/skills/behavior-diff/scripts/prepare_shared.py b/plugin/skills/behavior-diff/scripts/prepare_shared.py index b7f6d0e..ddfbf38 100755 --- a/plugin/skills/behavior-diff/scripts/prepare_shared.py +++ b/plugin/skills/behavior-diff/scripts/prepare_shared.py @@ -142,6 +142,14 @@ def handoff(path, project, working, root, data=None): "files": inventory(files), "private": private_paths, "settings": settings, + "codex_isolation": { + name: os.environ[name] + for name in ( + "BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT", + "BEHAVIOR_DIFF_CODEX_DENY_READ", + ) + if os.environ.get(name) + }, "argv": argv, } @@ -620,6 +628,16 @@ def verify(root, as_json=False): raise PreparationError("prepared files changed after freeze") context = frozen.get("context") if context: + isolation = { + name: os.environ[name] + for name in ( + "BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT", + "BEHAVIOR_DIFF_CODEX_DENY_READ", + ) + if os.environ.get(name) + } + if isolation != context.get("codex_isolation", {}): + raise PreparationError("Codex isolation settings changed after freeze") for name in context["private"]: private_file(name) for name, expected in context["files"].items(): diff --git a/plugin/skills/behavior-diff/scripts/run-trial.sh b/plugin/skills/behavior-diff/scripts/run-trial.sh index c02cdca..c01fb29 100755 --- a/plugin/skills/behavior-diff/scripts/run-trial.sh +++ b/plugin/skills/behavior-diff/scripts/run-trial.sh @@ -129,10 +129,19 @@ if [ "$agent" = claude ]; then --output-format stream-json --verbose >"$trace_dir/trace.jsonl" 2>"$trace_dir/stderr.log" || cli_exit=$? record_status "{\"cli_exit_code\":$cli_exit}" elif [ "$agent" = codex ]; then - # Codex has no per-tool allowlist; the workspace-write sandbox scoped to - # this variant copy is the equivalent containment. - codex exec --ephemeral --skip-git-repo-check -s workspace-write \ - -m "$model" --json "$task" "$trace_dir/codex-raw.jsonl" 2>"$trace_dir/stderr.log" || cli_exit=$? + if [ -n "${BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT:-}" ]; then + isolation_args=(exec --workspace "$PWD" --root "$BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT" --model "$model") + if [ -n "${BEHAVIOR_DIFF_CODEX_DENY_READ:-}" ]; then + isolation_args+=(--deny-read "$BEHAVIOR_DIFF_CODEX_DENY_READ") + fi + printf '%s\n' "$task" | + python3 "$scripts/codex_isolated.py" "${isolation_args[@]}" \ + >"$trace_dir/codex-raw.jsonl" 2>"$trace_dir/stderr.log" || cli_exit=${PIPESTATUS[1]} + else + # Codex has no per-tool allowlist; use its normal workspace-write sandbox. + codex exec --ephemeral --skip-git-repo-check -s workspace-write \ + -m "$model" --json "$task" "$trace_dir/codex-raw.jsonl" 2>"$trace_dir/stderr.log" || cli_exit=$? + fi record_status "{\"phase\":\"normalizing\",\"cli_exit_code\":$cli_exit}" jq -c 'select(.type == "item.completed" and (.item.item_type // .item.type) == "command_execution") diff --git a/tests/codex-isolation-test.py b/tests/codex-isolation-test.py new file mode 100755 index 0000000..b2f2cfd --- /dev/null +++ b/tests/codex-isolation-test.py @@ -0,0 +1,343 @@ +#!/usr/bin/env python3 +"""Synthetic CLI contracts; no model, network client, or real credentials are used.""" + +import json +import os +import shlex +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +HELPER = ( + Path(__file__).resolve().parents[1] + / "plugin/skills/behavior-diff/scripts/codex_isolated.py" +) +SANDBOX = Path("/usr/bin/sandbox-exec") +MODEL = "synthetic-exact-model-2099" + +# This executable only runs authored shell instructions. It cannot call a model. +CODEX_STUB = r"""#!/bin/sh +if [ "$1" = "--version" ]; then + if [ -e "$CODEX_HOME/auth.json" ]; then + printf 'Credentials reached non-model preflight\n' >&2 + exit 87 + fi + printf 'codex-cli 0.165.0\n' + exit 0 +fi +printf '%s\n' "$@" > "$PWD/invocation-args" +printf '%s\n' "$HOME" > "$PWD/isolated-home" +printf '%s\n' "$CODEX_HOME" > "$PWD/isolated-codex-home" +if [ -n "${OPENAI_API_KEY+x}" ] || [ -n "${SYNTHETIC_PRIVATE_ENV+x}" ]; then + printf 'Inherited private environment\n' >&2 + exit 88 +fi +if [ -f "$CODEX_HOME/auth.json" ]; then + printf 'present\n' > "$PWD/auth-observation" +else + printf 'absent\n' > "$PWD/auth-observation" +fi +/bin/sh -s +status=$? +if [ "$status" -ne 0 ]; then + printf 'Synthetic child failure\n' >&2 + exit "$status" +fi +printf '%s\n' '{"type":"item.completed","item":{"id":"synthetic-command","type":"command_execution","command":"synthetic sandbox file checks","aggregated_output":"file checks passed","exit_code":0,"status":"completed"}}' +""" + + +def quote(path): + return shlex.quote(str(path)) + + +class IsolationHarness(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix="codex-isolation-test-") + self.addCleanup(self.temporary.cleanup) + self.base = Path(self.temporary.name).resolve() + self.home = self.base / "original-home" + self.home.mkdir() + self.auth_home = self.home / "configured-codex" + self.auth_home.mkdir() + self.auth = self.auth_home / "auth.json" + self.auth.write_text('{"synthetic_auth":"not-a-real-credential"}\n') + self.root = self.base / "private" + self.root.mkdir(mode=0o700) + self.workspace = self.root / "workspace" + self.workspace.mkdir() + self.source = self.base / "protected-source" + self.source.mkdir() + self.source_canary = self.source / "source-canary" + self.source_canary.write_text("synthetic source canary\n") + self.home_canary = self.home / "home-canary" + self.home_canary.write_text("synthetic home canary\n") + self.sibling = self.root / "other-trial" + self.sibling.mkdir() + self.sibling_canary = self.sibling / "private-canary" + self.sibling_canary.write_text("synthetic private canary\n") + self.purpose_canary = self.root / "purpose" + self.purpose_canary.write_text("synthetic private purpose\n") + self.bin = self.base / "synthetic-runtime" / "bin" + self.bin.mkdir(parents=True) + self.codex = self.bin / "codex" + self.codex.write_text(CODEX_STUB) + self.codex.chmod(0o755) + self.env = { + "HOME": str(self.home), + "CODEX_HOME": str(self.auth_home), + "PATH": f"{self.bin}:/usr/bin:/bin:/usr/sbin:/sbin", + "TMPDIR": str(self.base), + "OPENAI_API_KEY": "synthetic-never-use", + "SYNTHETIC_PRIVATE_ENV": "must-not-reach-child", + } + + def cli(self, mode="exec", *, prompt="", changes=None, prefix=(), extra=()): + options = { + "--workspace": str(self.workspace), + "--root": str(self.root), + "--deny-read": str(self.source), + "--codex": str(self.codex), + } + options.update(changes or {}) + arguments = [sys.executable, str(HELPER), mode] + for flag, value in options.items(): + arguments.extend((flag, value)) + if mode == "exec": + arguments.extend(("--model", MODEL)) + arguments.extend(extra) + return subprocess.run( + [*prefix, *arguments], + input=prompt, + capture_output=True, + text=True, + env=self.env, + cwd=self.workspace, + timeout=30, + check=False, + ) + + def assert_not_invoked(self): + self.assertFalse((self.workspace / "invocation-args").exists()) + + def assert_cleaned(self): + self.assertEqual(list(self.root.glob(".codex-runtime-*")), []) + self.assertEqual(list(self.root.glob(".codex-canary-*")), []) + self.assertEqual(list(self.root.rglob("auth.json")), []) + self.assertEqual( + self.auth.read_text(), '{"synthetic_auth":"not-a-real-credential"}\n' + ) + + def assert_rejected(self, changes, diagnostic): + result = self.cli(changes=changes) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertEqual(result.stdout, "") + self.assertIn(diagnostic, result.stderr.lower()) + self.assert_not_invoked() + self.assert_cleaned() + + +class IsolationCLI(IsolationHarness): + def test_reject_root_equal_to_or_inside_workspace(self): + for root in (self.workspace, self.workspace / "nested-private"): + with self.subTest(root=root): + self.assert_rejected({"--root": str(root)}, "workspace") + + def test_reject_workspace_inside_denied_source(self): + self.assert_rejected({"--workspace": str(self.source)}, "deny") + + def test_reject_nonprivate_root(self): + self.root.chmod(0o755) + self.assert_rejected({}, "permission") + + def test_reject_relative_denied_path(self): + self.assert_rejected({"--deny-read": "relative-source"}, "absolute") + + def test_reject_symlink_root_and_workspace(self): + for flag, destination in ( + ("--root", self.root), + ("--workspace", self.workspace), + ): + link = self.base / (flag.removeprefix("--") + "-link") + link.symlink_to(destination, target_is_directory=True) + with self.subTest(flag=flag): + self.assert_rejected({flag: str(link)}, "symlink") + + def test_reject_runtime_parent_overlapping_protected_source(self): + executable = self.source / "codex" + executable.write_text(CODEX_STUB) + executable.chmod(0o755) + self.assert_rejected({"--codex": str(executable)}, "runtime") + + def test_exec_requires_exact_model_argument(self): + result = subprocess.run( + [ + sys.executable, + str(HELPER), + "exec", + "--workspace", + str(self.workspace), + "--root", + str(self.root), + "--codex", + str(self.codex), + ], + capture_output=True, + text=True, + env=self.env, + timeout=30, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("--model", result.stderr) + self.assert_not_invoked() + + @unittest.skipIf(sys.platform == "darwin", "non-macOS failure contract") + def test_unsupported_platform_fails_without_child(self): + result = self.cli() + self.assertNotEqual(result.returncode, 0) + self.assertEqual(result.stdout, "") + self.assertIn("macos", result.stderr.lower()) + self.assert_not_invoked() + self.assert_cleaned() + + +@unittest.skipUnless(sys.platform == "darwin", "requires macOS sandbox-exec") +class MacOSIsolationCLI(IsolationHarness): + def test_probe_is_non_model_and_credential_free(self): + result = self.cli("probe") + self.assertEqual(result.returncode, 0, result.stderr) + payload = json.loads(result.stdout) + self.assertEqual(payload["status"], "pass") + self.assertTrue(payload["checks"]) + self.assertLess(len(result.stdout), 8192) + self.assert_not_invoked() + self.assert_cleaned() + + def test_runtime_and_workspace_can_be_canonicalized(self): + script = ( + "import os, pathlib; " + "paths = [os.environ['CODEX_HOME'], os.getcwd()]; " + "assert all(os.path.realpath(p, strict=True) == p for p in paths); " + "pathlib.Path('canonicalized').write_text('runtime and workspace')" + ) + result = self.cli( + prompt=f"set -eu\n{quote(sys.executable)} -c {quote(script)}\n" + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + (self.workspace / "canonicalized").read_text(), "runtime and workspace" + ) + self.assert_cleaned() + + def file_checks_prompt(self): + lines = [ + "set -eu", + "printf 'workspace-write\\n' > allowed-file", + 'test "$(/bin/cat allowed-file)" = workspace-write', + 'printf "runtime-write\\n" > "$HOME/runtime-file"', + 'test "$(/bin/cat "$HOME/runtime-file")" = runtime-write', + 'printf "codex-write\\n" > "$CODEX_HOME/runtime-file"', + 'test "$(/bin/cat "$CODEX_HOME/runtime-file")" = codex-write', + ] + for label, path in ( + ("home", self.home_canary), + ("source", self.source_canary), + ("sibling", self.sibling_canary), + ("purpose", self.purpose_canary), + ): + link = self.workspace / f"{label}-escape" + link.symlink_to(path) + for candidate in (path, link): + lines.extend( + ( + f"if /bin/cat {quote(candidate)} >/dev/null 2>&1; then exit 81; fi", + f"if (printf changed > {quote(candidate)}) 2>/dev/null; then exit 82; fi", + ) + ) + return "\n".join(lines) + "\n" + + def test_real_profile_allows_workspace_and_denies_private_reads_and_writes(self): + result = self.cli(prompt=self.file_checks_prompt()) + self.assertEqual(result.returncode, 0, result.stderr) + events = [json.loads(line) for line in result.stdout.splitlines()] + self.assertEqual(len(events), 1) + self.assertEqual(events[0]["item"]["type"], "command_execution") + self.assertEqual(events[0]["item"]["exit_code"], 0) + self.assertEqual( + (self.workspace / "allowed-file").read_text(), "workspace-write\n" + ) + self.assertEqual(self.source_canary.read_text(), "synthetic source canary\n") + self.assertEqual(self.home_canary.read_text(), "synthetic home canary\n") + self.assertEqual(self.sibling_canary.read_text(), "synthetic private canary\n") + arguments = (self.workspace / "invocation-args").read_text().splitlines() + self.assertEqual(arguments[arguments.index("--model") + 1], MODEL) + for flag in ( + "--ignore-user-config", + "--ignore-rules", + "--ephemeral", + "--skip-git-repo-check", + "--json", + ): + self.assertIn(flag, arguments) + self.assertIn("exec", arguments) + self.assertEqual((self.workspace / "auth-observation").read_text(), "present\n") + self.assertNotEqual( + (self.workspace / "isolated-home").read_text().strip(), str(self.home) + ) + self.assert_cleaned() + + def test_repeated_denied_paths_remain_protected(self): + additional = self.base / "additional-source" + additional.mkdir() + canary = additional / "canary" + canary.write_text("synthetic additional source\n") + prompt = ( + "set -eu\n" + f"if /bin/cat {quote(canary)} >/dev/null 2>&1; then exit 83; fi\n" + f"if (printf changed > {quote(canary)}) 2>/dev/null; then exit 84; fi\n" + ) + result = self.cli(prompt=prompt, extra=("--deny-read", str(additional))) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(canary.read_text(), "synthetic additional source\n") + self.assert_cleaned() + + def test_root_reuse_creates_fresh_runtime_for_every_exec(self): + homes = [] + for _ in range(2): + result = self.cli(prompt="exit 0\n") + self.assertEqual(result.returncode, 0, result.stderr) + homes.append((self.workspace / "isolated-codex-home").read_text().strip()) + self.assert_cleaned() + self.assertNotEqual(*homes) + + def test_auth_and_runtime_removed_after_child_failure(self): + result = self.cli(prompt="exit 17\n") + self.assertEqual(result.returncode, 17, result.stderr) + self.assertEqual(result.stdout, "") + self.assertIn("Synthetic child failure", result.stderr) + self.assertEqual((self.workspace / "auth-observation").read_text(), "present\n") + runtime_home = Path( + (self.workspace / "isolated-codex-home").read_text().strip() + ) + self.assertFalse(runtime_home.exists()) + self.assert_cleaned() + + def test_failed_same_profile_preflight_never_invokes_model_command(self): + profile = '(version 1)(allow default)(deny process-exec (literal "/bin/sh"))' + result = self.cli( + prompt="printf should-not-run > forbidden-invocation\n", + prefix=(str(SANDBOX), "-p", profile), + ) + self.assertNotEqual(result.returncode, 0) + self.assertEqual(result.stdout, "") + self.assertTrue(result.stderr) + self.assert_not_invoked() + self.assertFalse((self.workspace / "forbidden-invocation").exists()) + self.assert_cleaned() + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/shared-data-test.py b/tests/shared-data-test.py index 3ea633f..957cb40 100755 --- a/tests/shared-data-test.py +++ b/tests/shared-data-test.py @@ -465,6 +465,25 @@ def test_complete_staged_handoff_rejects_every_context_and_settings_mutation(sel Path(record["notes_file"]).chmod(0o644) self.assertNotEqual(self.verify(stage).returncode, 0) + def test_frozen_handoff_rejects_isolation_environment_changes(self): + project, data = self.standard() + self.env["BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT"] = str(self.tmp / "runtime") + self.env["BEHAVIOR_DIFF_CODEX_DENY_READ"] = str(project) + stage = self.staged(project, data, handoff=self.handoff_file()) + result = self.verify(stage) + self.assertEqual(result.returncode, 0, result.stderr) + for name in ( + "BEHAVIOR_DIFF_CODEX_ISOLATION_ROOT", + "BEHAVIOR_DIFF_CODEX_DENY_READ", + ): + original = self.env.pop(name) + self.assertNotEqual(self.verify(stage).returncode, 0) + self.env[name] = original + "-different" + self.assertNotEqual(self.verify(stage).returncode, 0) + self.env[name] = original + result = self.verify(stage) + self.assertEqual(result.returncode, 0, result.stderr) + def test_direct_freeze_uses_archive_not_unrelated_work_and_rejects_stale_after( self, ):