Strengthen verification evidence and native context checks

.github/workflows/ci.yml

@@ -89,6 +89,8 @@ jobs:
             plugin/skills/behavior-diff/scripts/source.py \
             plugin/skills/behavior-diff/scripts/ci_context.py \
             plugin/skills/behavior-diff/scripts/prepare_shared.py \
+            plugin/skills/behavior-diff/scripts/native_context.py \
+            plugin/skills/behavior-diff/scripts/preparation_evidence.py \
             plugin/skills/behavior-diff/scripts/reporting/*.py
 
       - name: Run hook unit checks
@@ -106,8 +108,11 @@ jobs:
           python3 tests/ci-context-test.py
           python3 tests/report-ci-context-test.py
 
-      - name: Check shared-data preparation boundaries
-        run: python3 tests/shared-data-test.py
+      - name: Check preparation and evidence boundaries
+        run: |
+          python3 tests/shared-data-test.py
+          python3 tests/preparation-evidence-test.py
+          python3 tests/native-context-boundary-test.py
 
       - name: Check isolated Codex launcher without models
         run: python3 tests/codex-isolation-test.py

docs/architecture.md

@@ -396,8 +396,8 @@ shared-data stage.
 The private helper
 [`prepare_shared.py`](../plugin/skills/behavior-diff/scripts/prepare_shared.py)
 delivers a reviewed data directory to every trial without changing the user's
-checkout or adding runner options. It only delivers and verifies: the skill
-designs the data, and the helper does not generate data, launch models, or infer
+checkout. It only delivers and verifies: the skill designs the data, and the helper
+does not generate data, launch models, or infer
 provenance. The optional private `--handoff` descriptor names task, purpose,
 scenario, optional source, notes, required references, custom tools and settings.
 It reuses the existing input readers and freezes those inputs plus shipped
@@ -455,6 +455,31 @@ existing runner only after re-verification in the same environment. Semantic
 capability probes, source closure and privacy review remain with the agent;
 digests cannot prove that a probe ran or succeeded.
 
+Optional private `preparation_evidence` binds native preparer records and the
+reviewed input file to this inventory. `preparation_evidence.py` checks the
+supported Codex native export for one complete, non-inherited legacy-history
+turn with matching plain-text user input. Changed or missing files fail verify.
+Unsupported or absent records remain unavailable/unverified. The native thread's
+model field is not per-turn execution telemetry: neither a resolved model nor the
+entire historical context is claimed verified. These private records never become
+trial inputs or assessment content; validation does not authenticate a fabricated
+export.
+
+Optional `context_requirements_file` freezes a declared native filesystem
+restriction separately from preparation provenance. The existing runner prepares
+trial snapshots and checks the selected native policy before starting any model
+worker. Unsupported hosts, conflicting configuration or failed access canaries
+stop launch rather than silently changing routes. The ordinary route remains
+unchanged when this contract is absent; it provides no new isolation claim.
+
+The per-trial private marker records enforcement and observed forbidden-path
+attempts. `reporting/context_boundary.py` is the shared eligibility rule:
+violations or unavailable required markers exclude raw trial evidence from the
+assessment and eligible counts. Successful filesystem canaries do not establish
+clean injected, connector, browser or cloud context; an unobserved read attempt
+is not proof of its absence.
+A prepared trial remains ineligible until post-trial observation finishes.
+
 The local [`audit_evidence.py`](../plugin/skills/behavior-diff/scripts/audit_evidence.py)
 indexes every expected trial and every original normalized/native JSONL event
 with exact line, byte and tool-part pointers, preserving duplicate identities,
@@ -473,11 +498,14 @@ One trial is shown below. The coordinator repeats this sequence for all
 ```text
 behavior-diff.sh        run-trial.sh          Agent CLI
        |                     |                     |
+       | stage all snapshots |                     |
+       | optional native filesystem preflight      |
        |-- start ----------->|                     |
        |                     |-- task + model ---->|
        |                     |< host events -------|
        |                     |                     |
        |                     | capture/normalize trace.jsonl
+       |                     | optional boundary marker
        |< process ends ------|
        |
        | wait for all 2 x N trials
@@ -491,8 +519,8 @@ behavior-diff.sh        run-trial.sh          Agent CLI
 The modules below live in the skill's [`scripts/`](../plugin/skills/behavior-diff/scripts/) directory.
 
 - **Model-based interpretation:** `decisions.py` reads the task, trial actions,
-  final answers, numbered instruction-diff hunks, frozen purpose, and bounded
-  recorded returns. Extractor prompts travel over stdin, not command-line
+  final answers, numbered instruction-diff hunks, frozen purpose, bounded recorded
+  returns and explicitly declared decision artifacts. Extractor prompts travel over stdin, not command-line
   arguments, so large comparisons do not hit the operating system's per-argument
   size limit. The same extraction call defines observable target criteria,
   required evidence, and per-trial outcomes (`met`, `not_met`, `uncertain`, or
@@ -556,9 +584,10 @@ The modules below live in the skill's [`scripts/`](../plugin/skills/behavior-dif
   compares recorded command sequences. `reporting/instruction.py` supplies the
   instruction diff. `reporting/content.py` derives shared wording and evidence
   limits. `reporting/target.py` validates purpose, criteria, exact completed-trial
-  membership, trial-local source references, and contiguous final-answer excerpts.
-  Source consistency needs a final-answer excerpt and readable recorded sources
-  matching every frozen `source_requirements` group for that check. Alternatives
+  membership, trial-local source references, and contiguous output excerpts.
+  Output can come from the final answer or declared captured decision artifacts.
+  Source consistency also requires readable recorded sources matching every frozen
+  `source_requirements` group for that check. Alternatives
   within a group are OR; groups are AND. Descriptor matching constrains provenance,
   not semantic truth. A trustworthy captured call can establish that a call was
   made even when its return was omitted; it cannot establish source agreement.
@@ -616,8 +645,9 @@ distinct. Links between decisions and edits do not prove causality.
 
 ### Recorded evidence bounds
 
-`reporting/evidence.py` selects only recorded read/search/list returns and
-conservatively recognized read-only shell commands. Stable IDs use the trial
+`reporting/evidence.py` selects recorded read/search/list returns,
+conservatively recognized read-only shell commands, and explicitly declared saved
+decision artifacts. Stable tool IDs use the trial
 name and recorded tool-call ordinal. Source descriptors retain supplied ranges;
 excerpt ranges are character offsets in returned text, **not file line numbers**.
 Text blocks are joined with newlines; images are not sent as text evidence.
@@ -632,6 +662,16 @@ tools. API request-body flags, mutating methods, compound commands, errors and
 sensitive content remain excluded. This selects already-recorded text; it never
 invokes `gh` or fetches a source.
 
+Optional frozen scenario `artifact_requirements` bind output checks to at most
+sixteen exact project-relative paths. Only those retained trial-project files are
+read; there is no general filesystem scrape. No-follow directory-relative opens
+reject symlinks, hardlinks and nonregular files. Files over 64 KiB are omitted;
+bounded UTF-8 content is filtered before sharing the existing evidence budgets.
+Artifact records identify captured bytes by SHA-256, not proposed write input,
+authorship or a proven read action. Missing required artifacts prevent determinate
+judgments even when the final answer claims completion. Source-consistency checks
+still need their separately captured authoritative source returns.
+
 Limits are 4,096 Unicode characters per excerpt, 16,384 per trial, 131,072 per
 run, and 128 tool records per trial. Selection uses the saved `grades.tsv`
 inventory (directory fallback only without that inventory) and reserves a share

plugin/skills/behavior-diff/references/scenario-and-test-data.md

@@ -26,6 +26,34 @@ report the missing isolation prerequisite before deriving or freezing purpose.
 If case selection or tooling was informed by hindsight, still disclose that
 limit: isolation does not remove historical selection bias.
 
+Before deriving historical purpose, check whether the chosen fresh host can
+retain native supplied-input and completion records and expose the required
+context controls. Do not discover missing provenance only after making a claim
+of compliant preparation, or replace missing fresh preparation with a session
+that already knows the outcomes.
+
+After preparation, the private handoff may include `preparation_evidence` with
+`host`, `record_file`, `input_file`, `thread_id`, and `turn_id`. The files must
+be private and outside trial inputs. `input_file` is the reviewed plain-text
+preparer prompt; `record_file` is the host-native export, not an agent-written
+summary. Freeze binds both files by digest and verify rejects changed or missing
+records.
+
+The Codex adapter validates a native `ThreadReadResponse` containing one complete,
+non-inherited legacy-history turn and one matching plain-text user input.
+Partial, paginated, forked, compacted, ambiguous or failed records do not establish
+those facts. Unsupported host formats remain explicitly unavailable.
+
+These checks establish only the supplied user input and native completion
+represented in the provided record. They do not authenticate an arbitrary export
+or enumerate injected system context. Codex's thread model is configured/latest
+metadata, not per-turn execution telemetry: resolved execution model and the
+complete historical context boundary remain unavailable. Missing references
+remain unverified. Never turn those unavailable facets into a compliant-preparation
+claim; when the requested claim requires them, report the missing prerequisite.
+Native records and reviewed prompts stay out of trial copies, assessment input
+and public reports.
+
 1. **Goals.** Start from the frozen purpose goals. Name new behavior and
    behavior that must survive the change. Mark preservation goals as inferred
    unless stated.
@@ -91,8 +119,8 @@ limit: isolation does not remove historical selection bias.
      characters; `required_evidence` names what decides the check.
    - `evidence_kind` names exactly one kind of evidence: `recorded_action` for
      a recorded tool call such as a required code read, `output` for content
-     of the final answer, and `source_consistency` for a final answer that
-     must agree with recorded returned content. Split a compound requirement
+     of the final answer or a declared decision artifact, and `source_consistency`
+     for that output's agreement with recorded returned content. Split a compound requirement
      into separate checks. For example, "reads the code and reports the value
      it finds" becomes a `recorded_action` check for the read, an `output`
      check for the report, and a `source_consistency` check for agreement.
@@ -134,6 +162,29 @@ limit: isolation does not remove historical selection bias.
      before incidental reads, so keep required sources small; many large
      required returns can still exhaust that budget.
 
+   - Optional `artifact_requirements` selects saved decision files that an
+     `output` or `source_consistency` check needs:
+     `{"check":"reports-total","paths":["deliverables/decision.md"]}`.
+     Each check appears at most once; all its listed paths are required.
+     Declare these paths before trials, include the requested outputs in the
+     neutral task, and obtain authority for sending their contents to assessment.
+     Select at most sixteen distinct exact project-relative paths, each at most
+     120 ASCII characters. No globs, absolute paths, aliases, hidden paths, or
+     private evaluator, credential, or secret files are permitted.
+
+     The collector reads only these declared files from each retained trial
+     project. It rejects symlinks, hardlinks and nonregular files. UTF-8 files
+     over 64 KiB are omitted; eligible contents pass the existing sensitive-text
+     filter and share the existing assessment budgets. Missing, omitted and
+     truncated evidence remains explicit. A final-answer completion claim cannot
+     substitute for a missing required artifact.
+
+     Artifact records identify captured bytes and their SHA-256 digest, not who
+     wrote them or which tool read them. They cannot prove a `recorded_action`
+     or satisfy a required source-read return. For `source_consistency`, retain
+     the separate authoritative `source_requirements`. Select only outputs
+     reviewed as suitable for disclosure; filtering is not a secrets audit.
+
    Use plain text: no Markdown, links, or control characters. Generalize
    private detail as for purpose: no names, credentials, private URLs, source
    excerpts, raw notes, or later history. Never record outcomes, per-trial
@@ -168,11 +219,12 @@ attribute a difference solely to the selected rule.
 
 If the request explicitly requires a strict incremental-rule claim, identify the
 additional influences that must be excluded and establish those boundaries before
-launch. The optional [macOS Codex helper](#isolated-codex-decision-comparisons-on-macos)
-applies only when filesystem isolation is explicitly required and its documented
-boundary fits the required capabilities. It is not the normal route and never an
-automatic fallback for a runner or sandbox failure. A filesystem boundary alone
-does not establish causal isolation from every model or host influence.
+launch. For a required Codex local-filesystem boundary, use the existing runner's
+native permission-profile contract below when its capabilities fit. The optional
+[macOS Codex helper](#isolated-codex-decision-comparisons-on-macos) remains an
+explicitly selected, separately scoped route, never an automatic fallback.
+A filesystem boundary alone does not establish causal isolation from every
+model or host influence.
 
 If a required boundary or capability is unavailable, report that precise blocker.
 Do not weaken host permissions, invent a launcher, or silently substitute a
@@ -181,6 +233,40 @@ not block an ordinary supported-environment comparison solely because it cannot
 support a strict causal claim. Historical fresh-context preparation above remains
 required on either route; disclosing host exposure does not waive it.
 
+### Required native filesystem boundary
+
+When the agreed comparison requires excluding historical files from trial tool
+reads, freeze a private `context_requirements_file` in the handoff. Its JSON is:
+
+```json
+{"version":1,"scope":"codex-local-filesystem","deny_read":["/private/synthetic/historical-context"]}
+```
+
+Use 1–64 exact canonical absolute file or subtree paths, not globs or relative
+paths. Do not include private paths in public reports. The verified handoff
+passes this file to the existing runner; it does not add another executor.
+The supported route is native Codex permission profiles on macOS. It permits
+the native minimal runtime and trial workspace, explicitly denies the nominated
+paths, disables network, and never combines that profile with the legacy
+`--sandbox` override or approval escalation.
+
+Before any model worker starts, the runner checks configuration and exercises
+actual allowed workspace access and prohibited-read canaries in staged trial
+workspaces. Unsupported hosts, configuration conflicts and failed canaries stop
+launch. This is not approval to change global permissions or switch to Full
+Access. The minimal runtime does not automatically expose Homebrew or other
+host-installed tools: exercise the scenario's required commands under the same
+policy during semantic preflight, or report the missing capability.
+
+The private `context-boundary.json` records enforcement and any observed attempt
+to use a nominated forbidden path. Required missing/invalid markers and recorded
+violations exclude the trial's raw content from assessment and eligible counts.
+A denied attempt does not prove a successful read; conservative exclusion keeps
+that ambiguity visible. No observed attempt is not proof of clean context.
+Injected instructions, connectors, browser/cloud state and other non-filesystem
+channels remain unverified. Establish any additional required boundary separately;
+do not silently substitute detection-only checks for prevention.
+
 
 ## Preflight