From 70e389b45b138931aefbd65fc8cd163061df2d73 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 29 Oct 2026 23:11:12 +0800 Subject: [PATCH] fix: complete supported behavior rehearsals without false blockers Scope capabilities to owner claims, validate complete paginated Codex preparation records, and clarify controlled fresh preparation without weakening evidence or sandbox boundaries. Refs: DRC-5022 Signed-off-by: Kent Huang --- docs/architecture.md | 49 +++- e2e/README.md | 26 ++ plugin/skills/behavior-diff-live/SKILL.md | 8 + plugin/skills/behavior-diff/SKILL.md | 39 ++- .../references/scenario-and-test-data.md | 224 ++++++++++++++---- .../skills/behavior-diff/scripts/decisions.py | 37 +-- .../scripts/preparation_evidence.py | 69 +++++- plugin/skills/behavior-rehearse/SKILL.md | 36 ++- .../behavior-rehearse/references/execution.md | 83 +++++-- tests/preparation-evidence-test.py | 81 +++++++ tests/shared-data-test.py | 18 +- 11 files changed, 552 insertions(+), 118 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 549d759..635e047 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -470,13 +470,28 @@ digests cannot prove that a probe ran or succeeded. Optional private `preparation_evidence` binds native preparer records and the reviewed input file to this inventory. `preparation_evidence.py` checks the -supported Codex native export for one complete, non-inherited legacy-history -turn with matching plain-text user input. Changed or missing files fail verify. +supported Codex native export for one complete, non-inherited turn with matching +plain-text user input. Legacy exports carry full turns directly; paginated +exports retain metadata plus every unfiltered full-item turn-page request and +response through the terminal cursor. Incomplete chains, summary-only items and +additional turns fail validation. Changed or missing files fail verify. Unsupported or absent records remain unavailable/unverified. The native thread's model field is not per-turn execution telemetry: neither a resolved model nor the entire historical context is claimed verified. These private records never become trial inputs or assessment content; validation does not authenticate a fabricated export. +Ordinary historical rehearsal requires a fresh supplied history and reviewed +contemporary inputs, not attestation of every hidden context surface. Verified +input/completion facets support that bounded route despite the overall unverified +label; unavailable model/ambient-context facets remain disclosed. Invalid input, +inherited history or failed completion still blocks preparation. A stronger +isolation or model-attestation claim requires the owner's explicit goal and its +separate capabilities, rather than being inferred from a PR request. +Codex preparation explicitly disables memory, hooks, plugins, apps, multi-agent +features, web search and each configured MCP server for that process, verifying +effective configuration before a model turn. An empty MCP table is not a reset. +The private record keeps only sanitized control facts, not credential-bearing +configuration; these controls do not attest every hidden context surface. Optional `context_requirements_file` freezes a declared native filesystem restriction separately from preparation provenance. The existing runner prepares @@ -611,6 +626,11 @@ The modules below live in the skill's [`scripts/`](../plugin/skills/behavior-dif rewritten definitions still invalidate the whole target; `decisions.target_rejection` is reserved for that global failure. Extraction keeps the untouched raw response and reports `target_partially_rejected` or `target_rejected` as appropriate. + The extraction schema emits the decision chain before criteria reference it. + Criteria use final chain indexes and trial-local citations; action checks leave + output excerpts empty rather than quoting narrative as evidence of a call. + Output/source-consistency checks still require exact excerpts and the same + provenance validation. No invalid citation is repaired or silently accepted. With a scenario record, the complete frozen check set is mandatory. The report loader binds it to configuration, not model output; a conflicting persisted record or purpose invalidates the assessment without dropping observations. @@ -1073,6 +1093,10 @@ Behavior Diff owns the distinction between supported-environment findings and stronger isolation claims, including historical preparation in a fresh context without prior discussion or outcomes. Required safety or evidence boundaries cannot be waived merely to complete a journey. +If a sandboxed command tool cannot start a nested native sandbox, the host's +authorized command-approval mechanism owns execution access to the unchanged +preparation or runner command. Child sandboxes remain enabled. The skill does not +enable automatic approval or grant OS permission from model-run consent. The handoff reuses existing authorization and requests only missing terms. When the goal is ambiguous, it first inspects the change, then offers 2–4 @@ -1087,12 +1111,21 @@ independent review. Behavior Diff supplies the capability gap and supported recovery; the handoff carries it through within authority or explains the remaining decision. It does not build a replacement launcher. -The agreed question determines whether a simulation is sufficient or a live -service is required. The first final answer leads with what to do with the -change and why, followed by decision-changing risk and remaining required -evidence gaps. Non-decision-changing methodology and detailed evidence stay in -private links. A useful null result can complete the journey; missing required -evidence or failed execution cannot. +Behavior Diff maps the owner's question to claims before requiring capabilities. +Ordinary instruction rehearsals test agent decisions using controlled modeled +external services, disclosed in the initial proposal. Production instructions +requiring an artifact or upload do not alone require real generation or a live +service in the test. Real output quality, visual recognition and live integration +claims still require their corresponding capabilities; they cannot be replaced +by recorded simulated decisions. Missing capabilities block dependent claims, +not independent authorized behavioral work. An approved real-output scope needs +an explicit revised proposal before substitution. + +The first final answer leads with what to do with the change and why, followed +by decision-changing risk and remaining required evidence gaps. +Non-decision-changing methodology and detailed evidence stay in private links. +A useful null result can complete the journey; missing required evidence or +failed execution cannot. Explicit replay-only requests use saved evidence without new trials. Optional `replay.py` owns private snapshots, correction invalidation, confirmed revisions diff --git a/e2e/README.md b/e2e/README.md index 8c6275a..ebceeed 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -59,6 +59,32 @@ An actual owner conversation must still exercise correction, evidence inspection and a gap, and record real takeover points versus their baseline privately. Helper checks alone do not establish acceptance, handoff value or live latency. +### Capability-scope regression walkthrough + +After approval for model cost and data use, exercise these goal variants with +synthetic instruction changes on each supported plugin host. These are manual +acceptance checks, not CI or proof supplied by the replay helper: + +- **Ordinary behavioral rehearsal:** a change requires preparing an archive, + inspecting checksum results and waiting for human approval. The host lacks the + production archive service. Expect a proposal that names controlled simulation, + then preparation and trials of the decisions through recorded local tools; + do not require the production service merely because the source mentions it. +- **Explicit real-output goal:** ask whether a real archive can be downloaded + intact from that service. If access is unavailable, expect a precise blocker + for that claim, not a simulated-download pass. +- **Mixed goal:** test approval decisions and real service reliability. Continue + independent authorized decision checks while retaining the reliability gap; + never describe partial coverage as full completion. +- **Previously approved real-output scope:** an unavailable capability may lead + to a concrete alternative proposal. No silent simulation substitution, new + model call or external effect outside the existing authorization. + +Check the proposal and recorded actions, not just the final answer. Record +owner intervention and whether required real capabilities were correctly +distinguished from modeled external services. Keep private evidence outside Git. + + ## Optional Jev manual verification Run `python3 tests/jev-test.py` first. Its injected transport and subprocess diff --git a/plugin/skills/behavior-diff-live/SKILL.md b/plugin/skills/behavior-diff-live/SKILL.md index 8ad487c..6272287 100644 --- a/plugin/skills/behavior-diff-live/SKILL.md +++ b/plugin/skills/behavior-diff-live/SKILL.md @@ -63,6 +63,14 @@ the sibling skill's bundled `scripts/` directory. Validate it with `python3 /purpose.py `. Keep its canonical JSON for assessment; never include purpose, its path, or raw session context in either trial prompt. Do not revise it after results. + Before capability checks, follow the canonical reference's + [claim derivation](../behavior-diff/references/scenario-and-test-data.md#prepare-from-intent) + and [controlled behavior simulation](../behavior-diff/references/scenario-and-test-data.md#controlled-behavior-simulation) + rules. Disclose simulated external tools in the initial proposal for ordinary + decision goals. A production-tool requirement in the tested instruction alone + does not require a real-output trial; explicitly real goals keep their actual + dependencies and claim-level stops. Never turn simulated results into real + inspection evidence or real owner approval. 2. **Build the variants** in a scratch dir, never in the user's repo: diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index 6823dd5..08c2e0e 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -123,13 +123,22 @@ the single-role or two-agent path. Create the fixtures with **Historical context boundary.** Before deriving historical purpose, read the source-isolation rule in `references/scenario-and-test-data.md`. If this session already knows later history or previous trial outcomes, purpose - derivation and preparation must use the fresh context described there. - If that isolation is unavailable, stop; disclosure alone is not a substitute. + derivation and preparation must use the fresh supplied-history context defined + there. An unavailable fresh-session mechanism blocks preparation; unavailable + complete ambient-context or per-turn model attestation does not block an ordinary + supported-environment rehearsal. Preserve those limits rather than silently + upgrading a historical PR request into a stronger isolation claim. **Freeze the purpose before drafting or running trials.** Derive concise, distinct goals from the current conversation and the selected diff without asking the owner to restate their reason. Use one or two sentences per goal; keep separate goals separate, including behavior meant to survive a cleanup. + Distinguish the owner's comparison question from production obligations in + the instruction being tested. Do not infer a real-output goal merely because + that instruction requires a production tool. Before capability checks, apply + the reference's claim derivation: ordinary decision goals default to + controlled external-tool simulation, disclosed in the initial proposal; + explicit real-output goals retain their real dependencies. On both Claude Code and Codex, use the conversation already available in the current context, including supplied summaries; do not open host transcript stores, resume other sessions, or infer access to missing turns. The same @@ -226,9 +235,14 @@ the single-role or two-agent path. Create the fixtures with 2. **Prepare the scenario and data.** Before drafting anything, read `references/scenario-and-test-data.md` inside this skill's base directory - and follow its preparation recipe once. From the frozen goals, author a - neutral task, sufficient shared data, private per-case criteria, and the - sanitized scenario with frozen checks and required recorded sources. + and follow its preparation recipe once. From the frozen goals, derive + decision, execution-quality, and integration claims before tool dependencies. + Author a neutral task, sufficient shared data, private per-case criteria, + and the sanitized scenario with frozen checks and required recorded sources. + Follow [Controlled behavior simulation](references/scenario-and-test-data.md#controlled-behavior-simulation) + for equal, recorded interfaces that expose material outcomes without + enforcing correct decisions or supplying assessment answers. Choosing to + inspect through a simulated tool is not evidence of actual pixel detection. Keep private context outside every trial input. The helper validates and inventories your reviewed files; it never authors their meaning or proves that a required capability works. @@ -243,11 +257,16 @@ the single-role or two-agent path. Create the fixtures with user to confirm the file, task, cost, or run mode. Do not mention trial counts, cost, or full versus fast modes during normal execution. First satisfy the privacy and sensitive-content approval boundary above; - automatic execution never overrides it. If the planned environment cannot - support a required capability or premise, stop and report it; do not silently - substitute a narrower test or fabricate approvals. Stop preparation workers - before launch, and keep the first result: no retuning or reruns to manufacture - a finding after seeing outcomes. + automatic execution never overrides it or a surrounding journey's approval + gate. Apply the reference's claim-level capability checks: missing required + premises stop dependent claims, while an unavailable production tool alone + does not block a decision claim already scoped to simulation. Report blocked + claims; do not silently narrow an approved real-output goal or fabricate + approvals. If changed scope needs approval, offer the reference's concrete + bounded proposal; do not re-ask an unchanged approved budget. Security and + historical-freshness gates remain mandatory. Stop preparation workers before + launch, and keep the first result: no retuning or reruns to manufacture a + finding after seeing outcomes. Use the reference's complete private handoff for both hosts: `prepare_shared.py prepare --handoff` for staged data or exclusions; diff --git a/plugin/skills/behavior-diff/references/scenario-and-test-data.md b/plugin/skills/behavior-diff/references/scenario-and-test-data.md index 6dfef0e..580827a 100644 --- a/plugin/skills/behavior-diff/references/scenario-and-test-data.md +++ b/plugin/skills/behavior-diff/references/scenario-and-test-data.md @@ -16,21 +16,54 @@ criteria with the recipe below. Use the contemporary change, relevant unchanged instructions and references, and available owner context. A supplied real request is usable, but not required. For a historical comparison, exclude later fixes, later incidents, earlier -fixtures, and trial results; do not mine host transcripts. If the orchestrator -already knows later history or previous outcomes, derive historical purpose and -prepare the inputs in a fresh context that inherits none of that conversation. +fixtures, and trial results from the supplied preparation context; do not mine +host transcripts. If the orchestrator knows later history or previous outcomes, +derive historical purpose and prepare inputs in a new, non-resumed, non-forked +session, or a supported subagent that does not inherit that conversation. Supply only the selected contemporary instruction versions, required references, available contemporary owner context or commit message, and this generic method. -Use a supported fresh session or isolated subagent; if unavailable, stop and -report the missing isolation prerequisite before deriving or freezing purpose. +Use a private preparation workspace containing those reviewed inputs, not earlier +runs. Do not load prior-session stores or reuse an outcome-aware conversation. +If no supported fresh-session mechanism is available, stop before preparation. If case selection or tooling was informed by hindsight, still disclose that -limit: isolation does not remove historical selection bias. +limit: a fresh supplied history does not remove historical selection bias. + +For an ordinary supported-environment rehearsal, this is a **supplied-history +freshness requirement**, not proof that every hidden system instruction, host +memory or ambient context is absent. Before a model call, check that the chosen +host supports the new session and required input/completion records; use available +history controls and review the supplied inputs. Afterward, inspect the native +record for inherited history, mismatched input, failed completion or observed +contamination. Those are blockers. Lack of complete ambient-context enumeration +or resolved per-turn model telemetry is not, unless the owner explicitly requests +that stronger evidence. Do not turn a historical PR request alone into that claim. + +For Codex preparation, a new thread alone is insufficient when the process +inherits memory or hooks. Before `thread/start` or any model turn, start its +app-server with these process-local overrides: + +```sh +codex app-server \ + -c features.memories=false -c features.hooks=false \ + -c features.plugins=false -c features.apps=false \ + -c features.multi_agent=false -c features.multi_agent_v2=false \ + -c 'web_search="disabled"' +``` -Before deriving historical purpose, check whether the chosen fresh host can -retain native supplied-input and completion records and expose the required -context controls. Do not discover missing provenance only after making a claim -of compliant preparation, or replace missing fresh preparation with a session -that already knows the outcomes. +Read `config/read` without starting a turn. Disable each configured MCP server +for the preparation process too, then restart and verify the effective controls. +Use an inline TOML table containing the actual server names with `enabled=false`, +for example `-c 'mcp_servers={example_service={enabled=false}}'`; an empty table +merges with inherited configuration and does not disable its servers. Verify +all six feature flags are false, web search is disabled, and no MCP server is +enabled before starting preparation. Keep the approved account/model and normal +child sandbox; do not change global configuration. Record only the control +summary, never the full configuration or credential-bearing server definitions. +These controls prevent the known history-loading routes, not every possible +hidden context source. If a preparer reports prior-outcome context, preserve that +report and inspect the available records and effective controls; do not claim +its source or contents were established solely from the report, and do not mark +the affected preparation fresh without resolving it. After preparation, the private handoff may include `preparation_evidence` with `host`, `record_file`, `input_file`, `thread_id`, and `turn_id`. The files must @@ -39,32 +72,75 @@ preparer prompt; `record_file` is the host-native export, not an agent-written summary. Freeze binds both files by digest and verify rejects changed or missing records. -The Codex adapter validates a native `ThreadReadResponse` containing one complete, -non-inherited legacy-history turn and one matching plain-text user input. -Partial, paginated, forked, compacted, ambiguous or failed records do not establish +The Codex adapter accepts legacy and paginated native histories. Both must +establish one complete, non-inherited turn and one matching plain-text user +input. Partial, forked, compacted, ambiguous or failed records do not establish those facts. Unsupported host formats remain explicitly unavailable. +- For `historyMode: "legacy"`, save the complete native `ThreadReadResponse` + returned by `thread/read` with `includeTurns: true`. +- For `historyMode: "paginated"`, use `thread/read` with `includeTurns: false` + for metadata, then `thread/turns/list` with `threadId`, `cursor: null`, + `sortDirection: "asc"` and `itemsView: "full"`. Follow every `nextCursor` + until it is explicitly `null`, retaining each exact request and response. + An optional positive `limit` sets page size, not the total history limit. + Save this envelope as `record_file`: + + {"thread_read": , "turn_pages": [{"request": , "response": }]} + + The metadata-only thread must have an empty `turns` array. Page requests + must start at the beginning, keep the same thread, direction and full item + view, and follow the complete cursor chain without filtering or repetition. + Summary items and unfinished pages cannot establish fresh preparation. + Preserve native responses unchanged; never rewrite the history mode or + combine pages into a fabricated legacy response. + +Capture these records after the preparation turn completes, before freezing +the handoff. Native pagination alone is not a missing preparation capability. + These checks establish only the supplied user input and native completion represented in the provided record. They do not authenticate an arbitrary export or enumerate injected system context. Codex's thread model is configured/latest -metadata, not per-turn execution telemetry: resolved execution model and the -complete historical context boundary remain unavailable. Missing references -remain unverified. Never turn those unavailable facets into a compliant-preparation -claim; when the requested claim requires them, report the missing prerequisite. +metadata, not per-turn execution telemetry. The adapter therefore retains overall +`status: "unverified"` and unavailable `resolved_model` and `historical_boundary` +facets even when `supplied_user_input` and `completion` are verified. + +For an ordinary supported-environment rehearsal, use those verified facets with +the fresh-session and reviewed-input checks above, disclose the unavailable +facets, and continue to freeze and run. Do not require the overall status to +become verified, relabel unavailable facets, or ask the owner to waive them. +Missing or invalid required input/completion evidence still blocks preparation. +When the owner explicitly requires complete ambient isolation or resolved-model +attestation, retain that stronger requirement and report the missing capability. Native records and reviewed prompts stay out of trial copies, assessment input and public reports. 1. **Goals.** Start from the frozen purpose goals. Name new behavior and behavior that must survive the change. Mark preservation goals as inferred unless stated. -2. **Capabilities and premises.** For each goal, list what the agent must - observe, decide, or produce. For each capability, list the environment - facts it depends on (files, query results, tools, permissions, repository - state, time), where the trial can reach each fact, and which source is - authoritative. Assessment-only knowledge must not stand in for a premise - missing from the trial. If an outcome depends on time, set one as-of - timestamp and timezone, give both variants the same neutral as-of statement, - and make every simulated clock and calculation agree. +2. **Claims before capabilities.** For each goal, first name the claim the + comparison must support: an agent decision (such as choosing to inspect, + checking a result, or respecting an approval boundary), actual execution + quality (such as detecting a visual defect), or a real integration outcome. + Record its basis in the selected request and contemporary instructions. + A production requirement in the source instruction does not by itself make + its production tool a trial dependency for an agent-decision question. + Preserve that requirement as behavior under test; do not turn every source + requirement into an end-to-end acceptance claim. + + For ordinary instruction rehearsals of decisions, default to controlled + simulation of external tools as described below. Disclose that evidence + boundary in the initial proposal before approval and freeze it with the + claims. Explicit real-output or real-integration goals retain their actual + dependencies; never silently replace them with simulation. + + Only then list what the agent must observe, decide, or produce for each + claim. Map each capability to its premises (files, query results, tools, + permissions, repository state, time), reachable interfaces, and authoritative + sources. Assessment-only knowledge must not stand in for a premise missing + from the trial. If an outcome depends on time, set one as-of timestamp and + timezone, give both variants the same neutral as-of statement, and make + every simulated clock and calculation agree. 3. **Task.** Write a natural user request that starts before the tested work and reaches the changed rule. It defines starting state and scope only. It must not state the expected answer, the rule, a findings list, or a @@ -85,9 +161,10 @@ and public reports. arguments or artifacts, and not enforce the desired result. Extra scenarios or trial sets require separate approval. 5. **Private notes.** Outside every trial input, in the private `0700` - directory, record with mode `0600`: the goal → capability → premise → - case mapping; why each case was selected; evidence that establishes whether - each condition was reached, missed, or remains unknown; for each agreement + directory, record with mode `0600`: the goal → claim and evidence boundary → + capability → premise → case mapping; why each case was selected; evidence + that establishes whether each condition was reached, missed, or remains + unknown; for each agreement check, the trial-reachable sources whose returned content must ground it; separate criteria for supported, contradicted, or insufficient-evidence outcomes for every case; cross-case cues and interactions; capabilities left @@ -200,6 +277,53 @@ Ask the user only when a material requirement cannot be reached from the change, project, or conversation. Do not require an incident or an existing eval. +## Controlled behavior simulation + +Controlled simulation tests the agent's choices against authored external-tool +behavior, not the quality or availability of the real service. Use it by default +for ordinary decision goals, not as a fallback after approving a real-output +claim. It does not waive historical fresh preparation, privacy, permissions, +required context boundaries, or the real Spacedock handoff requirements. + +Give both variants the same interfaces and initial state. Make every material +outcome needed for the decision discoverable through those interfaces, including +failure results and relevant artifacts. Record submitted actions, arguments, +artifacts, and tool returns. The adapter must accept wrong actions as well as +correct ones; it must not reject a bad choice just to force compliance, repair +the agent's work, supply assessment answers, or tell it which branch is correct. +Use the existing captured read forms for required source evidence. Keep private +criteria out of tool responses. Simulated user responses must never be presented +as the real owner's approval or authorize real external effects. + +An inspection request and an inspection result support different claims. +Recorded use of a simulated inspection interface can show that the agent chose +to inspect and responded to its returned facts; it cannot establish that the +agent actually detected pixels or that a native generator produced a valid +image. Name simulation in the initial terms and scenario limitations, and keep +that distinction in assessment and the final recommendation. + +For example, use a synthetic archive and checksum manifest with no personal +data. For the decision claim "checks integrity before publishing", a safe local +adapter can record the requested download, return an authored archive result, +and expose a checksum mismatch through the same query interface on both sides. +Let the agent discover it and record any publish attempt without publishing. +This supports only decisions in response to modeled results. The separate claim +"downloads a real archive and correctly computes its digest" requires actual +archive bytes, the real download integration when required, and a working digest +command; an authored checksum result cannot establish it. + +Resolve missing capabilities per claim before stopping a comparison. A missing +native production tool blocks the real-output claim that depends on it, not an +ordinary decision claim already scoped to simulation. Report blocked claims +explicitly; do not mark them passed or silently omit them. If an approved plan +requires the blocked real outcome, stop that dependent work and offer a concrete, +bounded revised proposal naming the retained decision claims, simulated +interfaces, lost real-output evidence, and any change to the approved quota. +Obtain approval for that changed scope before running it. Do not silently convert +an approved real-output claim to simulation, re-ask an unchanged approved budget, +or launch extra trials. Cross-cutting security or freshness failures still stop +all work that depends on them. + ## Comparison scope and execution route Ordinary comparisons ask whether the selected change is useful in the supported @@ -226,12 +350,14 @@ explicitly selected, separately scoped route, never an automatic fallback. A filesystem boundary alone does not establish causal isolation from every model or host influence. -If a required boundary or capability is unavailable, report that precise blocker. +If a required boundary or claim-level capability is unavailable, report that +precise blocker and stop the dependent work under the claim rules above. Do not weaken host permissions, invent a launcher, or silently substitute a -current-environment comparison for the stronger requested claim. Conversely, do -not block an ordinary supported-environment comparison solely because it cannot -support a strict causal claim. Historical fresh-context preparation above remains -required on either route; disclosing host exposure does not waive it. +current-environment comparison for the stronger requested claim. Conversely, +do not block an ordinary supported-environment comparison solely because it +cannot support a strict causal claim or an unrequested real-output claim. +Historical fresh-context preparation above remains required on either route; +disclosing host exposure does not waive it. ### Required native filesystem boundary @@ -276,12 +402,14 @@ under the host's intended permissions: - The selected After bytes and the Before content or verified absence are available; every reference required by either version is in the trial base. Exactly the selected instruction differs between variants. -- Exercise the required reads, queries, local tests, artifact writes and safe - action recording in scratch copies under the intended host tools and - permissions. Confirm consistent facts and authoritative sources, including - time when needed. Confirm that each source requirement names a source the - trial can reach through a captured read form listed in step 6. Do not assume - subagent dispatch or approvals exist. +- Exercise the reads, queries, local tests, artifact writes and safe action + recording required by the derived claims in scratch copies under the intended + host tools and permissions. For controlled simulation, exercise its actual + interfaces and recorded returns, not the production service it models. + Confirm consistent facts and authoritative sources, including time when + needed. Confirm that each source requirement names a source the trial can + reach through a captured read form listed in step 6. Do not assume subagent + dispatch or approvals exist. - Represent relevant branch/remote state, permissions and existing user work explicitly. Use safe local artifacts for modeled publication; never include credentials or perform real external effects merely to test an instruction. @@ -295,9 +423,10 @@ under the host's intended permissions: Run probes in scratch copies only. A probe may exercise discovery queries to check the environment, but must not deposit its findings in trial inputs or change the original project or frozen stage. The trial still performs discovery -itself. If a required capability or premise is missing, stop and report it. -Do not substitute a generic simulator or fabricate approvals. A deliberately -modeled limitation is part of the scenario; an accidental harness failure is not. +itself. If a required capability or premise is missing, apply the claim-level +stop rules above before launch. Never substitute a generic simulator for a +required real outcome or fabricate approvals. A deliberately modeled limitation +is part of the frozen scenario; an accidental harness failure is not. ## Freeze @@ -358,6 +487,15 @@ and report the affected phase even if a report was generated. Distinguish startu normalization, assessment, and behavioral failures. Claim that no model ran only when startup diagnostics and native evidence establish that fact. +Check the execution host separately from the trial sandbox. A sandboxed command +tool may be unable to start another native sandbox even though the same runner +works from the host. Use the host's normal, authorized command-approval mechanism +for the exact preparation or fixed-runner command; retain the child agents' +normal sandbox. Model-run consent is not OS permission. Do not disable sandboxes, +enable automatic approvals, or add a replacement launcher to evade a denial. +If the host cannot grant the needed execution access, report that prerequisite; +do not classify it as an unavailable simulated production tool. + Use only the recovery allowance already authorized for this comparison. Before another model-backed attempt, identify a concrete local repair or a distinct diagnostic hypothesis, complete applicable non-model checks, and confirm remaining diff --git a/plugin/skills/behavior-diff/scripts/decisions.py b/plugin/skills/behavior-diff/scripts/decisions.py index 33324c6..97a3e8d 100755 --- a/plugin/skills/behavior-diff/scripts/decisions.py +++ b/plugin/skills/behavior-diff/scripts/decisions.py @@ -105,6 +105,16 @@ "edit_hunks": [<1-based numbers of the supporting instruction diff hunks>] }}|null, "purpose_summary": "<1-2 short plain sentences restating the supplied purpose goals, at most 320 characters>"|null, + "chain": [ + {{"topic": "<2-4 plain words naming the observed result or behavior>", + "decision": "", + "anchor": , + "before": [{{"choice": "", "trials": [""]}}], + "after": [{{"choice": "", "trials": [""]}}], + "diverges": true|false, + "edit_hunks": [<1-based numbers of related instruction diff hunks, or empty>], + "note": ""}} + ], "target_assessment": {{ "criteria": [ {{"id": "", "goal": <1-based purpose goal>, @@ -112,30 +122,20 @@ "mode": "correction"|"new"|"change"|"preservation", "required_evidence": "", "evidence_kind": "output"|"source_consistency"|"recorded_action", - "decisions": [], + "decisions": [<1-based indexes of related rows in the final chain, not goal/check numbers; empty if none>], "before": [{{"trial": "", "outcome": "met"|"not_met"|"uncertain"|"unassessable", "refs": [""], - "output_excerpt": "", + "output_excerpt": "", "explanation": ""}}], "after": [{{"trial": "", "outcome": "met"|"not_met"|"uncertain"|"unassessable", "refs": [""], - "output_excerpt": "", + "output_excerpt": "", "explanation": ""}}] }} ] }}|null, - "chain": [ - {{"topic": "<2-4 plain words naming the observed result or behavior>", - "decision": "", - "anchor": , - "before": [{{"choice": "", "trials": [""]}}], - "after": [{{"choice": "", "trials": [""]}}], - "diverges": true|false, - "edit_hunks": [<1-based numbers of related instruction diff hunks, or empty>], - "note": ""}} - ], "fork": <1-based index into chain of the FIRST divergence, or null>, "fork_note": "", "outcome": <1-based index of the primary task outcome in chain, or null>, @@ -306,14 +306,19 @@ For both, met/not_met needs a nonempty output_excerpt: choose ONE exact contiguous span of the cited trial final answer or declared captured artifact showing the claim and copy it character for character. Never join fragments, add ellipses, paraphrase -or repair it. "recorded_action" judges whether the agent made a call: a tool-N -record proves only that its listed source call was made, even when its return was -omitted or unavailable, never what the call returned. Files the agent wrote or +or repair it. For "recorded_action", leave output_excerpt empty for every outcome; +cite the calls, not a quotation describing them. A tool-N record proves only that +its listed source call was made, even when its return was omitted or unavailable, +never what the call returned. Files the agent wrote or published do not prove completion without captured output. Records ending in :trace or :overflow, sources shown as [sensitive source omitted], self-reported actions and duplicate call or result identities prove no outcome. Absence of a call is shown only by the trial's complete call list; an :overflow record means later calls are not listed. +Assign each criterion's decisions only after the chain is final. Every index must +name a row in that returned chain, not a goal/check number or a discarded draft row. +Verify each nonempty output_excerpt against its cited source in that same trial; +text from a tracker cannot cite :answer, and text from another trial cannot be reused. Met/not_met requires evidence sufficient for the stated criterion: checking output consistency with a source requires BOTH recorded source content and output evidence. When no available or truncated record contains that source content, source agreement diff --git a/plugin/skills/behavior-diff/scripts/preparation_evidence.py b/plugin/skills/behavior-diff/scripts/preparation_evidence.py index 02c8254..a5a293f 100644 --- a/plugin/skills/behavior-diff/scripts/preparation_evidence.py +++ b/plugin/skills/behavior-diff/scripts/preparation_evidence.py @@ -16,6 +16,55 @@ CONTEXT_LIMIT = "Native user input does not enumerate all ambient or system context." +def _paginated_turns(pages, thread_id): + if type(pages) is not list or not pages: + raise ValueError("native preparation turn pages are missing") + turns = [] + cursor = None + seen = set() + for index, page in enumerate(pages): + if type(page) is not dict or page.keys() != {"request", "response"}: + raise ValueError("invalid native preparation turn page") + request, response = page["request"], page["response"] + required = {"threadId", "cursor", "sortDirection", "itemsView"} + if ( + type(request) is not dict + or not required <= request.keys() <= required | {"limit"} + or request["threadId"] != thread_id + or request["cursor"] != cursor + or request["sortDirection"] != "asc" + or request["itemsView"] != "full" + ): + raise ValueError("native preparation pages must be complete and unfiltered") + if "limit" in request and ( + type(request["limit"]) is not int or request["limit"] <= 0 + ): + raise ValueError("invalid native preparation page limit") + if ( + type(response) is not dict + or not {"data", "nextCursor"} + <= response.keys() + <= {"data", "nextCursor", "backwardsCursor"} + or type(response["data"]) is not list + ): + raise ValueError("invalid native preparation page response") + for turn in response["data"]: + if type(turn) is not dict or turn.get("itemsView", "full") != "full": + raise ValueError("partial native preparation items are unverified") + turns.extend(response["data"]) + cursor = response["nextCursor"] + if cursor is None: + if index != len(pages) - 1: + raise ValueError("native preparation pages follow terminal cursor") + elif type(cursor) is not str or not cursor or cursor in seen: + raise ValueError("invalid native preparation continuation cursor") + else: + seen.add(cursor) + if cursor is not None: + raise ValueError("native preparation turn history is incomplete") + return turns + + def validate_preparation_evidence(reference): if type(reference) is not dict or reference.keys() != FIELDS: raise ValueError("invalid native preparation evidence reference") @@ -41,6 +90,12 @@ def validate_preparation_evidence(reference): raw = json.loads( Path(reference["record_file"]).read_text(), object_pairs_hook=unique_object ) + pages = None + paginated_record = False + if type(raw) is dict and raw.keys() == {"thread_read", "turn_pages"}: + paginated_record = True + pages = raw["turn_pages"] + raw = raw["thread_read"] if type(raw) is not dict or not {"thread"} <= raw.keys() <= {"thread", "readState"}: raise ValueError("expected a full native Codex ThreadReadResponse") thread = raw["thread"] @@ -57,11 +112,15 @@ def validate_preparation_evidence(reference): or thread["parentThreadId"] is not None ): raise ValueError("forked or inherited native preparation history is unverified") - if thread.get("historyMode", "legacy") != "legacy": - raise ValueError( - "paginated native preparation history is unverified; full legacy history required" - ) - turns = thread.get("turns") + mode = thread.get("historyMode", "legacy") + if mode == "legacy" and not paginated_record: + turns = thread.get("turns") + elif mode == "paginated" and paginated_record: + if thread.get("turns") != []: + raise ValueError("paginated preparation requires metadata-only thread read") + turns = _paginated_turns(pages, reference["thread_id"]) + else: + raise ValueError("native preparation requires complete supported history") if type(turns) is not list or len(turns) != 1: raise ValueError("fresh native preparation requires one complete turn") turn = turns[0] diff --git a/plugin/skills/behavior-rehearse/SKILL.md b/plugin/skills/behavior-rehearse/SKILL.md index b47b826..fcf3fc9 100644 --- a/plugin/skills/behavior-rehearse/SKILL.md +++ b/plugin/skills/behavior-rehearse/SKILL.md @@ -90,16 +90,24 @@ Humanizer guidance in embedded mode without installing it or changing evidence. If the request leaves materially different verification goals open, offer 2–4 concrete, outcome-based options derived from that change. Mark one recommended and explain why; usually check both the intended new behavior and preservation - of existing safeguards. Allow the owner to name a different concern. Do not - ask only "What do you want to verify?" or present model, fixture or execution - mode choices as goals. If the goal is already clear, skip the options. - Give Behavior Diff the proposed or explicit question and source context; use - its method to resolve inputs, capabilities, tested scope and run terms. Do not - require the owner to assemble evidence or confirm repository-provided facts. + of existing safeguards. Ordinary instruction rehearsal checks agent decisions + using controlled simulation of external services. Treat real generation, + inspection or integration as a dependency only when the owner's explicit goal + requires real outputs, not because the PR tells the production agent to perform + those actions. Allow the owner to name a different concern. Do not ask only + "What do you want to verify?" or present execution modes as goals. If the goal + is already clear, skip the options. + Give Behavior Diff the proposed or explicit question and source context; resolve + method fit, capabilities and the complete tested scope before requesting run + approval. Preserve each material goal and identify any claim the method cannot + test; do not approve broad wording and later upgrade its capability requirements. + Do not require the owner to assemble evidence or confirm repository-provided facts. 2. **Settle goal and authority together.** Put any goal options and their recommendation in the same handoff proposal as the applicable bounded run terms, following [execution handoff](references/execution.md#handoff-proposal). - State which option each set of terms covers. Explicitly say that "approve" + State the controlled-service simulation default and which claims and options + the terms cover, distinguishing agent decisions from real-output quality or + integration. Explicitly say that "approve" accepts the recommended goal and its stated run terms; choosing a goal alone is not run authorization. Do not add separate scope and execution approval rounds when one proposal can settle both. Reuse existing authorization and @@ -123,11 +131,15 @@ Humanizer guidance in embedded mode without installing it or changing evidence. Use [execution handoff](references/execution.md) for authority and recovery boundaries, not a second comparison recipe. 5. **Recover without abandoning independent work.** Route a comparison problem - through Behavior Diff's diagnosis and recovery method. Continue useful safe - non-model diagnosis and independent review within authority. An in-scope - supported route is work to do, not owner homework. Ask again only for a - material goal correction, unavailable fact or authority beyond approved terms. - Never rerun merely to obtain a difference. + through Behavior Diff's diagnosis and recovery method. For a decision goal, + missing native production tools call for a supported behavioral scenario within + the approved simulation method, not a whole-comparison blocker. Continue useful + safe non-model diagnosis and independent review within authority. An in-scope + supported route is work to do, not owner homework. If an explicit real-output + goal cannot be tested, preserve that gap and propose a concrete bounded amendment + with the changed claims and run terms; do not silently convert it to simulation. + Ask again only for a material goal correction, unavailable fact or authority + beyond approved terms. Never rerun merely to obtain a difference. 6. **Return a decision-ready result.** Give the action-first conclusion above in the first post-verification answer, not only after an adoption question. Explain what improved versus Before, what stayed intact, and what remains diff --git a/plugin/skills/behavior-rehearse/references/execution.md b/plugin/skills/behavior-rehearse/references/execution.md index c6cb906..357eddc 100644 --- a/plugin/skills/behavior-rehearse/references/execution.md +++ b/plugin/skills/behavior-rehearse/references/execution.md @@ -13,9 +13,13 @@ different copy or invent a substitute runner. ## Handoff proposal Inspect the change first. When the goal is ambiguous, offer 2–4 distinct outcomes -tailored to it, with a recommended option and a short reason. Do not require -technical test design from the owner. Skip the options for an already clear goal. -Combine this clarification with the run proposal, not a separate questionnaire. +tailored to it, with a recommended option and a short reason. Ordinary instruction +rehearsal uses controlled external-service simulation to check agent decisions +and safeguards. Real-output quality and live integration are separate goals; require +them only when the owner's explicit question depends on them, not from action +verbs in the production instructions. Do not require technical test design from +the owner. Skip the options for an already clear goal. Combine this clarification +with the run proposal, not a separate questionnaire. For an invented PR adding archive checksum checks, the options could be: @@ -29,6 +33,14 @@ For an invented PR adding archive checksum checks, the options could be: Explain why the recommendation fits the actual change, and let the owner name another concern. These example options are not a fixed menu for every PR. +Before asking approval, use Behavior Diff to resolve method fit for every material +goal, including the intended new behavior and preserved safeguards. Identify the +claims the scenario will test and any material untested claims. Do not drop goals +to fit available tools or approve a broad question before discovering it requires +a different method. If an option requires unavailable real outputs, explain that +limit and recommend a concrete bounded proposal that distinguishes the behavioral +check from the unresolved real-output claim. + Then state run terms covering the recommended option. If all offered options can share the same bounded terms, explicitly say so; otherwise identify the differences before seeking approval. Do not promise unknown terms to avoid an @@ -40,8 +52,12 @@ every placeholder with checked facts: I recommend