[Copilot]

Summary

• Add an opt-in comparison of native tools, TraceDecay core, and existing channel variants on independently checked tasks.
• Report matched task success, observed cost and timing variability, and artifact/setup identity without treating missing usage as zero.

Motivation

Adoption alone does not establish product value. Reproducible task and cost comparisons are needed to guide product decisions while preserving prior qualified evaluations.

Changes

• Runner. Add a held-out task cohort, native-tools condition, per-run artifact/task/fixture identities, and setup timing. Require an explicit shipping binary for baseline runs.
• Grader. Report per-task success and invalid attempts, Wilson intervals, observed token/cache/USD/time ranges, and deltas against native runs. Optionally join the no-model efficiency scorecard only when binary SHA-256 matches.
• Evidence. Document the opt-in run and defer dashboard, Work/Workflow, and grammar changes where required empirical measurements are unavailable.

Example, with a shipping binary built first:

TRACEDECAY_AGENT_EVALS=1 EVAL_INCLUDE_BASELINE=1 \
  TRACEDECAY_BIN=/path/to/shipping/tracedecay \
  HOSTS=claude CLAUDE_MODELS=sonnet REPS=3 \
  CHANNELS="native bare no-hints no-skills full" \
  SCENARIOS="baseline_order_stock_failure baseline_stock_check_rename baseline_currency_decision" \
  evals/agent_adoption/run.sh

Verification

• The real daemon held-publication hook journey passed through Bazel (1 test, 4.1 s). Both edits require distinct durable broker receipts before the hold releases; the path sets are logged by digest. The test checks the first worktree binding during the serialized open and both new bindings plus full publication after release.
• The unchanged twelve-client MCP/CLI/hook ownership and Doctor journey passed through Bazel (1 test, 5.3 s). The prior fleet attempt reported resident-memory pressure; full CI must confirm that failure clears.
• Strict Bazel Clippy passed for the hook suite. Pinned formatting, diff checks, and commitlint passed.
• CI passed the hermetic evaluation regressions on the preceding head. Full workspace CI on the updated branch remains required before merge.
• The paid live-model comparison study has not been run here.

• Refs Extend prior outcome evals into product-wide task effectiveness and total-cost baselines #4219

Remaining issue scope

This PR repairs and extends the comparison harness. The repeated product-wide effectiveness study, observed total-cost results, and evidence-backed keep/simplify/defer/remove decisions required by #4219 have not been produced, so merging the harness must not close that issue.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 185991e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Cross-referenced #4219 and the related oracle work in #4216. Three issues currently prevent these results from supporting a checked effectiveness comparison: the conditions grant different native edit permissions, the rename oracle accepts deletion of order behavior, and the memory-answer oracle accepts the opposite of the required facts. I exercised the actual grade_outcome implementation at this head with the adverse inputs in the inline comments; both incorrectly returned pass. No paid model turns were run.

Please resolve the condition mismatch and false-success cases before these scenarios enter the correctness numerator.

[ScriptedAlchemy]

Pushed b7a633f to address the outcome and capability issues from review (cross-referenced with #4219).

• All Claude conditions now have the same native read/write/edit capabilities; product conditions add their product tools.
• The currency scenario requires one exact JSON fact object, so contradictory prose, wrong facts, additional claims, and duplicate keys cannot pass.
• The stock-edit scenario now compiles the captured Rust source through Bazel and runs trusted behavior assertions outside the editable fixture. It checks the renamed and removed APIs, stock boundaries/reservations, order totals, empty orders, and insufficient stock. Missing execution infrastructure or compilation failure is unverified, never pass.

The new negative cases reproduced 15 failures before the changes. Afterward, the full Python grader selftest, all 75 scenario definitions, shell/Python syntax, pinned Rust formatting, commitlint, and diff checks pass. The real compiled stock-oracle cases are wired into the Hermetic evaluation regressions workflow and have not been executed locally; CI must establish those results. The effectiveness study itself remains unrun. Marked ready so the native lane is enabled.

[ScriptedAlchemy]

Pushed follow-up 1b29a6d01a031f970da639608ecb866b903cd1e2 on top of the earlier outcome-verification fixes.

• The runner now carries the digest of the actual selected scenario file alongside its decoded prompt. A scenario's ID no longer has to match its filename; every runner path receives the same digest, and source changes invalidate it.
• Joined scorecards retain finite numeric resource medians (restart latency, storage, RSS, and tool costs), omit unavailable values, and render units. Artifact validation and model-cost separation stay intact.
• The report explains unavailable behavioral execution as unverified. Unverified/invalid attempts remain unsuccessful in the task-success denominator.

Validation: the full Python self-test, all 75 scenario lint checks, shell/Python syntax checks, and diff checks pass. The resource join/render regressions failed before the fix. Independent review found no further blocker in this follow-up.

The initial stock-edit verifier and its adversarial self-test still need native Bazel CI execution. #4219 remains referenced rather than closed: repeated product experiments, measured total cost, and the keep/simplify/defer/remove decisions have not been run.

[ScriptedAlchemy]

Pushed follow-up commits 8a40ff2 and 5d0f61f after investigating the two failures in native CI job 119047721226 (167/169 tests passed).

The linked-worktree journey incorrectly required a synchronous acknowledgement even though the production callback deliberately stops waiting after 750 ms and the daemon can still durably admit its received edit. The test now checks both first edits publish their worktree bindings while full publication is held, without resubmission. It then verifies full publication within the existing shared completion budget. This tests delivery directly and does not increase the timeout.

The Claude cache test still fails at its final activation check after binary relocation; its root cause is not yet confirmed. Added exact per-file byte comparisons, unexpected-entrypoint checks, and a separate registration assertion immediately before that unchanged activation assertion so the next failure identifies the broken boundary. I have not labeled this failure resolved.

Pinned rustfmt, commitlint, and git diff --check passed. Native execution is assigned to the new CI run; no local native pass is claimed.

[ScriptedAlchemy]

Strengthened the first-edit regression in response to the remaining receipt finding. A canonical hook_file_edit_admitted operator event is emitted only after broker.admit returns its durable sequence, carrying the exact project root and relative paths. The linked-worktree journey now requires both distinct nonzero admission receipts independently of hook bindings before releasing full publication, then requires full publication within the same existing deadline. No callback resubmission or budget increase. Canonical log formatting also handles Windows path quoting. Pinned rustfmt, diff checks, and commitlint pass; native CI is pending. The earlier cache-content diagnostics remain in place so the next native run can identify that separate failure precisely.