[milindsrivastava1997]

Why

The planner has no reference answer, so a plan change can slip in unnoticed. Closes #941.

What

• crates/devtools/tests/corpus_snapshots.rs: plans each query in tests/corpus_snapshots/<corpus>/corpus.json with release stage_pipeline (default search, --no-memo, 240 s per query) and compares the selected plan with <id>.json.
• 12 o11y-bench query snapshots; q18 (Planning one ratio of filtered increases went from about 1 minute to over 28 minutes and 12 GB on main #1023) and q21 (a pane round can end on its wall-clock budget) have a skip_reason and are skipped.
• CI job corpus-snapshots (release, sccache, 45 min timeout); on failure it uploads the regenerated snapshots with each query's full document for the DAG viewer.
• docs/develop_docs/planner-snapshots.md, a Snapshot changes section in the PR template, and an AGENTS.md rule.

How

A snapshot holds only what must be stable: the selected Stage 2 candidate's display name and DAG, its cost (cost_per_second, usage, per_node) and accepted accuracy, materialized views, pane retention, warnings, and proven_optimal (search status optimal). A query that does not plan keeps its error up to the search report and its rejection causes. Floats are rounded to 12 significant digits. Candidate ids, the search report, the Stage 1/2 candidate lists and the memo are left out: pane-width rounds have wall-clock budgets, so a different round, with different ids and counters, can find the same plan. A search ended by a wall-clock budget (status time_limit, or the whole 240 s used) fails instead of being recorded. Only the final pane round's status is checked; an earlier round cut short by its own wall-clock budget is not reported.

Before

No check pins planner output on real queries.

After

A PR that changes any corpus plan fails corpus-snapshots until the snapshots are regenerated (UPDATE_SNAPSHOTS=1 cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored) and the change is explained.

Snapshot changes

All snapshots are new in this PR. Compared with this PR's earlier revision, the format changed from the whole stage_pipeline document to the plan alone, so the earlier stored_outputs → materialized_views rename (#991) no longer shows separately. q18 and q21 were removed (skip_reason). On main, q05 and q06 do not plan (every candidate is rejected as executor_cannot_compile; the search stops at its evaluation budget of 128 refinements, not at a time limit, so the result is deterministic) and q11 is infeasible; their snapshots record that. Rebased onto main after #989 (sketch state-byte pricing): no snapshot changed.

Verification

• Regenerated, then ran the check twice and once pinned to one core: all passed, so the snapshots are identical across runs.
• Runtime: about 137–168 s on a 56-core machine (peak memory 96 MB), 334 s on one core. At half the speed of one core q25 needed more than 120 s, hence the 240 s limit.
• Unit tests for the snapshot contents, time-limit detection and diff message; cargo fmt --all --check; cargo clippy -p asap-devtools --tests -- -D warnings.

🤖 Generated with Claude Code

[milindsrivastava1997]

FYI @zzylol

[zzylol]

Thanks, the snapshot idea is what we want for #1008 (DAG viewer and memo in CI). It can't merge as is, though. On current main, all 14 snapshots fail, and the cause is not a changed plan.

1. The snapshot compares search internals that vary from run to run (high)

corpus_snapshots.rs:79-98 (strip_timings) and :181 compare the whole document. We ran the merged tree twice per query, plus once under taskset -c 0, on one machine. q19, q21 and q25 differed between runs, while the selected DAG, materialized_views and cost_per_second were identical:

query	selected id	status	peak_milp_constraints
q19	pane-round-12 / -12 / -15	evaluation limit / evaluation limit / refinement_limit	2220 / 2218 / 2463
q21	pane-round-15-joint-plan-3	refinement_limit / time_limit (600 s) / refinement_limit	–
q25	pane-round-12 / -14 / -15	–	2221 / 2344 / 2461

The CI failure on this PR (peak_milp_constraints 2084 → 2086 for q18) is the same variation. Between the PR's base and the CI merge base, the only planner change is a clone() → copy.

Suggested fix: snapshot only the plan.

• Keep: the selected candidate's dag and display_name; its costs (cost_per_second, per_node, usage) and accepted entries, written without the generated id; materialized_views, pane_retention_ms and accuracy_guarantees; for a failed query, error cut before "Search report". Round floats to about 12 significant digits.
• Drop: search.candidate_connections (counters, bounds, status, pane rounds, catalog sizes), the pane-round-N-joint-plan-K ids, and the Stage 1 and Stage 2 candidate lists.

2. Time limits (high)

• q18 now hits the 600 s limit every time on main. With 3000 s it was still running at 28 min 45 s with 12 GB RSS. It took 53–78 s at your base. This is a planner regression, filed as #1023. Please drop q18 from the corpus, or mark it known-slow, until that is fixed.
• 600 s per query only delays the failure, since hitting the limit already fails the test. Suggested: about 60 s per query, and timeout-minutes: 45 on the job.

3. hit_time_limit misses some time limits (medium, :65)

It only compares full_planning_ms with the limit. The seed round's budget (route.rs:114) and the pane-frontier deadline can also end a search. Suggested: also fail when status == "time_limit".

4. Smaller items

• docs/develop_docs/planner-snapshots.md:44-45: "under about 10 s and 500 MB" and "identical across two runs plus taskset" no longer hold on main.
• .github/workflows/rust.yml:170-177: caches the whole release target on top of sccache. Suggested: Swatinem/rust-cache or sccache alone.
• :126: clear the regenerated directory at the start of a run.
• After rebasing: #991 renamed stored_outputs to materialized_views (and built_at to refresh), so every snapshot changes once. Explain this under "Snapshot changes".

Once the snapshot holds only the plan, please run it twice plus once under taskset -c 0 on main and confirm the files are identical. I can then merge it. The memo enumeration counts from #1003 will be added to the snapshot after that.

Addressed in 9e65c86 and b8791aa: snapshots hold only the selected plan, cost, accuracy and proven_optimal; skip_reason for q18/q21; time_limit status detected; 240 s per query. Checked twice + one core; independent review done.