[zzylol]

Closes #1004. Part of #1008.

Why

The default search (candidate-connection MILP) prices candidates through its own coefficients. Nothing checked that its selected plan is the cheapest of the candidate DAGs the memo stands for. #1009 lists those DAGs (memo::enumeration::{enumerate, compose}) and #1024 writes them into the stage_pipeline document. This PR prices them.

What

stage_pipeline --price-enumerated <n> (off by default) takes the first n listed choices of memo.enumeration. It runs each choice's composed DAG through Stage 2 and Stage 3, the same way the selected plan is checked and costed. For each choice it writes valid, rejection, rejection_reason, cost_per_second and cheaper_than_selected. It also writes the summary priced: {count, cheaper_than_selected, selected_cost_per_second}. Without the flag, priced stays null and the choices have no pricing fields (the contract's shape).

How

asap_devtools::memo_pricing::price_choices takes each choice and does four things:

1. compose gives the choice's DAG. Each query's operator references are replaced by their root groups' composed DAGs.
2. validate_structure checks the composed roots.
3. Stage 2 (optimize_physical_asap) builds every materialization assignment.
4. Stage 3 (select_plan) checks and prices them under the run's PlanningModels: cost model, accuracy model (ignored or enforced), deployment capabilities and executor compile check. The run's calibration factor is frozen.

A choice is valid when one of its physical plans passes. Its cost is the cheapest valid plan's cost. A choice is cheaper_than_selected when it is valid and costs less than the selected plan by more than Stage 3's tie tolerance (1e-9 relative).

Key code interfaces added / modified

crates/devtools/src/memo_pricing.rs (new module)

/// One enumerated choice priced as the selected plan is (new type).
pub struct PricedChoice {
    /// Stage 2 implemented it and Stage 3 accepted one of its physical plans. // new
    pub valid: bool,
    /// Stage 3 rejection cause of its first physical plan in snake_case, or
    /// compose_failed / invalid_structure / stage2_failed / stage3_failed / rejected; None when valid. // new
    pub rejection: Option<String>,
    /// That rejection's one-line reason. // new
    pub rejection_reason: Option<String>,
    /// Cheapest valid physical plan's cost per second; None when invalid or under cost model 3. // new
    pub cost_per_second: Option<f64>,
    /// Valid and cheaper than the selected plan beyond the 1e-9 tie tolerance. // new
    pub cheaper_than_selected: bool,
}

/// What the choices are compared against (new type).
pub struct SelectedBaseline {
    /// The selected plan's cost per second. // new
    pub cost_per_second: Option<f64>,
    /// The run's calibration factor, frozen when pricing each choice. // new
    pub calibration_factor: Option<f64>,
}
impl SelectedBaseline {
    /// The baseline of a run's Stage 3 selection. // new
    pub fn of(selection: &Selection) -> Self;
}

/// Compose, implement (Stage 2) and check/price (Stage 3) each choice; query i is
/// stage0_roots[i] with entries[i]. `selected: None` prices without comparing. // new
pub fn price_choices(
    memo: &CandidateMemo,
    choices: &[MemoChoice],
    stage0_roots: &[QueryRoot],
    entries: &[QueryWorkloadEntry],
    data: &DataWorkload,
    models: PlanningModels<'_>,
    selected: Option<SelectedBaseline>,
) -> Vec<PricedChoice>;

crates/devtools/src/bin/stage_pipeline.rs: new CLI flag --price-enumerated <n> (default 0 = off). It fills the contract's #1004 fields in memo.enumeration. The flag is refused with --no-memo. It is also refused with --exhaustive plus measured operation costs and no frozen_calibration_factor, because the exhaustive search reports no calibration factor to cost each choice on the selection's scale.

Usage (run at this head):

# The issue's query every minute, ε 0.01, δ 0.001 (workload file committed by #1024);
# price every listed choice (the enumeration's step is the last one complete within --max-enumerated 200).
stage_pipeline --workload tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json \
  --price-enumerated 200 --out ratio.json

// ratio.json → memo.enumeration (accuracy guarantees ignored, the default)
"step": "pass2", "per_step": {"pass1": {"count": 49, "complete": true}, "pass2": {"count": 169, "complete": true}, ...},
"choices": [{"id": "E1", ..., "valid": true, "rejection": null, "cost_per_second": 90.07..., "cheaper_than_selected": false}, ...],
"priced": {"count": 169, "cheaper_than_selected": 0, "selected_cost_per_second": 78.93}

Major algorithm changes

None. This is a devtool check. Planning, search and selection are unchanged.

Before this PR

The document listed the memo's candidate DAGs without costs. Nobody could tell whether the MILP's selection was the cheapest of them.

After this PR

On small workloads, --price-enumerated prices every listed DAG and flags any valid DAG cheaper than the selection. The viewer (#1006) can mark those red.

Evidence

All measured on this head with release builds, on a shared host (load average about 450 on 56 cores).

Workload	Step priced	Choices	Valid	Cheaper than selected	Selected cost/s
planner-layering-1 (default caps)	pass1	5	2 (3 miss Q2's 100 ms latency bound)	0	4.377
planner-layering-1 (--max-enumerated 20000 --memo-enumeration-limit 20000)	concrete_states (final memo, complete)	720	396 (324 latency_bound_exceeded)	0. The selected choice is E1, the cheapest	4.377
ratio of quantiles, guarantees ignored	pass2	169	169	0 (cheapest listed 90.07)	78.93
ratio of quantiles, guarantees enforced	pass1	49	1 (48 no_accuracy_guarantee)	0	237.63
ratio of quantiles without series identity (library test)	pass1	343 (27 whole-budget)	1 (342 executor_cannot_compile or no_accuracy_guarantee)	0	(see below)

Timing: pricing the 343 choices takes 0.21–0.40 s. Pricing the 169 ratio choices adds no measurable time over the run without pricing. Pricing Example 1's 720 final-memo choices adds about 50 s.

Note on the issue's numbers: the issue's 8 groups / 343 DAGs were measured without each series' identity column. stage_pipeline lowers PromQL with it, so the two quantile_over_time(0.5, data[5m]) share a group: 7 groups, 49 Pass 1 DAGs (as #1024 also notes). The 343-DAG memo is priced in a library test. Without the identity, the executor compiles none of its per-series summaries, so only the exact DAG is valid. That DAG is not cheaper than the plan selected for the same query with identity. The default search finds no plan for the identity-less lowering, even with a 300 s budget, so that test compares against the selection of the identity lowering.

Verification

• crates/devtools/tests/memo_pricing.rs (new):
  • example1_no_listed_choice_is_cheaper_than_the_selected_plan: stage_pipeline --price-enumerated on Example 1. Every listed choice is priced, the enumeration is complete, and priced matches the Stage 3 selected cost. No valid choice is cheaper, and the latency rejections are named.
  • ratio_of_quantiles_no_listed_choice_is_cheaper_than_the_selected_plan: the same check on the issue's workload, with guarantees ignored and enforced.
  • ratio_of_quantiles_343_pass1_choices_none_cheaper_than_the_selected_plan: the identity-less Pass 1 memo enumerates completely (343 choices, 27 whole-budget). All are priced, and none is cheaper than the selection.
  • a_worse_selection_is_caught_by_cheaper_than_selected: a baseline one step worse than the cheapest (the next-cheapest cost) flags exactly the cheapest choices.
• crates/devtools/tests/stage_pipeline.rs: memo_section_lists_each_step_the_selected_choice_and_the_enumeration now also checks that without the flag priced is null and no choice has pricing fields.
• cargo fmt --all --check; cargo clippy -p asap-devtools --all-targets -- -D warnings; the tests above, run in release.

Architectural decisions

• The pricing code lives in a devtools library module and does not touch plan-selection internals. It uses the public Stage 2 (optimize_physical_asap) and Stage 3 (select_plan) entry points with the run's own PlanningModels. The alternative, the MILP's internal materialize_selected_candidates, needs the search's candidate graph, which an enumerated choice has none of.
• Each choice tries every materialization assignment (Stage 2 enumerates them all), not only the MILP's choice for that DAG. So "cheaper" means "a cheaper way to run this logical DAG exists".

Limitations and follow-up

• When the listed step is a prefix of a large final memo (for example, the ratio's concrete_states), only that prefix is priced. A clean result there does not prove the selection is optimal.
• Under cost model 3, cost_per_second is null, so cheaper_than_selected is always false.

🤖 Generated with Claude Code