Repository navigation
fix(planner): Stage 3 rejects plans the executor cannot compile; no PromQL top-k heap over its instant vector - #805
Merged
Conversation
…QL top-k offers no heap over its instant vector With accuracy guarantees ignored, counter top-k and rate queries selected exact per-series panes followed by a query-time CountSketchWithHeap the executor refuses to compile. Stage 3 now asks the deployment executor's compiler (ExecutorCompileCheck; plan_installable defaults to the Planner's executor) and rejects such a candidate as ExecutorCannotCompile. Pass 1 no longer offers a PromQL top-k's own heap over its instant vector, one row per item, which an exact top-k dominates (TopKHeapOverOneRowPerItem); the fused whole-expression heap stays. Exact top-k answers over PromQL rows without a series identity keep their label columns, so topk(k, sum by (job) (...)) answers each job.
zzylol
force-pushed
the
fix/stage3-executor-compile-check
branch
from
October 21, 2026 07:23
10abaac to
fefe284
Compare
This was referenced Oct 21, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#799 lets the MVP benchmark compare plans on cost alone. With guarantees ignored and no raw data retained, all 60 counter top-k and rate queries in the ASAPQuery random workloads (
q08-topk-count,q09-rate, e.g.topk(k, sum by (G) (increase(data_total[W])))) select a plan the executor refuses to compile. Those plans keep exact Increase/Rate panes, then at query time build a CountSketchWithHeap and read its top-k estimate. The executor fails with "invalid keyed evaluation shape or capacity" or "summary update column missing or ambiguous".What
RejectionCause::ExecutorCannotCompile, and the reason is the executor's own error.plan_installableruns the Planner executor's owncompile, so the planner and the executor can't drift. It compiles each stored output's build and the query run, exactly as a runtime installs the plan.PruneReason::TopKHeapOverOneRowPerItem. The fused whole-expression heap over the inner aggregate's input, where an item's rows repeat, is still offered. SQL top-k keeps its heaps.sum by (job), the exact top-k now keeps the label columns. Before,topk(10, sum by (job) (...))answered values with nojoblabel.How
PlanSelector::offerasksmodels.executor_compile_checkonly for a candidate that could still be selected, after pricing. Compiling is the costliest check, so it runs last. A rejected candidate drops out, and the next best is selected.PlannerExecutorCompileCheckrebuilds what the runtime installs: it applies timings, expands pane chains, and builds anInstallablePlan. ThenInstallablePlan::check_compilesrunsasap_executor::physical_planner::compile. It compiles each stored output over its raw-sample windows and stored inputs, then the query run over the stored outputs it reads and the query-time windows. A plan that isn't installable for another reason, such as a scalar root, passes the check, soplan_installablestill reports that reason as before.prune_topk_heaps_over_one_row_per_itemruns afteradd_whole_expression_targets. It applies only when every scan below the top-k reads a time series.implement_topkappends the child's other Utf8 label columns after[keys.., topk_k]when PromQL rows have no series identity.Key code interfaces added / modified
crates/plan-selection/src/pass/mod.rscrates/plan-selection/src/lib.rscrates/planner/src/installable.rscrates/logical-asap-optimization/src/per_computation_candidates.rsUsage
Before this PR
With guarantees ignored, 60 of 60 counter top-k and rate queries selected a query-time heap over exact per-series values, and the executor refused to compile the plan. Stage 3 had no way to know that. Exact
topk(k, sum by (G) (...))answers also lost theirGlabels.After this PR
Those queries select the exact plan, which compiles and answers exactly. Any future candidate the executor can't compile is rejected with a named cause instead of being selected.
Verification
cargo test --workspace --locked --no-fail-faston the head, rebased on main at ee26f24: 151 test binaries, 1,822 passed, 0 failed, 36 ignored.runtime_install::ignoring_guarantees_counter_topk_plans_compile_and_answer_exactly:topk(10, sum by (job) (increase|rate(data_total[5m])))with the setting on and no raw data. Before the fix, planning selected the query-time heap and the executor refused it. Now the selected plan has no sketch,check_compilespasses, and the answers equalquery_time_referenceand the exact top 10 jobs, with theirjoblabels, to 1e-9.runtime_install::check_compiles_reports_the_executors_error: a plan with a changed answer type is refused with the executor's reason.plan_selection::tests::a_plan_the_executor_cannot_compile_is_rejected: the refused candidate is rejected asExecutorCannotCompilewith the executor's reason, and another is selected.stage_pipeline::a_topk_heap_over_one_row_per_item_is_pruned_with_its_reason: replacesuncomposable_candidate_is_recorded_with_its_error. That test's uncomposable choice was exactly such a heap, and no PromQL candidate is uncomposable any more.default_candidate_limit_is_ten_thousandnow uses quantile queries, three candidates each, because top-k now has one Pass 1 candidate.promql_topk_over_grouped_aggregate_sorts_by_its_measurenow expects thejobcolumn.topk(10, sum by (job) (increase(data_total[5m])))rejected P4-m1 asExecutorCannotCompile("summary update column missing or ambiguous") and selected exact P2-m1.asapquery_random_workloadfrom the joint-search branch) on all 60q08-topk-countandq09-ratequeries across seed-862, seed-862-long and seed-862-x5. It used--no-raw --failure-probability 0.3 --checkwith the setting on. All 60 planned, compiled and ran. Every evaluation matched the exact answer: recall 1.0, max relative error 0.Follow-ups, not in this PR:
SummaryEstimate{TopK}output shape still doesn't match the executor's keyed evaluation (the ignoredweighted_topk_bindingtest).plan_installablenow rejects it whenever it would be selected.