Repository navigation
[Refactor] Derive Pass 1's intent → family list (summary_candidates) from the family-capability table #953
Description
Example
Pass 1 decides which sketch families to try for count_over_time(m[5m]) (AggIntent::Count) from a hand-written list, summary_candidates (crates/logical-asap-optimization/src/realization.rs:30):
AggIntent::Count { .. } => &[SketchAlgorithm::Cms, SketchAlgorithm::UnivMon],Whether a family can answer that count (a Total readout) is decided separately, by the family-capability table, FamilyCapability::of_per_instance_sketch (crates/types/src/ir/schema/family_capability.rs:150). The IR's SummaryEstimate validation and the executor read only that table (#938).
#938 removed Count Sketch's Total from the table. It also had to delete SketchAlgorithm::CountSketch from the Count list above in the same PR (commit b1df4a8). If only the table had changed, Pass 1 would still offer a Count Sketch for every count, and the IR would reject the SummaryEstimate it builds. #697 shows the same kind of drift: Pass 1 offered KMV for distinct counts before the executor had a KMV kernel.
| Change | Edits today | Edits after this issue |
|---|---|---|
| A family stops answering a statistic (#938, Count Sketch total) | table + summary_candidates |
table |
| A family starts answering a statistic | table + summary_candidates |
table |
| A new intent gets a sketch readout | summary_candidates + statistic (per_computation_candidates.rs:3866) |
statistic |
Concrete maintenance problem
"Which family answers which statistic" still has two sources of truth: the table, read as family → statistics, and summary_candidates, written by hand as intent → families. The two agree today. Inverting the table (intent → StatisticKind → every family whose readouts contain it) gives the same lists, except in three places:
| Intent | Families derived from the table | summary_candidates |
Rule that summary_candidates encodes |
|---|---|---|---|
Quantile |
Kll, DDSketch | same | — |
Count (Total) |
Cms, UnivMon | same | — |
FrequencyL2, FrequencyEntropy |
UnivMon | same | — |
TopK |
CmsWithHeap, CountSketchWithHeap | same | — |
Cardinality (one column) |
Hll, Kmv, UnivMon, Theta | Hll, Kmv, UnivMon | Theta has no executor implementation |
Cardinality (several columns) |
Hll, Kmv, UnivMon, Theta | Hll, Kmv | UnivMon reads one value stream, not a tuple |
| all | table order | listed most-preferred first | preference order |
The intent → statistic step is also written twice: summary_candidates matches on the intents, and statistic(intent) (per_computation_candidates.rs:3866) maps each of the same intents to its SketchStatistic.
Other copies checked:
- Not duplicates. These already read the table: the Pass 2 sharing and rollup rules (
asap_aware_cse.rs),SummaryCoverage, the executor'svalidate_sketch_evaluationandsupported_summary_families, andDeploymentCapabilities::supports. These hold per-family data the table does not: per-family sizing (accuracy/estimators/mod.rssize_params) and kernel construction (executor/src/summary_kernels/factory.rs). - Theta exclusion. The executor already records that it does not build Theta (
EVALUATED_FAMILIES,executor/src/capability.rs:347).summary_candidatesrepeats this as a comment. - Local guarantees ignore the statistic.
local_guarantee(accuracy/estimators/mod.rs:65) returns a guarantee for anySketchStatistic. For example, an HLL gets its relative-standard-error bound even for aTotal, which the table says HLL does not answer. It is not reached today, because callers pass IR-valid estimates. It should still fail closed by checkingFamilyCapability::answersfirst.
Behavior that must remain unchanged
For every intent, Pass 1 offers the same summary candidates in the same order, and prunes the same ones with the same PruneReason.
Before and after structure
- Before:
summary_candidatesis a hand-writtenmatchfrom intent to families. - After:
summary_candidates(intent)takesstatistic(intent).kind()and keeps the families whoseFamilyCapabilityreadouts contain it, in a fixed preference order. It then applies the two input-shape exclusions above: UnivMon is dropped for a multi-column distinct count, and families with no implementation are dropped. Families with no implementation come from one list, not a comment. local_guaranteereturnsNonewhenFamilyCapability::of(family).answers(query)is false.
Why the proposed structure is minimally complex
No new type is added. The table and statistic already exist. The only new data is a preference order over SketchAlgorithm, which today is implicit in the order of the match arms.
Existing unit and end-to-end protection
crates/logical-asap-optimization/tests/per_computation_candidates.rschecks the candidate lists.- Add one test: for every approximable intent, the derived list equals today's list.
- Add one test: changing a table row (dropping a readout) removes that family from Pass 1's candidates without any other edit.
Compatibility or migration impact
None. The change is internal to Pass 1 and does not change public types.
Related: #937 and #938 (one table for family readouts), #697 (an earlier drift between Pass 1's list and the executor), #750 (the table's heap column).
Activity