Skip to content

[Refactor] Derive Pass 1's intent → family list (summary_candidates) from the family-capability table #953

Open

Description

@zzylol

Example

Pass 1 decides which sketch families to try for count_over_time(m[5m]) (AggIntent::Count) from a hand-written list, summary_candidates (crates/logical-asap-optimization/src/realization.rs:30):

AggIntent::Count { .. } => &[SketchAlgorithm::Cms, SketchAlgorithm::UnivMon],

Whether a family can answer that count (a Total readout) is decided separately, by the family-capability table, FamilyCapability::of_per_instance_sketch (crates/types/src/ir/schema/family_capability.rs:150). The IR's SummaryEstimate validation and the executor read only that table (#938).

#938 removed Count Sketch's Total from the table. It also had to delete SketchAlgorithm::CountSketch from the Count list above in the same PR (commit b1df4a8). If only the table had changed, Pass 1 would still offer a Count Sketch for every count, and the IR would reject the SummaryEstimate it builds. #697 shows the same kind of drift: Pass 1 offered KMV for distinct counts before the executor had a KMV kernel.

Change Edits today Edits after this issue
A family stops answering a statistic (#938, Count Sketch total) table + summary_candidates table
A family starts answering a statistic table + summary_candidates table
A new intent gets a sketch readout summary_candidates + statistic (per_computation_candidates.rs:3866) statistic

Concrete maintenance problem

"Which family answers which statistic" still has two sources of truth: the table, read as family → statistics, and summary_candidates, written by hand as intent → families. The two agree today. Inverting the table (intent → StatisticKind → every family whose readouts contain it) gives the same lists, except in three places:

Intent Families derived from the table summary_candidates Rule that summary_candidates encodes
Quantile Kll, DDSketch same —
Count (Total) Cms, UnivMon same —
FrequencyL2, FrequencyEntropy UnivMon same —
TopK CmsWithHeap, CountSketchWithHeap same —
Cardinality (one column) Hll, Kmv, UnivMon, Theta Hll, Kmv, UnivMon Theta has no executor implementation
Cardinality (several columns) Hll, Kmv, UnivMon, Theta Hll, Kmv UnivMon reads one value stream, not a tuple
all table order listed most-preferred first preference order

The intent → statistic step is also written twice: summary_candidates matches on the intents, and statistic(intent) (per_computation_candidates.rs:3866) maps each of the same intents to its SketchStatistic.

Other copies checked:

  • Not duplicates. These already read the table: the Pass 2 sharing and rollup rules (asap_aware_cse.rs), SummaryCoverage, the executor's validate_sketch_evaluation and supported_summary_families, and DeploymentCapabilities::supports. These hold per-family data the table does not: per-family sizing (accuracy/estimators/mod.rs size_params) and kernel construction (executor/src/summary_kernels/factory.rs).
  • Theta exclusion. The executor already records that it does not build Theta (EVALUATED_FAMILIES, executor/src/capability.rs:347). summary_candidates repeats this as a comment.
  • Local guarantees ignore the statistic. local_guarantee (accuracy/estimators/mod.rs:65) returns a guarantee for any SketchStatistic. For example, an HLL gets its relative-standard-error bound even for a Total, which the table says HLL does not answer. It is not reached today, because callers pass IR-valid estimates. It should still fail closed by checking FamilyCapability::answers first.

Behavior that must remain unchanged

For every intent, Pass 1 offers the same summary candidates in the same order, and prunes the same ones with the same PruneReason.

Before and after structure

  • Before: summary_candidates is a hand-written match from intent to families.
  • After: summary_candidates(intent) takes statistic(intent).kind() and keeps the families whose FamilyCapability readouts contain it, in a fixed preference order. It then applies the two input-shape exclusions above: UnivMon is dropped for a multi-column distinct count, and families with no implementation are dropped. Families with no implementation come from one list, not a comment.
  • local_guarantee returns None when FamilyCapability::of(family).answers(query) is false.

Why the proposed structure is minimally complex

No new type is added. The table and statistic already exist. The only new data is a preference order over SketchAlgorithm, which today is implicit in the order of the match arms.

Existing unit and end-to-end protection

  • crates/logical-asap-optimization/tests/per_computation_candidates.rs checks the candidate lists.
  • Add one test: for every approximable intent, the derived list equals today's list.
  • Add one test: changing a table row (dropping a readout) removes that family from Pass 1's candidates without any other edit.

Compatibility or migration impact

None. The change is internal to Pass 1 and does not change public types.

Related: #937 and #938 (one table for family readouts), #697 (an earlier drift between Pass 1's list and the executor), #750 (the table's heap column).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

extensibilityrefactorreplacement-strategyASAP replacement candidates and physical strategiesschemaSchema, type, key, and nullability contracts

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions