[GordonYuanyc]

Closes #953. Closes #946. Part of #947 (HydraCms and KLL past k remain; see Limitations). Part of #1012 (removes the parallel capability vocabulary and the two "not read yet" lists).

Why

Two kinds of sketch facts were hand-written in the Planner, and they drifted from each other and from asap_sketchlib.

• Capabilities ([Refactor] Derive Pass 1's intent → family list (summary_candidates) from the family-capability table #953, Stage 1 Pass 1 review: per-computation candidates over memo groups #1012). Which statistics a sketch family answers was written down twice: the family-capability table (family → statistics) and Pass 1's summary_candidates (intent → families). fix(types,executor): one family-capability table decides which statistic a family answers #938 had to edit both; Executor has no KMV kernel although Pass 1 offers KMV for distinct counts #697 is an earlier drift. The team then asked for the table itself to live in sketchlib ("make it one table in sketchlib"). asap_sketchlib asap-aware-mapping: generalize summary/CSE selection into a ReplacementStrategy trait with multiple candidates #251 added it. Reading it through a second vocabulary still left duplication:
  • SketchAlgorithm, StatisticKind and MergedError paralleled sketchlib's enums. The conversions ended in _ => return None, so a family or statistic sketchlib added would have been skipped silently.
  • Two lists named readouts that are in the table but not read: NOT_READ_YET (types) and UNEVALUATED_READOUTS (executor).
  • "Heap families answer top-k" was hard-coded in four places.
• Memory (Plan selection underestimates top-k heap memory by about 9× #946, Plan selection and the executor disagree on summary state memory #947).
  • Plan selection priced top-k heap slots at 24 B. A sketchlib TopKHeap slot holds 150–225 B plus the key.
  • Plan selection and the executor disagreed on UnivMon and DDSketch. The executor reserved a flat 4 KiB for any DDSketch, and a UnivMon state planned at 736 KiB was reserved at 2.25 MiB.
  • asap_sketchlib feat(dag-viewer): lane-based Compare/Union DAG-merge view #249 added memory_bytes / estimated_memory_bytes for these families.

What

• Pins asap_sketchlib to main 21b6357 (QueryExpr: merge BinaryOp into Compare/Arithmetic (Instance 2 of #220) #245–feat(dag-viewer): lane-based Compare/Union DAG-merge view #249, asap-aware-mapping: generalize summary/CSE selection into a ReplacementStrategy trait with multiple candidates #251, and asap-aware-mapping: group-by-lattice roll-up reuse (fine-to-coarse) as a ReplacementStrategy #254, which renamed the capability vocabulary to the Planner's names). The executor no longer needs the experimental feature, since KMV left it.
• One vocabulary. SketchAlgorithm, StatisticKind and MergedError are now asap_sketchlib's types, re-exported from asap-types. These enums are exhaustive, so a family or statistic added in sketchlib is a compile error in the Planner until it is handled.
• One table. FamilyCapability's sketch rows are sketchlib's rows (src/capability.rs), read unchanged. The exact accumulators keep Planner rows.
• Theta is deleted. sketchlib has no Theta sketch, and nothing built one.
• Pass 1's summary candidates are derived from that table. The four hard-coded "heap family" checks now read answers(TopK).
• One support declaration. The executor's EVALUATED_READOUTS lists what it evaluates per family. It replaces EVALUATED_FAMILIES + UNEVALUATED_READOUTS in the executor and NOT_READ_YET in types.
• Plan selection prices every sketch state except KLL with sketchlib's estimated_memory_bytes for the type the executor builds.
• The executor charges DDSketch and UnivMon with sketchlib's byte counts.
• KLL merge refuses a different m instead of reaching sketchlib's new panic.

For #953, Pass 1 offers the same candidates in the same order with the same prune reasons. Memory prices change for the heap families, UnivMon, DDSketch and KMV (see Before / After).

How

1. One table. FamilyCapability { merge_inputs, readouts }.

   • of_per_instance_sketch(algorithm) returns algorithm.capability()'s readouts unchanged.

   • The merge rule goes through an exhaustive From<asap_sketchlib::MergeInputRule>. The Planner's MergeInputRule adds DisjointInTime for the exact counter accumulators.

   • A Hydra kind maps to sketchlib's HydraCms / HydraCountSketch / HydraKll row.

   • The rows the Planner reads, at 21b6357. Merge: D = Disjoint, O = MayOverlap. Error: U = AsIfBuiltOverUnion, E = Exact, A = Approximate.

     Family	Row	Change from main's Planner table
     Kll	D: Quantile U, Rank U	adds Rank
     DDSketch	D: Quantile U	none (sketchlib asap-aware-mapping: group-by-lattice roll-up reuse (fine-to-coarse) as a ReplacementStrategy #254 dropped the side-field Total/Sum/Min/Max, per asap_sketchlib#250)
     Cms	D: ItemFrequency U, Total E	Total U → E
     CountSketch	D: ItemFrequency U	none
     CmsWithHeap / CountSketchWithHeap	D: ItemFrequency U, TopK A	adds ItemFrequency (the executor does not declare it)
     UnivMon	D: Total E, FrequencyL2 U, Distinct U, FrequencyEntropy A, GSum A	Total U → E; adds GSum (not declared)
     Hll / Kmv	O: Distinct U	none
     HydraCms	D: ItemFrequency U, Total U	adds Total (the executor still rejects a HydraCms group total)
     HydraCountSketch	D: ItemFrequency U	none
     HydraKll	D: Quantile U, Rank U	was no merge, Quantile A

2. Pass 1. summary_candidates(intent) maps the intent to its statistic and keeps the families in FAMILY_PREFERENCE whose row answers it. It drops UnivMon for a multi-column distinct count.
   • FAMILY_PREFERENCE is the families the Planner sizes.
   • size_params lists sketchlib's 19 other families by name (no _ arm), so a new sketchlib family fails to compile there until it is sized or listed.
3. Plan selection. sketch_shape calls each family's estimated_memory_bytes with the shape the state is assumed to reach. KLL keeps the Planner's own formula; see Architectural decisions.
4. Executor support. supported_summary_families() returns EVALUATED_READOUTS. validate_sketch_evaluation checks the table row and that list. A test checks that each declared readout is in the family's table row and that the factory and kernels answer exactly what is declared.
5. Executor memory. DDSketchAccumulator::approx_memory_bytes returns memory_bytes(). UnivMon charges sketchlib's estimated_memory_bytes plus its own fields and the key bytes it retains. The other families already charged sketchlib's memory_bytes().

Key code interfaces added / modified

asap_sketchlib src/capability.rs at 21b6357. This is the one capability table. It comes from ProjectASAP/asap_sketchlib#251 and #254; the Planner reads its types directly.

/// A public sketch family; the key of the table and the Planner's `SketchAlgorithm`. External, exhaustive, serde.
pub enum SketchAlgorithm {
    Cms, CountSketch, CountSketchWithL2Norm, CmsWithHeap, CountSketchWithHeap,
    CmsWithSpaceSaving, CountSketchWithSpaceSaving, CmsFold, CountSketchFold,
    Hll, HllHip, Kmv, Kll, KllDynamic, DDSketch, SpaceSaving, Elastic, Coco, Bloom,
    SetAggregator, UnivMon, UnivMonTwoTier, UnivMonQ, HydraCms, HydraCountSketch,
    HydraKll, HydraHll, HydraUnivMon,
}
/// A statistic one state answers; the Planner's `StatisticKind`. External, exhaustive, serde.
pub enum StatisticKind {
    Total, ItemFrequency, Quantile, Rank, Distinct, FrequencyL2, FrequencyEntropy,
    GSum, TopK, Membership, KeyDistinct, PartialKeySum, Sample,
}
/// How a merge's inputs may relate. External.
pub enum MergeInputRule { Disjoint, MayOverlap }
/// The error of a readout from a merged state; the Planner's `MergedError`. External.
pub enum MergedError { Exact, AsIfBuiltOverUnion, Approximate }
/// One row of the table. External.
pub struct FamilyCapability {
    pub merge_inputs: Option<MergeInputRule>,
    pub readouts: &'static [(StatisticKind, MergedError)],
}
impl SketchAlgorithm { pub fn capability(self) -> FamilyCapability; }

crates/types/src/ir/schema/

/// A specific sketch algorithm. Now asap_sketchlib's enum, re-exported (was a Planner enum).
pub use asap_sketchlib::SketchAlgorithm;                             // was `pub enum SketchAlgorithm { UnivMon, Kll, Cms, Hll, DDSketch, CmsWithHeap, Kmv, Theta, CountSketch, CountSketchWithHeap }`
/// Statistic kind and merged error: asap_sketchlib's, re-exported.
pub use asap_sketchlib::{MergedError, StatisticKind};                // was Planner enums of the same names
/// Concrete sizing parameters per offered family.
pub enum SketchParams { /* … */ Theta { k: u32 } /* removed */ }
/// One summary family's row of the family-capability table. Stays.
pub struct FamilyCapability {
    /// How the inputs of a merge must relate; `None` when there is no sound merge.
    pub merge_inputs: Option<MergeInputRule>,                        // unchanged
    /// Each statistic one state answers, with its merged error: sketchlib's row, unchanged.
    pub readouts: &'static [(StatisticKind, MergedError)],           // unchanged type; now sketchlib's slice
}
/// The Planner's merge rule: sketchlib's plus DisjointInTime for exact counter states. Stays.
pub enum MergeInputRule { Disjoint, DisjointInTime, MayOverlap }      // unchanged
impl From<asap_sketchlib::MergeInputRule> for MergeInputRule;         // new, exhaustive
impl FamilyCapability {
    /// sketchlib's row of a per-instance family.
    pub fn of_per_instance_sketch(algorithm: &SketchAlgorithm) -> Self; // unchanged signature; reads `algorithm.capability()`
    /// The capability of any summary family; a Hydra reads its sketchlib Hydra row.
    pub fn of(family: &FieldDataType) -> Self;                       // unchanged signature
    pub fn answers(&self, statistic: &SketchStatistic) -> bool;      // unchanged
    pub fn merged_error(&self, statistic: &SketchStatistic) -> Option<MergedError>; // unchanged
    pub fn merges_groups(family: &FieldDataType) -> bool;            // unchanged
}

crates/executor/src/capability.rs

/// What the executor evaluates from a per-instance state of each family: its supported-summary declaration.
pub(crate) const EVALUATED_READOUTS: &[(SketchAlgorithm, &[StatisticKind])]; // new: Cms Total; heaps TopK; Hll, Kmv Distinct; Kll, DDSketch Quantile; UnivMon Total, L2, Distinct, Entropy
pub(crate) const EVALUATED_FAMILIES: [SketchAlgorithm; 8];                    // removed
pub(crate) const UNEVALUATED_READOUTS: [(SketchAlgorithm, StatisticKind); 1]; // removed

crates/logical-asap-optimization/src/realization.rs

/// Every per-instance family, most-preferred first: the order Pass 1 offers candidates in.
const FAMILY_PREFERENCE: &[SketchAlgorithm];                         // new: the families the Planner sizes
/// The families Pass 1 tries for `intent`, derived from the family-capability table.
pub fn summary_candidates(intent: &AggIntent) -> Vec<SketchAlgorithm>; // was a hand-written match returning `&'static [SketchAlgorithm]`
/// The sketch statistic `intent` reads, if any. Moved here; `per_computation_candidates::statistic` is deleted.
pub(crate) fn sketch_statistic(intent: &AggIntent) -> Option<SketchStatistic>; // new (moved)

crates/plan-selection/src/lib.rs

/// CPU ops per row and bytes per state of a sketch: bytes now from sketchlib's `estimated_memory_bytes`, except KLL.
fn sketch_shape(params: &SketchParams, input_rows: StateInputRows) -> (f64, u64); // signature unchanged
/// Assumed bytes of a Count-Min / Count Sketch top-k heap key: three 16-byte labels in the executor's tuple-key encoding.
const ASSUMED_HEAP_KEY_BYTES: usize = 3 * (1 + 4 + 16);              // new
/// Assumed bytes of a UnivMon heap key: one 16-byte label value plus the executor's type tag.
const ASSUMED_VALUE_KEY_BYTES: usize = 1 + 16;                       // new
// removed: ddsketch_state_bytes, ddsketch_bins, DDSKETCH_GROW_CHUNK_BINS

crates/executor/src/summary_kernels/

impl AggregateCore for DDSketchAccumulator {
    fn approx_memory_bytes(&self) -> usize;   // new override: `inner.memory_bytes()` (was the trait's 4096 default)
}
impl AggregateCore for UnivMonAccumulator {
    fn approx_memory_bytes(&self) -> usize;   // was a hand formula (16 B/counter, 256 B/heap slot); now own fields + `UnivMon::estimated_memory_bytes` + retained key bytes
}
impl AggregateCore for DatasketchesKLLAccumulator {
    fn merge_with(&self, other: &dyn AggregateCore) -> Result<Box<dyn AggregateCore>, KernelError>; // signature unchanged; now refuses a different k or m (was k only)
}

End to end, with numbers from the tests below:

quantile_over_time(0.99, m[5m])                          // the user's query
→ AggIntent::Quantile                                    // frontend
→ summary_candidates(..) = [Kll, DDSketch]               // Pass 1, from sketchlib's table (unchanged list)
→ sketch_shape(DDSketch α = 0.01, rows unknown)          // plan selection prices 37,360 B (was 18,576 B)
→ DDSketchAccumulator::approx_memory_bytes()             // executor reserves memory_bytes(): 26,536 B after
                                                         // 10,000 signed rows spread over nine decades (was 4,096 B flat)

Major algorithm changes

Pass 1 candidate selection (#953).

• Old: a match from intent to a hand-written family list.
• New:
  1. sketch_statistic(intent);
  2. filter FAMILY_PREFERENCE by FamilyCapability::of_per_instance_sketch(f).answers(statistic);
  3. drop UnivMon for a multi-column distinct count.
• Invariant: identical lists for every intent. Checked by summary_candidates_keep_the_hand_written_lists.

State-bytes cost model (#946, #947).

• Old: hand-copied sketchlib internals.
• New: estimated_memory_bytes per family.
  • Exact for Count-Min, Count Sketch, HLL and for KMV at its distinct bound.
  • An upper bound for DDSketch (worst case over insertion orders and merges), the heap families and UnivMon.
• Invariant: the executor's charge ≤ the priced bytes plus the accumulator's own fields ≤ 1.5 × the charge once the state is full. Checked by state_bytes_bound_the_executors_charge. KLL past k is the exception (see Limitations).
• Complexity: unchanged, O(1) per priced state. UnivMon's charge stays O(1) per row instead of walking its heaps.

Before this PR

• Capabilities. sketchlib could say what a family answers. The Planner kept its own copy in two places, read sketchlib through a second vocabulary, and kept two lists of readouts nobody reads.
• Pricing.
  • Plan selection priced a Count-Min-with-heap state (5×1024, heap 1000) at 64,960 B, of which the heap was 24,000 B.
  • A UnivMon state (16 layers, 5×1024, heap 256) was priced at 753,664 B and reserved at about 2,359,296 B, so a plan the Planner accepted could fail with MemoryLimit.
  • Every DDSketch state was reserved at 4,096 B, whatever it held.

After this PR

• Capabilities. A row change in sketchlib changes Pass 1 and IR validation with no Planner edit. A new family or statistic is a compile error until the Planner handles it. The executor's support changes only when EVALUATED_READOUTS is edited.
• Pricing. Plan selection and the executor read the same sketchlib functions:
  • Count-Min-with-heap: priced at 328,184 B (heap 287,056 B with the 63 B key assumption; 224,056 B with u64 keys, as Plan selection underestimates top-k heap memory by about 9× #946 measured).
  • UnivMon: priced at 1,631,880 B; reserved at 1,562,248 B plus its own fields and retained key bytes.
  • DDSketch: priced at sketchlib's worst case (37,360 B at α = 0.01, rows unknown); reserved at what it holds.

Evidence

• Execution example: the end-to-end example above, with numbers from state_bytes_bound_the_executors_charge.
• Before/after measurement: the byte figures in Before / After come from sketchlib's estimates and the test's executor states.
• Screenshot, diagram: not applicable.

Verification

Unit tests

• state_bytes_bound_the_executors_charge (new, plan-selection): builds every per-group family through the executor's factory and feeds rows shaped as the pricing assumes. It fails if plan selection and the executor disagree beyond the stated bounds. Priced ÷ charged with the state full:

  Family	Priced ÷ charged
  Count-Min, Count Sketch, HLL, KMV	1.000
  DDSketch	1.408
  Count-Min / Count Sketch with heap (16×1, heap 256)	1.404
  UnivMon against its charge	1.000
  UnivMon against the bytes it holds (named 1.6× bound)	1.523
  KLL up to k	1.000

  Runs in 2.1 s.

• family_capability_table (types): every sketch row equals sketchlib's row (Hydra kinds included), and every exact row is pinned.

• supported_summary_families_match_the_factory_and_evaluations (executor): each declared readout is in the family's table row, and the factory and kernels answer exactly what is declared.

• summary_candidates_keep_the_hand_written_lists, a_dropped_readout_drops_the_family, preference_order_names_each_family_once (new, logical-asap-optimization):

  • the derived lists equal the old ones;
  • dropping a readout drops that family from Pass 1;
  • the preference order has no duplicate.

• local_guarantee_needs_a_statistic_the_family_answers (new): a local guarantee is None for a statistic the family does not answer.

• merge_rejects_a_different_k_or_m (extended, executor).

End-to-end tests

• Whole workspace after the vocabulary change (cargo test --workspace --no-fail-fast): 2,236 passed, 38 failed, 33 ignored. All 38 failures came from one intermediate choice: dropping Count-Min's and UnivMon's Total, which took away approximate count(). That choice was reverted in sketchlib asap-aware-mapping: group-by-lattice roll-up reuse (fine-to-coarse) as a ReplacementStrategy #254, since both totals are read from the sketch's own counters. The 13 affected suites then pass: 873 tests, including asap-plan-selection --lib (374), asap-logical-asap-optimization --lib (233) and its per_computation_candidates target (35), stage_pipeline (47), and the executor, PromQL count and integration targets.
• Earlier runs of the memory-pricing commits are unchanged: 2,275 passed, 0 failed, 33 ignored.

Other checks

• cargo clippy --workspace --all-targets --all-features --locked -- -D warnings: clean.
• cargo fmt --all -- --check: clean.
• Not run: the dag-viewer Python/Node tests, tools/verify_metricsql_external_consumer.sh, tools/verify_metricsql_vendored_baseline.py.

Architectural decisions

• The table lives in sketchlib; implementation status stays in the Planner.
  • The exact accumulators (including DisjointInTime) stay in the Planner.
  • The table mirrors sketchlib unfiltered. What the executor evaluates is its own declaration (EVALUATED_READOUTS), a positive list, so a sketchlib bump cannot widen the declared support.
  • Theta is deleted: sketchlib has no Theta sketch and nothing built one. Adding Theta to sketchlib is the way back.
• sketchlib's names are the Planner's. sketchlib asap-aware-mapping: group-by-lattice roll-up reuse (fine-to-coarse) as a ReplacementStrategy #254 renamed its enums and derives serde, so SketchAlgorithm / StatisticKind serialize as before ("Cms", "Quantile", …) and deployment config is unchanged.
• Derived Ord follows sketchlib's variant order, not the Planner's old order. Every test passes with it.
• Cells that change but are harmless.
  • Count-Min and UnivMon Total: AsIfBuiltOverUnion → Exact. Every reader accepts both alike.
  • Heap families' ItemFrequency and HydraCms's Total are now in the table. IR validation accepts them, but the executor does not declare them (HydraCms total is still rejected by validate_sketch_evaluation), and Pass 1 has no intent that reads them.
  • UnivMon's readout order follows sketchlib's.
  • HydraKll becomes mergeable (Disjoint, Quantile AsIfBuiltOverUnion): sketchlib asap-aware-mapping: generalize summary/CSE selection into a ReplacementStrategy trait with multiple candidates #251 made that merge sound. Nothing builds a HydraKll yet; Pass 1 offers no Hydra and the executor builds only HydraCms.
• KLL keeps the Planner's formula. sketchlib's KLL estimate also counts a CDF cache the executor never builds. With it, a fine-gcd plan choice flipped the wrong way: priced cheaper while holding 19% more. Rejected: taking sketchlib's estimate now.
• DDSketch is priced at sketchlib's worst case. This is the owner's decision, so a Planner-accepted plan does not hit MemoryLimit. Rejected: keeping the Planner's clustered, non-negative model.
• UnivMon's charge uses sketchlib's estimate, not memory_bytes(). memory_bytes() walks every heap, and the build charges memory after every row (about 50× slower).
• plan-selection dev-depends on asap-executor for the Plan selection and the executor disagree on summary state memory #947 test, rather than exposing sketch_shape publicly.

Limitations and follow-up

• HydraCms (Plan selection and the executor disagree on summary state memory #947). Plan selection prices an independent Count-Min per group, while the executor holds one shared grid; they agree only at about 79 groups (5 × 32 grid of 64 × 3 cells). Pass 1 offers no Hydra today. This is left out of the test and needs its own fix.
• KLL past k rows (Plan selection and the executor disagree on summary state memory #947). Main's formula can price below the executor's charge by up to sketchlib's compaction scratch, 2·(k−1) items. The test checks only that slack. The fix waits for a sketchlib KLL estimate without the CDF cache.
• DDSketch prices rise 2–16× for narrow-range data. Small panes are hit most. This may make maintained DDSketch states less attractive than they really are.
• Key-size assumptions. 63 B for Count-Min / Count Sketch heap keys and 17 B for UnivMon keys are assumptions: no workload statistic gives label sizes.
• asap-types now depends on asap_sketchlib (a git dependency until 0.4.1 is released), so asap-types cannot be published to crates.io before then.
• Measured-table interpolation. sketch_shape's bytes are the size axis there, so deployments with a measured table get slightly different interpolation weights.
• Invalid α. An α outside (0, 1) is priced as an empty DDSketch, as before.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

[zzylol]

@GordonYuanyc I pushed one commit on top of yours (af848d9), at the owner's request, to finish removing the duplicated capability facts tracked in #1012. Your memory-pricing commits are unchanged.

sketchlib side (merged as ProjectASAP/asap_sketchlib#254, 21b6357)

• The capability table now uses the Planner's names: SketchAlgorithm, StatisticKind, MergeInputRule, MergedError::Approximate, Cms*, DDSketch.
• The enums are exhaustive and serde-derived.
• Per asap_sketchlib#250, rows no longer list side-field aggregates (DDSketch Total/Sum/Min/Max and others). Count-Min and UnivMon keep Total, because they read it from their own counters, and approximate count() still uses them.
• UnivMon-Q's GSum/Distinct/Quantile/Rank are now Approximate.

This PR

• The Planner uses sketchlib's SketchAlgorithm / StatisticKind / MergedError directly. Gone: the Planner enums, Readouts, NOT_READ_YET, and the _ => return None conversions. A new sketchlib family or statistic is now a compile error here.
• Theta is deleted, along with UNIMPLEMENTED_FAMILIES.
• The executor declares what it supports in one positive list, EVALUATED_READOUTS, replacing EVALUATED_FAMILIES and UNEVALUATED_READOUTS. This also fixes the review point that a sketchlib bump could widen what the executor declares.
• The four hard-coded "heap families answer top-k" checks now read answers(TopK). The pcc.rs statistic() wrapper is gone.

Tests

• The full workspace ran on the intermediate state. The only failures (38) came from briefly dropping CMS/UnivMon Total, which was then reverted.
• After the revert, all 13 affected suites pass (873 tests), as do clippy (-D warnings) and fmt.

The PR body's What / How / interfaces / Verification / decisions sections are updated. Please take a look, especially the derived-Ord note: SketchAlgorithm now sorts in sketchlib's variant order.

🤖 Generated with Claude Code