Skip to content

feat: jointly price multi-input plans and mixed schedules - #940

Merged
zzylol merged 7 commits into
feat/supplier-dependent-direct-costfrom
feat/multi-input-schedule-cost
Oct 26, 2026
Merged

zzylol merged 7 commits into
feat/supplier-dependent-direct-costfrom
feat/multi-input-schedule-cost

Conversation

@zzylol

@zzylol zzylol commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem and resulting behavior

Direct pricing previously admitted one supplier per operator and only a limited set of recurrence/range combinations. Multi-input cost could not preserve correlations between supplier configurations; large or mixed schedules fell back before MILP selection.

This PR prices complete ordered input tuples and represents mixed execution demand with exact schedule atoms. Consumer connections, pricing configurations, retention tiers, execution frequency and holding costs are selected jointly. It builds on #939 and introduces no new runtime operator or scheduling policy.

Scenarios handled

Scenario Before this PR After this PR
Binary arithmetic or a supported relational join Unary-only direct interface. All input positions are priced together; tuple selectors prevent an unpriced Cartesian combination.
m + m / two positions reading one producer Multi-input interface unavailable. Positions remain distinct; shared upstream execution is counted once per execution boundary.
Two approximate inputs, such as a DDS quantile ratio Multi-sketch closure excluded. Distinct sketches are checked against divided failure budgets before admission; final composed guarantees remain mandatory.
Five queries with fixed, estimated and nonrepeating demand Small subset / integer lattice restrictions. Exact fixed/phase atoms, additive estimates and batch-max terms work together; the production-path test requires a validated joint-cost solve.
59/61/67/71-second periods and arbitrary rate ratios Rate lattice could exceed 4096 ticks. A linear weighted-lifetime equation needs neither LCM rate ticks nor denominator enumeration.
Different range steps/spans over shared panes Matching contexts only. Evaluation/request demand stays separate, and a selected retention tier must equal the largest selected reader span.
Ingestion panes versus eligible query-kept panes Query-kept ranges not certified directly. Ingestion and refresh costs follow their respective execution boundaries; real evaluator parity tests cover both.
Very long ranges with a few different steps Timestamp enumeration could dominate construction and validation. Exact arithmetic intersection counts compress coverage; the evaluator also avoids timestamp expansion for these cases.
Repeated SQL distinct queries with a confidence target Seed attempts could repeatedly consume the budget on an estimator lacking a confidence certificate. The feasibility seed excludes unresolved confidence contracts; the full candidate catalog and final accuracy validation remain unchanged.
Zero expected query demand Positive-rate admission only. Zero-rate readers use the evaluator's longest-latency fallback when no positive-weight reader is selected.

Key code interfaces added / modified

  • CandidatePriceOption.compatible_input_tuples and required_range_span_ms: complete supplier combinations and selected-reader retention contexts.
  • PriceOptionProjection: tuple/edge consistency and retention maximum witnesses while retaining original physical IDs.
  • QueryScheduleRates / general_schedule::encode: demand atoms and linear holding-time constraints.
  • cost::execution_demand: exact arithmetic range intersection counting shared with schedule construction.

Major algorithm changes

  • Stream local input tuples with deadline/domain bounds; merge identical priced interfaces, deduplicate shared ancestors, and preserve confidence/source certificates. Whole-result bounds account conservatively for branching logical expressions.
  • For an unmaterialized node, enforce sum(rate * reach * mean) = sum(rate * reach * query_latency) using binary/continuous gates. Materialized state uses the longest reached latency; query-kept state refreshes per request.
  • Fixed intervals/phases follow existing evaluator semantics; unknown phase joins the first selected known phase, estimated rates add, and nonrepeating batches take a maximum. No timestamp-union policy is invented for different fixed intervals.
  • Common-step ranges use nested counts; up to 12 distinct range geometries use arithmetic intersections and Möbius inversion, with checked LCM arithmetic. Larger domains retain exact bounded offset construction.

Scope

Only currently supported operators/timing assignments are offered. This does not enable new ingestion joins, Concat/SetOp executor bindings, or unsupported materializations. Extreme domains, exhausted budgets and ill-conditioned coefficients report explicit fallback; nothing is rounded or silently omitted. Restricted seed bounds remain separate from full-catalog bounds. This is modeled cost, not measured runtime performance, and does not implement the proposed general DP decomposition.

Validation

  • cargo test -p asap-plan-selection --lib: 372 passed after removing the obsolete encoders.
  • Integration suites joint_search_equivalence, production_rollup_workloads, rollup_controlled, sketch_rollup_planning: 23 passed.
  • Strict clippy (-D warnings) passed for plan-selection library/tests and the production workload integration target; formatting and diff checks passed.
  • Repeated SQL confidence-seed regression failed before the fix, passed afterward, then passed in the complete integration run. Accuracy targets and the 10-second budget were unchanged.
  • The five-query mixed binary workload produced one validated joint-price plan at modeled cost 1.8260036842767293e-13/s, with no direct-cost fallback. The full search used its 30-second budget and did not prove global optimality.
  • Full workspace CI remains pending; these results are local targeted acceptance, not a claim that every workspace test or runtime schedule is covered.

CI fixes and removal of duplicate schedule implementations

Scenario Before this follow-up After this follow-up
Optional schedule-atom pricing runs out of construction time Joint extraction fell back, but fixed extraction returned the same unsupported-profile error as fatal. Both adapters skip unsupported profiles; both share the reserved construction deadline, leaving budget for validated candidate search.
Malformed recurrence, such as a zero fixed interval Invalid input fails. Still fails; fallback does not hide invalid input.
Nine-query no-raw-data workload / workload example archive CI returned no selected plan or aborted during schedule construction. Both unchanged production acceptance tests pass locally with original accuracy and timing requirements.
Mixed schedule representation General atoms, old subset tables and old fixed lattices coexisted in production types/dispatch. One QueryScheduleRates representation and one mixed-schedule encoder. Old encoders and fields are deleted; behavioral selection tests use the current encoder and independent evaluator-based expected costs.

The equal-schedule fast path remains a specialization of the same cost contract. The full DAG evaluator remains the validation authority, not a second planner. No general DP implementation is introduced.

Validation: deterministic schedule-timeout regression failed before the fix and passes afterward; nine-query planning/installation and workload examples pass. Final local library tests: 372 passed. All 23 search-equivalence and rollup integration tests passed, as did strict plan-selection clippy, formatting and diff checks. Full remote CI passed before merge.

Final local stage-pipeline acceptance on the cleaned implementation: 47 passed, 0 failed, including the complete workload archive and nine-query installable planning scenarios. Full remote CI remains pending.

Candidate-cost overflow recovery

Scenario: Example 2 can select a physical configuration whose analytical cost arithmetic overflows. Previously that stopped search before a usable incumbent. Scalar-price search now skips the unpriceable assignment, records cost_not_representable, and continues looking for a validated plan. Unknown cost is not infeasibility: exclusions of unpriceable assignments invalidate full-domain optimality/infeasibility claims and reset the reported bound conservatively. Other evaluator errors remain fatal. The new regression covers both a later feasible candidate and an entirely unpriceable domain; Example 2 retains its original requirements and budget.

TopK contract regression

Scenario: a temporal TopK search visits a tighter error allocation first, with heap capacity 200 instead of the default 100. The integration test now verifies every visited heap-backed candidate preserves the requested k, minimum capacity, entity identity and update expression, rather than requiring a fixed first-candidate capacity. Default sizing remains covered by its unit test. The original assertion failed locally; all 13 tests in promql_to_post_asap pass after this correction.

Stable sketch execution and coverage fixtures

Scenarios: a search archive may contain tightened, large sketch configurations or multiple KLL capacities. Rollup execution and coverage tests now use the existing explicit local-rule fixtures, so they test all intended sketch rules and coverage semantics without depending on solver visitation. The rollup memory limit and result assertions remain unchanged. The former fixtures failed locally; corrected rollup (3) and coverage (8) tests pass. The integration package was exercised in batches: 330 passed, 23 pre-existing ignored; affected tests also pass strict clippy. Full remote CI passed before merge.

Merge path: this stacked PR is merged into #939’s branch, then carried to main by #939. A local merge check confirmed the combined tree exactly matches this CI-tested head.

@zzylol
zzylol merged commit ab8645a into feat/supplier-dependent-direct-cost Oct 26, 2026
2 checks passed
zzylol added a commit that referenced this pull request Oct 26, 2026
* docs: explain conditional DP and MILP decomposition

* docs: clarify how DP reduces MILP candidate choices

* docs: require feasibility-only accuracy and direct DAG costing

* fix: remove approximate-candidate accuracy surcharges

* test: validate production KMV rollup selection under budget

* fix: keep unresolved estimator contracts out of feasibility seed

* docs: explain jointly computed execution DAG costs and coverage

* feat: price query-time execution DAGs directly in joint MILP

* docs: describe expanded direct pricing and remaining coverage

* feat: extend direct pricing to fixed sketches and query schedules

* docs: record direct ingestion pricing and its certificates

* feat: directly price fixed-retention ingestion panes

* feat: directly price rebuilt and retained query-time panes

* docs: describe direct pane cost coverage and remaining contexts

* feat: price HLL panes and fixed-worker compaction directly

* docs: clarify direct DAG pricing coverage and numerical validation

* feat: extend direct pricing and validate production rollup workloads

* docs: update direct pricing coverage and materialization constraints

* test: isolate planner presentation checks from exhaustive panes

* feat: jointly price supplier and worker configurations

* fix(planner): archive only completed candidate evaluations

* fix(planner): archive only completed candidate evaluations

* test(executor): use explicit frequency operator candidates

* test(executor): use explicit frequency operator candidates

* fix(planner): exclude unresolved confidence from feasibility seeds

* fix(planner): continue past unpriceable candidates without false certificates

* test(planner): validate TopK contracts across visited capacities

* test(planner): decouple sketch execution fixtures from search archives

* feat: jointly price multi-input plans and mixed schedules (#940)

* feat: jointly price multi-input plans and mixed schedules

* fix(planner): preserve search after unsupported direct pricing

* refactor(milp): keep one mixed-schedule representation and encoder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant