Repository navigation
feat: jointly price multi-input plans and mixed schedules - #940
Merged
zzylol merged 7 commits intoOct 26, 2026
Merged
Conversation
…put-schedule-cost
…put-schedule-cost
…put-schedule-cost
…put-schedule-cost
zzylol
added a commit
that referenced
this pull request
Oct 26, 2026
* docs: explain conditional DP and MILP decomposition * docs: clarify how DP reduces MILP candidate choices * docs: require feasibility-only accuracy and direct DAG costing * fix: remove approximate-candidate accuracy surcharges * test: validate production KMV rollup selection under budget * fix: keep unresolved estimator contracts out of feasibility seed * docs: explain jointly computed execution DAG costs and coverage * feat: price query-time execution DAGs directly in joint MILP * docs: describe expanded direct pricing and remaining coverage * feat: extend direct pricing to fixed sketches and query schedules * docs: record direct ingestion pricing and its certificates * feat: directly price fixed-retention ingestion panes * feat: directly price rebuilt and retained query-time panes * docs: describe direct pane cost coverage and remaining contexts * feat: price HLL panes and fixed-worker compaction directly * docs: clarify direct DAG pricing coverage and numerical validation * feat: extend direct pricing and validate production rollup workloads * docs: update direct pricing coverage and materialization constraints * test: isolate planner presentation checks from exhaustive panes * feat: jointly price supplier and worker configurations * fix(planner): archive only completed candidate evaluations * fix(planner): archive only completed candidate evaluations * test(executor): use explicit frequency operator candidates * test(executor): use explicit frequency operator candidates * fix(planner): exclude unresolved confidence from feasibility seeds * fix(planner): continue past unpriceable candidates without false certificates * test(planner): validate TopK contracts across visited capacities * test(planner): decouple sketch execution fixtures from search archives * feat: jointly price multi-input plans and mixed schedules (#940) * feat: jointly price multi-input plans and mixed schedules * fix(planner): preserve search after unsupported direct pricing * refactor(milp): keep one mixed-schedule representation and encoder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and resulting behavior
Direct pricing previously admitted one supplier per operator and only a limited set of recurrence/range combinations. Multi-input cost could not preserve correlations between supplier configurations; large or mixed schedules fell back before MILP selection.
This PR prices complete ordered input tuples and represents mixed execution demand with exact schedule atoms. Consumer connections, pricing configurations, retention tiers, execution frequency and holding costs are selected jointly. It builds on #939 and introduces no new runtime operator or scheduling policy.
Scenarios handled
m + m/ two positions reading one producerKey code interfaces added / modified
CandidatePriceOption.compatible_input_tuplesandrequired_range_span_ms: complete supplier combinations and selected-reader retention contexts.PriceOptionProjection: tuple/edge consistency and retention maximum witnesses while retaining original physical IDs.QueryScheduleRates/general_schedule::encode: demand atoms and linear holding-time constraints.cost::execution_demand: exact arithmetic range intersection counting shared with schedule construction.Major algorithm changes
sum(rate * reach * mean) = sum(rate * reach * query_latency)using binary/continuous gates. Materialized state uses the longest reached latency; query-kept state refreshes per request.Scope
Only currently supported operators/timing assignments are offered. This does not enable new ingestion joins, Concat/SetOp executor bindings, or unsupported materializations. Extreme domains, exhausted budgets and ill-conditioned coefficients report explicit fallback; nothing is rounded or silently omitted. Restricted seed bounds remain separate from full-catalog bounds. This is modeled cost, not measured runtime performance, and does not implement the proposed general DP decomposition.
Validation
cargo test -p asap-plan-selection --lib: 372 passed after removing the obsolete encoders.joint_search_equivalence,production_rollup_workloads,rollup_controlled,sketch_rollup_planning: 23 passed.-D warnings) passed for plan-selection library/tests and the production workload integration target; formatting and diff checks passed.1.8260036842767293e-13/s, with no direct-cost fallback. The full search used its 30-second budget and did not prove global optimality.CI fixes and removal of duplicate schedule implementations
QueryScheduleRatesrepresentation and one mixed-schedule encoder. Old encoders and fields are deleted; behavioral selection tests use the current encoder and independent evaluator-based expected costs.The equal-schedule fast path remains a specialization of the same cost contract. The full DAG evaluator remains the validation authority, not a second planner. No general DP implementation is introduced.
Validation: deterministic schedule-timeout regression failed before the fix and passes afterward; nine-query planning/installation and workload examples pass. Final local library tests: 372 passed. All 23 search-equivalence and rollup integration tests passed, as did strict plan-selection clippy, formatting and diff checks. Full remote CI passed before merge.
Final local stage-pipeline acceptance on the cleaned implementation: 47 passed, 0 failed, including the complete workload archive and nine-query installable planning scenarios. Full remote CI remains pending.
Candidate-cost overflow recovery
Scenario: Example 2 can select a physical configuration whose analytical cost arithmetic overflows. Previously that stopped search before a usable incumbent. Scalar-price search now skips the unpriceable assignment, records
cost_not_representable, and continues looking for a validated plan. Unknown cost is not infeasibility: exclusions of unpriceable assignments invalidate full-domain optimality/infeasibility claims and reset the reported bound conservatively. Other evaluator errors remain fatal. The new regression covers both a later feasible candidate and an entirely unpriceable domain; Example 2 retains its original requirements and budget.TopK contract regression
Scenario: a temporal TopK search visits a tighter error allocation first, with heap capacity 200 instead of the default 100. The integration test now verifies every visited heap-backed candidate preserves the requested k, minimum capacity, entity identity and update expression, rather than requiring a fixed first-candidate capacity. Default sizing remains covered by its unit test. The original assertion failed locally; all 13 tests in
promql_to_post_asappass after this correction.Stable sketch execution and coverage fixtures
Scenarios: a search archive may contain tightened, large sketch configurations or multiple KLL capacities. Rollup execution and coverage tests now use the existing explicit local-rule fixtures, so they test all intended sketch rules and coverage semantics without depending on solver visitation. The rollup memory limit and result assertions remain unchanged. The former fixtures failed locally; corrected rollup (3) and coverage (8) tests pass. The integration package was exercised in batches: 330 passed, 23 pre-existing ignored; affected tests also pass strict clippy. Full remote CI passed before merge.
Merge path: this stacked PR is merged into #939’s branch, then carried to main by #939. A local merge check confirmed the combined tree exactly matches this CI-tested head.