Repository navigation
Compress pane candidate search and progressively expand legal widths - #936
Merged
zzylol merged 13 commits intoOct 25, 2026
Merged
Conversation
zzylol
marked this pull request as ready for review
October 25, 2026 05:47
zzylol
added a commit
that referenced
this pull request
Oct 25, 2026
* refactor: share execution demand accounting with joint candidate planning * refactor: evaluate summary materialization for selected readers * feat: select physical configurations with validated cost refinement * fix: charge request overhead for literal queries and CPU capacity * feat: report candidate model, solve, and validation metrics * feat: expose candidate-connection planning with scoped search reports * refactor: move shared candidate evaluation out of legacy search * fix: exclude unavailable query-time raw scans before candidate search * feat: reuse local pane rules in candidate planning and report generation time * fix: assign visible IDs to validated joint plans * fix: preserve ranked-row contracts when lowering fused and top-k candidates * feat: share concrete pane states and seed joint search with validated contracts * docs: record capability parity gates and ASAPQuery workload measurements * feat: share pane states across compatible scan-label projections * feat: migrate certified query accuracy pruning to candidate search * feat: add certified ingestion cost bounds to candidate selection * docs: measure migrated sharing pruning and ingestion bounds * docs: require scope-based module organization at planner cutover * fix: retain feasible solver proposals without overstating cost bounds * docs: identify mixed responsibilities for final module cleanup * feat: expand shared state capacities and pane horizons in logical memo * feat: restore shared non-summary query materialization * feat: reuse conservative per-query pruning before memo expansion * feat: complete joint candidate sharing and objective selection * feat: replace legacy planning with bounded candidate-connection search * fix: reject unexecutable raw window candidates before pane expansion * fix: preserve physical contracts and feasible workload plans * perf: refine seed materialization with bounded MILP proposals * test: enforce raw-source seed legality under both accuracy policies * fix(planner): bound compilation and validate pane edges linearly * refactor(planner): remove retired Cartesian workload enumeration * test(planner): verify explicit rules independently of search archives * fix(planner): preserve small-cost distinctions in MILP scaling * test: migrate planner examples to explicit rule fixtures * fix(cost): reject overflowing physical state configurations * Make disjoint pane coverage union scale with sorted intervals * test(planner): validate sharing rewrites and selected pane coverage * fix(accuracy): gate interpolated DDS quantiles on source domain * test(planner): make signed DDS and compile rejection fixtures explicit * test(planner): compare facade plans independently of search timings * Honor planning deadlines while flattening and exporting physical DAGs * fix(planner): retain budget reports without a feasible incumbent * fix(planner): seed composed DDS budgets before query validation * test: pin valid DDS and pane fixtures independently of plan ranking * docs: record validated joint planner workload results and limits * docs(planner): describe the active candidate connection search * test(planner): make composed summary rejection fixtures deterministic * fix(viewer): display current candidate-connection search reports * docs: record complete regression and viewer coverage * Compress pane candidate search and progressively expand legal widths (#936) * Expand pane width search progressively within the global budget * Share compatible supplier storage and index configuration connections * perf(planner): memoize exact supplier compatibility interfaces * Skip unchanged pane domains and preserve identities across search rounds * Preserve independent terminal reader bindings for repeated outputs * perf(planner): factor equivalent terminal readers with independent bindings * Name validated plans once by their pane search phase * Document pane search compression contracts and domain limits * Preserve the last complete inventory snapshot in timeout reports * test(planner): benchmark pane search compression and workload quality * test(planner): separate fixed-catalog growth from bounded search outcomes * Avoid pane search restarts for window-free workloads and retain diagnostics * docs(planner): record window-free sharing regression and final validation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merged through #934 into main at 7bbade2. The main commit has the identical Git tree as CI-validated d379c4e. Retains the single CandidateConnections production engine.
Before this PR
Dense pane workloads repeatedly compute and store compatible supplier lists. Complete width generation can consume the planning budget before useful larger models are solved. Equivalent terminal readers duplicate candidate bindings.
After this PR
Memoize exact compatibility, share identical supplier-list storage, and factor contract-equivalent terminal unary readers while preserving independent output bindings. Progressively expose legal widths under one budget and retain the best validated plan. Queries without pane-eligible windows go directly to one complete-domain search. Incomplete domains never claim a global optimum or bound.
Verified effect on the final executable
Three sequential before/after repetitions per workload, alternating order, with the same 30-second budget and input fingerprints. These are development-build measurements of the whole process. All 24 performance runs pass accuracy-contract, coverage, finite-cost, and certificate checks; every selected modeled cost equals its baseline.
These are bounded-search results across different reached domains, not a complete-domain compression proof. One case solves a larger intermediate model while using less memory. Supplier-list storage sharing alone does not remove MILP connection decisions. Three repetitions show observed variability, not statistical significance; execution cost remains a model estimate.
The same final executable passes 30/30 ASAPQuery workloads (270 query requirements) and 9/9 viewer workloads (40 requirements). All 30 ASAPQuery modeled costs equal the archived baseline; all nine viewer costs equal their earlier accepted runs. Final SQL regression validation restores one shared KLL at exactly baseline cost, proving optimality in 58 evaluations.
Final executable source: fe171f6; SHA-256: a3234d62d2a6717e3772aba240959f6a224a4e1e031fe9cba27854079f8622f2. Repeated samples, input hashes and complete plan artifacts are under /mydata/pane-compression-validation/final-effect/. The repository evaluation report documents reproduction and earlier measurements.
Key code interfaces added / modified
Major algorithm changes
Intern exact supplier sets; factor only equivalent terminal unary readers; expand nested legal width domains under a shared budget. Preserve independent bindings, full-domain certificate semantics and cross-round plan/cost/accuracy identities. Internal readers and pane merges are not factored. Output-assignment options can still grow quadratically; the fixed-catalog test proves affine producer-connection growth, not a general polynomial search bound.
Validation
No new sketch/window algorithm or general cost-based DP is introduced.
Rollup-heavy follow-up verification
Three sequential before/after repetitions of
memo-rollupheavy-t11-service-endpoint-5m, same 30-second budget and final executable:All six runs accept both queries with enforced accuracy. All selected DAGs share one Scan node but use two separate exact COUNT(DISTINCT) aggregates (service versus service/endpoint); none selects a cross-group rollup. Thus this measures reduced planning overhead on a rollup-heavy workload, not reduced execution cost from rollup. Pending widths remain, so different reached domains and lack of a global optimum certificate still apply. Artifacts: /mydata/pane-compression-validation/rollup-effect/.