Repository navigation
refactor: route precompute outputs on StoredOutputId, not a bare u64 - #787
Merged
Merged
Conversation
The runtime's materialization index was keyed by a bare `u64` whose meaning lived only in the field name, `materializations_by_policy_fingerprint`. That name points at PolicyFingerprint, which its own module doc calls a "Legacy routing wrapper", superseded by an explicit `stored_output_id`. Every caller had to know which of the two identities that integer was. Key the index on StoredOutputId, the type whose doc says "Runtime routing uses this identity, never the semantic definition hash", and rename the field to `materializations_by_output`. `runtime_materializations`, the three accessors and the `Index` impl follow. No serialized bytes change. InstalledPrecomputePlan is `#[derive(Debug, Clone)]` with no Serialize at all, its own doc calling it an internal execution index derived from a validated plan. StoredOutputId and PolicyFingerprint are the same u64 behind two `#[serde(transparent)]` newtypes with conversions both ways, so even `state_schema_id`, which does reach a published document, formats the identical string. Fixtures keep holding raw ids. `from_raw_ids` is the one place that says so, rather than each test wrapping literals, and `From<u64> for StoredOutputId` lets a value already established as an output id convert without ceremony. This is step 1 of #785. It does not retire PolicyFingerprint: `from_config` still derives one from an explicit `stored_output_id`, so adoption changes how a fingerprint is computed rather than removing it. Retirement needs `stored_output_id` to become required, which is a separate change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
…frontend only (#843) fix(deps): pin ASAPPlanner 7565c47e; depend on asap-planner's PromQL frontend only Pin both Planner crates to ASAPPlanner 7565c47e, which brings: - #780: a gauge declared non-negative (SourceFacts::values_non_negative) lets a per-series quantile sketch's relative error carry through sum and avg, and a quantile estimate carries its samples' sign; - #782: the SQL and MetricsQL frontends are optional features. asap-planner is now used with default-features = false, features = ["promql"], so DataFusion and the SQL and MetricsQL frontends drop out of the build (Cargo.lock: -1,847 lines). The sync script accepts a pin followed by other keys, such as features. Not yet: ASAPPlanner #787 (StoredOutput.coverage and inputs replace pane); that adaptation comes next.
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
…selection Pin ASAPPlanner 4111e620 (#787: StoredOutput gives its SummaryCoverage, definition id and selection, and its inputs instead of a pane). - A definition's identity is the plan's coverage.definition_id; the SDS's own sub-DAG hash is gone (and with it the sha2 dependency). - The definition id leaves the selection out, so outputs of one id with other label constraints are other SummaryDefinitions (DefinitionKey: id and label constraints), and a summary instance is keyed by the id and its concrete Selection: the label constraints and the pane end the plan's relative time binds to. - A definition's pane width and each output's offset come from its selection's relative time; its RawSampleWindow is the stored output's raw-sample input; its stored inputs are the plan's inputs.stored_outputs. A selection that is not one left-open time box, or several raw-sample windows, is refused.
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
…coverage, ignore_accuracy_guarantees, pane-width SDS keys, --stream-start-ms (#844) * feat(sds): adopt ASAPPlanner #787: the plan's definition id and full selection Pin ASAPPlanner 4111e620 (#787: StoredOutput gives its SummaryCoverage, definition id and selection, and its inputs instead of a pane). - A definition's identity is the plan's coverage.definition_id; the SDS's own sub-DAG hash is gone (and with it the sha2 dependency). - The definition id leaves the selection out, so outputs of one id with other label constraints are other SummaryDefinitions (DefinitionKey: id and label constraints), and a summary instance is keyed by the id and its concrete Selection: the label constraints and the pane end the plan's relative time binds to. - A definition's pane width and each output's offset come from its selection's relative time; its RawSampleWindow is the stored output's raw-sample input; its stored inputs are the plan's inputs.stored_outputs. A selection that is not one left-open time box, or several raw-sample windows, is refused. * feat(deps): pin ASAPPlanner main ef52e175; plan with ignore_accuracy_guarantees - Both Planner crates move to ef52e175 (sketchlib planner-pin/0.4.0-plus: interpolated and signed DDSketch quantiles, top-k with series labels, Stage 3's executor compile check, joint search, shared stored summaries). - control_plane: --plan-selection-settings (the Planner's PlanSelectionSettings JSON) and --ignore-accuracy-guarantees; prints the queries accepted without a guarantee. --measured-operation-costs reads MeasuredOperationCosts. - serde_json float_roundtrip: a plan file reloads to the plan it was written from (its predicted usage is floats). - data_plane/tests/planner_executor_answers.rs: the data plane answers as the Planner's executor for DDSketch panes over zero and signed samples, interpolated quantiles, top-k labels, counter top-k without guarantees, and jointly planned shared states. * feat(deps): pin ASAPPlanner main 05fb36e4 (#808: faster installable-plan compile check) * fix(sds): panes of one definition id and two widths are separate definitions A jointly planned workload can store one computation (one definition id, the same label constraints) in panes of two widths, e.g. per-series sums in 1-second panes for `sum_over_time(data[1m])` every second and in 30-second panes for `sum_over_time(data[20m])` every 30 s. The SDS keyed definitions by (id, label constraints), so both shared one definition with the first width: the 20-minute query merged forty 1-second panes and answered 1/30 of its sum. DefinitionKey and Selection now carry the pane width: a definition is (id, label constraints, pane width), and a summary instance's concrete selection is (label constraints, width, end), the interval (end - width, end]. * feat(precompute): --stream-start-ms builds the first pane that starts at the stream's start The first pane each stored output sees is skipped, since it may have started before the data plane did. When the producers start their stream at the data plane's start (the benchmark feeds from an aligned start S), that pane is complete, and a query whose window reaches back to S was refused: e.g. `rate(data_total[20m])` and `sum_over_time(data[20m])` at S + 20 min. A producer can now declare where its stream starts (PrecomputeOptions:: stream_start_ms, `--stream-start-ms`): a first pane starting at or after it is built; one starting before it is still skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of #785.
The runtime's materialization index was keyed by a bare
u64:The integer's meaning lived only in the field name, and that name points at
PolicyFingerprint, which its own module doc calls a "Legacy routing wrapper" superseded by an explicitstored_output_id. Every caller had to know which of the two identities the integer was.It is now keyed on
StoredOutputId, whose doc says "Runtime routing uses this identity, never the semantic definition hash", and the field ismaterializations_by_output.runtime_materializations, the three accessors and theIndeximpl follow.No serialized bytes change
Three things were verified rather than assumed:
InstalledPrecomputePlanis#[derive(Debug, Clone)]with noSerialize. Its doc calls it an internal execution index derived from a validated plan, and it is rebuilt byfrom_precompute_planon decode.PolicyFingerprintappears as a serde field on no struct. Both it andStoredOutputIdare#[serde(transparent)]newtypes over the sameu64, with conversions in both directions.state_schema_id, which formats"{BACKEND_COMPAT}:summary-state:v1:{id}". Same number, identical string.So no schema bump, which is a correction to what #785 originally claimed.
Fixtures
Tests keep holding raw ids. A
#[cfg(test)] from_raw_idssays that once, at the boundary, instead of every fixture wrapping its literals, andFrom<u64> for StoredOutputIdlets a value already established as an output id convert without ceremony. Production construction still goes throughfrom_precompute_plan, which takes the ids from the plan.What this does not do
It does not retire
PolicyFingerprint.from_configstill derives one from an explicitstored_output_id:So adoption changes how a fingerprint is computed, not whether one exists. Retirement needs
stored_output_idto become required so that fallback has no callers; that is a separate change.Validation
19 files, +156 / −111.
cargo test -p data_plane --lib946 passed,-p control_plane --lib433 passed,cargo fmt --checkandcargo clippy --all-targetswith-D warningsclean.Rebased onto
mainafter #763 merged, so it targetsInstalledPrecomputePlanrather than theStreamingConfigthat #763 replaced.🤖 Generated with Claude Code