Skip to content

refactor: route precompute outputs on StoredOutputId, not a bare u64 - #787

Merged
zzylol merged 1 commit into
mainfrom
refactor/route-on-stored-output-id
Sep 29, 2026
Merged

zzylol merged 1 commit into
mainfrom
refactor/route-on-stored-output-id

Conversation

@zzylol

@zzylol zzylol commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Step 1 of #785.

The runtime's materialization index was keyed by a bare u64:

pub(crate) materializations_by_policy_fingerprint: HashMap<u64, PrecomputeMaterialization>,

The integer's meaning lived only in the field name, and that name points at PolicyFingerprint, which its own module doc calls a "Legacy routing wrapper" superseded by an explicit stored_output_id. Every caller had to know which of the two identities the integer was.

It is now keyed on StoredOutputId, whose doc says "Runtime routing uses this identity, never the semantic definition hash", and the field is materializations_by_output. runtime_materializations, the three accessors and the Index impl follow.

No serialized bytes change

Three things were verified rather than assumed:

  • InstalledPrecomputePlan is #[derive(Debug, Clone)] with no Serialize. Its doc calls it an internal execution index derived from a validated plan, and it is rebuilt by from_precompute_plan on decode.
  • PolicyFingerprint appears as a serde field on no struct. Both it and StoredOutputId are #[serde(transparent)] newtypes over the same u64, with conversions in both directions.
  • The one place the value reaches a published document is state_schema_id, which formats "{BACKEND_COMPAT}:summary-state:v1:{id}". Same number, identical string.

So no schema bump, which is a correction to what #785 originally claimed.

Fixtures

Tests keep holding raw ids. A #[cfg(test)] from_raw_ids says that once, at the boundary, instead of every fixture wrapping its literals, and From<u64> for StoredOutputId lets a value already established as an output id convert without ceremony. Production construction still goes through from_precompute_plan, which takes the ids from the plan.

What this does not do

It does not retire PolicyFingerprint. from_config still derives one from an explicit stored_output_id:

if let Some(output) = cfg.stored_output_id {
    return output.fingerprint();
}

So adoption changes how a fingerprint is computed, not whether one exists. Retirement needs stored_output_id to become required so that fallback has no callers; that is a separate change.

Validation

19 files, +156 / −111. cargo test -p data_plane --lib 946 passed, -p control_plane --lib 433 passed, cargo fmt --check and cargo clippy --all-targets with -D warnings clean.

Rebased onto main after #763 merged, so it targets InstalledPrecomputePlan rather than the StreamingConfig that #763 replaced.

🤖 Generated with Claude Code

The runtime's materialization index was keyed by a bare `u64` whose meaning
lived only in the field name, `materializations_by_policy_fingerprint`. That
name points at PolicyFingerprint, which its own module doc calls a "Legacy
routing wrapper", superseded by an explicit `stored_output_id`. Every caller
had to know which of the two identities that integer was.

Key the index on StoredOutputId, the type whose doc says "Runtime routing uses
this identity, never the semantic definition hash", and rename the field to
`materializations_by_output`. `runtime_materializations`, the three accessors
and the `Index` impl follow.

No serialized bytes change. InstalledPrecomputePlan is `#[derive(Debug, Clone)]`
with no Serialize at all, its own doc calling it an internal execution index
derived from a validated plan. StoredOutputId and PolicyFingerprint are the
same u64 behind two `#[serde(transparent)]` newtypes with conversions both
ways, so even `state_schema_id`, which does reach a published document, formats
the identical string.

Fixtures keep holding raw ids. `from_raw_ids` is the one place that says so,
rather than each test wrapping literals, and `From<u64> for StoredOutputId`
lets a value already established as an output id convert without ceremony.

This is step 1 of #785. It does not retire PolicyFingerprint: `from_config`
still derives one from an explicit `stored_output_id`, so adoption changes how
a fingerprint is computed rather than removing it. Retirement needs
`stored_output_id` to become required, which is a separate change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@zzylol
zzylol merged commit e0bf5e1 into main Sep 29, 2026
1 check passed
@zzylol
zzylol deleted the refactor/route-on-stored-output-id branch September 29, 2026 18:39
zzylol added a commit that referenced this pull request Oct 21, 2026
…frontend only (#843)

fix(deps): pin ASAPPlanner 7565c47e; depend on asap-planner's PromQL frontend only

Pin both Planner crates to ASAPPlanner 7565c47e, which brings:
- #780: a gauge declared non-negative (SourceFacts::values_non_negative)
  lets a per-series quantile sketch's relative error carry through sum and
  avg, and a quantile estimate carries its samples' sign;
- #782: the SQL and MetricsQL frontends are optional features.

asap-planner is now used with default-features = false, features =
["promql"], so DataFusion and the SQL and MetricsQL frontends drop out of
the build (Cargo.lock: -1,847 lines). The sync script accepts a pin
followed by other keys, such as features.

Not yet: ASAPPlanner #787 (StoredOutput.coverage and inputs replace pane);
that adaptation comes next.
zzylol added a commit that referenced this pull request Oct 21, 2026
…selection

Pin ASAPPlanner 4111e620 (#787: StoredOutput gives its SummaryCoverage,
definition id and selection, and its inputs instead of a pane).

- A definition's identity is the plan's coverage.definition_id; the SDS's
  own sub-DAG hash is gone (and with it the sha2 dependency).
- The definition id leaves the selection out, so outputs of one id with
  other label constraints are other SummaryDefinitions (DefinitionKey: id
  and label constraints), and a summary instance is keyed by the id and its
  concrete Selection: the label constraints and the pane end the plan's
  relative time binds to.
- A definition's pane width and each output's offset come from its
  selection's relative time; its RawSampleWindow is the stored output's
  raw-sample input; its stored inputs are the plan's
  inputs.stored_outputs. A selection that is not one left-open time box, or
  several raw-sample windows, is refused.
zzylol added a commit that referenced this pull request Oct 21, 2026
…coverage, ignore_accuracy_guarantees, pane-width SDS keys, --stream-start-ms (#844)

* feat(sds): adopt ASAPPlanner #787: the plan's definition id and full selection

Pin ASAPPlanner 4111e620 (#787: StoredOutput gives its SummaryCoverage,
definition id and selection, and its inputs instead of a pane).

- A definition's identity is the plan's coverage.definition_id; the SDS's
  own sub-DAG hash is gone (and with it the sha2 dependency).
- The definition id leaves the selection out, so outputs of one id with
  other label constraints are other SummaryDefinitions (DefinitionKey: id
  and label constraints), and a summary instance is keyed by the id and its
  concrete Selection: the label constraints and the pane end the plan's
  relative time binds to.
- A definition's pane width and each output's offset come from its
  selection's relative time; its RawSampleWindow is the stored output's
  raw-sample input; its stored inputs are the plan's
  inputs.stored_outputs. A selection that is not one left-open time box, or
  several raw-sample windows, is refused.

* feat(deps): pin ASAPPlanner main ef52e175; plan with ignore_accuracy_guarantees

- Both Planner crates move to ef52e175 (sketchlib planner-pin/0.4.0-plus:
  interpolated and signed DDSketch quantiles, top-k with series labels,
  Stage 3's executor compile check, joint search, shared stored summaries).
- control_plane: --plan-selection-settings (the Planner's
  PlanSelectionSettings JSON) and --ignore-accuracy-guarantees; prints the
  queries accepted without a guarantee. --measured-operation-costs reads
  MeasuredOperationCosts.
- serde_json float_roundtrip: a plan file reloads to the plan it was written
  from (its predicted usage is floats).
- data_plane/tests/planner_executor_answers.rs: the data plane answers as
  the Planner's executor for DDSketch panes over zero and signed samples,
  interpolated quantiles, top-k labels, counter top-k without guarantees,
  and jointly planned shared states.

* feat(deps): pin ASAPPlanner main 05fb36e4 (#808: faster installable-plan compile check)

* fix(sds): panes of one definition id and two widths are separate definitions

A jointly planned workload can store one computation (one definition id,
the same label constraints) in panes of two widths, e.g. per-series sums in
1-second panes for `sum_over_time(data[1m])` every second and in 30-second
panes for `sum_over_time(data[20m])` every 30 s. The SDS keyed definitions
by (id, label constraints), so both shared one definition with the first
width: the 20-minute query merged forty 1-second panes and answered 1/30 of
its sum.

DefinitionKey and Selection now carry the pane width: a definition is (id,
label constraints, pane width), and a summary instance's concrete selection
is (label constraints, width, end), the interval (end - width, end].

* feat(precompute): --stream-start-ms builds the first pane that starts at the stream's start

The first pane each stored output sees is skipped, since it may have started
before the data plane did. When the producers start their stream at the
data plane's start (the benchmark feeds from an aligned start S), that pane
is complete, and a query whose window reaches back to S was refused: e.g.
`rate(data_total[20m])` and `sum_over_time(data[20m])` at S + 20 min.

A producer can now declare where its stream starts (PrecomputeOptions::
stream_start_ms, `--stream-start-ms`): a first pane starting at or after it
is built; one starting before it is still skipped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant