Repository navigation
perf(data-plane): intern series at decode; bound per-series state; drop resent samples - #834
Merged
Merged
Conversation
zzylol
force-pushed
the
feat/intern-series
branch
from
October 21, 2026 00:12
072bb48 to
b6739a3
Compare
zzylol
changed the base branch from
feat/columnar-engine
to
feat/per-sample-ingestion
October 21, 2026 00:12
zzylol
force-pushed
the
feat/per-sample-ingestion
branch
from
October 21, 2026 00:21
d68efef to
41ccfbe
Compare
zzylol
force-pushed
the
feat/intern-series
branch
from
October 21, 2026 00:22
b6739a3 to
132d418
Compare
…by id Each label set is interned once, when Remote Write or a raw-sample fetch delivers it: an ahash of its label bytes finds or creates one shared Series with a dense u32 id and its metric. Samples carry the Arc<Series>; the raw-sample window's per-series cells, the per-sample group numbers and the instant selector's latest-sample map key by the id, and the metric is read from the series instead of the label map.
…e batcher Follow-ups from the per-sample ingestion review: - An open pane numbers its own groups, so it holds a summary state only for the series that took a sample in it, not one per series ever seen. - A series unseen for longer than its definition's retention plus one pane is forgotten, with its cached cells in the raw-sample window; a query-time window forgets series unseen for the plan's longest retention past its width. - A sample is looked up once, by its interned series id, and a run of samples into one pane reuses the series' group number there. - A sample not after its series' previous one in the pane (a resent Remote Write sample) is dropped and counted in samples_dropped_duplicate instead of failing the pane or being counted twice. - The output timestamp offset is an error when the build gives none, instead of defaulting to zero. - Pane is SummaryInstance, and SeriesWindow is RawSampleWindowBatcher.
zzylol
force-pushed
the
feat/intern-series
branch
from
October 21, 2026 00:27
132d418 to
244e56d
Compare
zzylol
marked this pull request as ready for review
October 21, 2026 00:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: main → this PR → no-forward mode.
Why
After #830, the remaining per-sample work outside the summary state update was label-set handling. Remote Write decoding built a new
BTreeMap<String, String>per series per request. Each raw-sample window's per-series cell cache and the per-sample group numbers then hashed and compared that whole map once per run of samples, and the metric was looked up in it for every sample. With one sample per series per request, that is per sample.What
Series, with a denseu32id and its metric (series_ids).RawSample::labelsis anArc<Series>, which derefs to the label map:SeriesWindow), the per-sample group numbers and the instant selector's latest-sample map key by the id;Review follow-ups from #830, in the second commit:
ensure_groupsgave every open pane a state for every series ever numbered). A series unseen for longer than its definition'sretention_msplus one pane is forgotten, with its cached cells. A query-timeRawSampleWindowforgets series unseen for the plan's longest retention past its width.u32id, and a run of samples into one pane reuses the group number there; the label map is not hashed per sample.samples_dropped_duplicatestatus key, as Prometheus drops it. Before, a counter's state refused it and the whole pane failed; any other summary state counted it twice.calibratereturns an error (the pane fails and the next pane retries) instead of defaulting the output timestamp offset to 0.Pane→SummaryInstance,SeriesWindow→RawSampleWindowBatcher(raw_sample_window.rs).The SDS is unchanged: its panes are the plan's output batches, whose series identity column is dictionary-encoded text in the plan's schema.
How
series_ids::intern_pairssorts the Remote Write pairs by name only when they are not already sorted. It hashes each name and value with its length, so a byte moving across a label boundary changes the hash. It looks the hash up under a read lock and compares labels only against that hash's entries; only a new label set takes the write lock. The interner is process-wide and never forgets a series, so its memory grows with every distinct series ever seen (a follow-up if series churn matters).Usage
No new flags.
/api/v1/status/asapgainsprecompute.samples_dropped_duplicate.Before this PR
Per series per request: a new label map. Per run of samples in each per-series cache: a SipHash of the map plus a map comparison. Per sample: a
__name__lookup.After this PR
Per series per request: one ahash over the label bytes and one comparison with the interned set. Everything downstream uses the
u32id.Verification
cargo test --workspace --lockedpasses (fmt and clippy not run for this draft; the benefit harness was not rerun). Tests that buildRawSamples now intern their label maps (series_ids::intern). Newseries_idstests, which fail without an interner:Also new:
per_sample::a_resent_sample_is_dropped_not_failing_or_doubling_the_pane: Example 1 with one sample resent. One duplicate is counted, no pane fails, and every pane equals the plan's build without the resent sample.precompute::tests::per_series_state_follows_the_active_series: Example 4b, 10 series for 25 min, then one series to 200 min. The open pane holds one state, and nine series are forgotten.The parity test of #830 (
per_sample_panes_equal_the_batch_build) and thequery_time_referencecomparisons still pass.