Repository navigation
fix(executor): a DDSketch takes zero and negative samples - #802
Merged
Merged
Conversation
The DDSketch updater refused every sample <= 0, so a DDSketch build over a gauge that ever reports exactly 0 (or a negative value) failed at run time: #795's P5-m1 over a gauge declared non-negative, and every DDSketch selected under ignore_accuracy_guarantees. asap_sketchlib's DdSketch keeps a zero bucket and a negative store, so the updater now accepts zero and any finite sample whose magnitude is within ddsketch_indexable_bounds, positive or negative, and adds it with try_add. NaN, +-Inf, magnitudes above the range, and nonzero magnitudes below it (which the zero bucket would take without the relative error) are still refused. The DDSketch guarantee (relative error alpha per quantile) does not assume positive values; whether it carries through sum and avg still depends on the samples' declared sign, unchanged. Tests: the updater's domain and its quantiles over 0, 0, 3, 5 and over signed samples; through plan_installable and a stand-in runtime, a DDSketch-pane plan over a gauge reporting zero (avg of p99s, declared non-negative) and over signed samples including all-negative series (max of p99s), each equal to query_time_reference and within 0.05 of the exact interpolated values. Two executor tests that locked the old positive-only domain now check the new one.
zzylol
force-pushed
the
fix/ddsketch-zero-negative
branch
from
October 21, 2026 06:05
5a22466 to
cb0c6bf
Compare
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
zzylol
added a commit
that referenced
this pull request
Oct 21, 2026
…n sketchlib planner-pin/0.4.0-plus (#803) * chore: pin asap_sketchlib to planner-pin/0.4.0-plus (63e5655) Release 0.4.0 (eb04e7f) plus the open sketchlib PRs #245 (check_weight, try_update_weighted, rescore_heap), #246 (KLL lazy-buffer follow-ups), #247 (Count Sketch stack medians) and #248 (DdSketch::quantile_interpolated). Moves to a release once they merge. * fix(executor): DDSketch quantiles interpolate as PromQL and percentile_cont do The DDSketch kernel read sketchlib's nearest rank ceil(q n). PromQL quantile_over_time and SQL percentile_cont interpolate linearly between the order statistics at ranks floor(q (n - 1)) and ceil(q (n - 1)), so answers missed the exact ones by more than alpha: [20, 40] at q = 0.9 read 40 against 38. The kernel now reads DdSketch::quantile_interpolated (sketchlib #248). An empty sketch still gives no answer, and a q outside [0, 1] (NaN included) is still an error; the Planner never sends one, since Pass 1 prunes the sketch candidate (QuantileOutsideUnitInterval) and the exact plan answers +-Inf. The Planner's DDSketch rule (ddsketch_relative_error_alpha_v1) states alpha relative error on the value read and assumes no rank convention; the interpolated read is within alpha when both neighbours share a sign, which the sign rule already requires before the bound composes. The bound is unchanged. The two integration tests that rebuilt interpolation from nearest-rank reads now call quantile_interpolated. * fix(executor): top-k heaps keep the old kernel's membership sketchlib's heap rescores only the key an update touches, so a resident that shares a cell with it keeps a stale count, and a new key could be admitted against that stale minimum and evict a resident whose current estimate is higher. The old kernel rescored every candidate after every update. Each heap update now writes the counters, calls rescore_heap (sketchlib #245), and only then offers the updated key, so admission compares against current estimates; O(heap_size) per update, as before. rows() already reads every retained key at its current estimate, and a merge re-estimates both heaps' keys, so the readout needs no extra pass. A reference-model test (written first, failing at update 15 without the rescore) replays colliding Count-Min and Count Sketch streams and checks after every update that the retained keys match rescoring every candidate, up to exact ties. Weight checks now use sketchlib's check_weight and try_update_weighted: the counters stay finite per probed cell, as in the old kernel, instead of the Planner-side bound on the summed weight magnitude, so a merge checks the merged counters instead. * test(executor): #802's signed-sample quantiles expect PromQL's interpolated values * style(executor): rustfmt the DDSketch quantile tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From the #799 review.
Why
The executor's DDSketch updater refused every sample
<= 0(factory.rs,value > 0.0). A DDSketch build over a gauge that ever reports exactly 0, or a negative value, therefore failed at run time with "DDS maintenance input is outside its positive representable domain". That covers #795's P5-m1 over a gauge declared non-negative, and every DDSketch selected underignore_accuracy_guarantees.asap_sketchlib'sDdSketch(sketchlib #141) has a zero bucket and a negative store, so the restriction was the executor's, not the sketch's.What
DDSketchAccumulatorUpdater::validate_single_inputaccepts zero and any finite sample whose magnitude is withinddsketch_indexable_bounds(alpha), positive or negative.update_singleadds the sample withDdSketch::try_addand returns its error instead of dropping it.alphaper quantile holds for signed samples (negative values are binned by magnitude) and is exact at zero. Whether it carries throughsumandavgstill depends on the samples' declared sign (feat: declare non-negative gauges so per-series quantile sketches compose through avg (#772) #780, fix(logical): a quantile of merged panes keeps its samples' sign, so pane DDSketches compose through avg #795).How
One validation rule in the updater, so every path that updates a DDSketch takes it: the kernel factory,
StateColumnUpdater, and precompute.Before this PR
avg by (region, service, job) (quantile_over_time(0.99, data[100s]))over a non-negative gauge whose samples include 0 planned P3-m1 (merged DDSketch panes), but building a pane failed at the first zero sample.After this PR
The same plan builds and answers. Answers equal
query_time_referenceand are within 0.05 of the exact average of PromQL's interpolated p99s. Signed samples, including all-negative series, work too.Verification
crates/executor/src/summary_kernels/factory.rs:accepts_zero_and_signed_samples_within_its_indexable_rangequantiles_over_zero_and_signed_samples: over 0, 0, 3, 5 and over −7, −3, 0, 4, each quantile is within alpha of the sample at rank ⌈q·n⌉.crates/planner/tests/runtime_install.rs, throughplan_installableand the stand-in runtime:a_ddsketch_over_a_gauge_reporting_zero_runs_within_alpha:avgof per-series p99s with each window exactly 0, 0, 3, 5 scaled by the series, and again with 1,100 samples a third of which are zero.a_ddsketch_over_signed_samples_runs_within_alpha:maxof per-series p99s; four series cross zero and four are all negative.query_time_referenceand are within 0.05 of exact.precompute_rejects_nonfinite_and_out_of_range_dds_updates(renamed) andper_sample_updates_keep_each_kernels_input_contract.cargo test --workspace --locked: all tests pass.