Skip to content

feat: declare non-negative gauges so per-series quantile sketches compose through avg (#772) - #780

Merged
zzylol merged 4 commits into
mainfrom
feat/non-negative-gauge-declaration
Oct 21, 2026
Merged

zzylol merged 4 commits into
mainfrom
feat/non-negative-gauge-declaration

Conversation

@zzylol

@zzylol zzylol commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Why

avg by (region) (quantile_over_time(0.99, data[100s])) over a gauge that is never negative (a latency, say) can only be planned exactly. A per-series DDSketch quantile is within a relative error, but that bound carries through avg and sum only over non-negative values. That sign, values_non_negative[data], was never bound for a quantile estimate, and a data workload could not declare a gauge non-negative. So Stage 3 rejected every sketch candidate.

What

  • SourceFacts can declare every sample of a metric non-negative (values_non_negative), for a gauge as well as a counter.
  • A quantile estimate over a metric's raw samples carries that metric's sign. With the declaration, the per-series DDSketch candidate's relative bound composes through avg and sum and the candidate is valid. Without it, the candidate is still rejected. Through max it was already valid, since max needs no sign.
  • Follow-ups from the review of feat: rate, increase and irate of a declared counter are non-negative Count-Min weights (#773) #777 (increase and rate over a counter are non-negative weights for Count-Min top-k #773), each with a test written first:
    • offset. An offset keeps a metric's sample sign. topk(10, sum by (job) (rate(data_total[5m] offset 1h))) over a declared counter keeps its non-negative Count-Min weights. The gauge sign above also holds under offset.
    • irate. topk(k, irate(m[w])) is a heavy-hitter ranking, as topk(k, rate(m[w])) is. It used to be an exact sort and limit.
  • Follow-up from this PR's own review: a quantile whose φ is outside [0, 1] gets no sketch candidate and never carries its samples' sign. PromQL answers it with -Inf or +Inf, and the executor's DDSketch and KLL refuse it. Before, avg by (region) (quantile_over_time(-1, x[100s])) over a declared non-negative gauge selected a DDSketch, and so did bare quantile_over_time(2, x[100s]) and max by (...) (quantile_over_time(-1, ...)) even without a declaration. All of them now select the exact plan.

How

  • The declaration. Plan selection binds the sign statistic values_non_negative[m] to proven (1) when m is declared a counter or declared non-negative (declared_non_negative). Before, only a declared counter counted.
  • Quantiles carry the sign. In Stage 1, estimates_samples_of, which feeds only the sign statistic of a fold, also follows a quantile estimate whose update reads the sample value of raw samples. A quantile of some values is one of them, or lies between two of them, so it has their sign. The existing relative_fold rule then gates the bound on that metric's sign instead of an anonymous unknown.
  • offset. samples_of_metric passes through a TimeShift that carries only an offset. An @ anchor still ends it, as the review asked.
  • irate. The frontend's direct per-series ranking check and the counter-ranking check now include IRate.
  • φ outside [0, 1]. Pass 1 prunes such a quantile's sketch candidates with the new PruneReason::QuantileOutsideUnitInterval, and estimates_samples_of follows a quantile estimate only for φ in [0, 1].

Key code interfaces added / modified

crates/types/src/workload/mod.rs

/// What is known about one source. The type stays; one field is added.
pub struct SourceFacts {
    /// Series in the source (default: unknown).
    pub series_count: Option<u64>, // unchanged
    /// Declared Prometheus type (default: unknown).
    pub metric_type: Option<MetricType>, // unchanged
    /// Whether every sample of the metric is non-negative; default `false`
    /// (unknown), omitted from JSON when false. Declares it for a gauge; a
    /// counter always is. The sign a relative bound needs through sum/avg.
    pub values_non_negative: bool, // new
    /// How often each series is sampled (default: unknown).
    pub sample_interval: Option<DurationMs>, // unchanged
    /// Bounds on the source's stream norms (default: unknown).
    pub stream_norms: Option<StreamNorms>, // unchanged
    /// Bound on distinct items per evaluation (default: unknown).
    pub max_distinct_per_evaluation: Option<u64>, // unchanged
}

JSON: "sources": {"data": {"metric_type": "gauge", "values_non_negative": true}}.

Usage (the test a_non_negative_gauge_carries_a_ddsketch_relative_bound_through_avg):

// The user's query, within relative error 0.01 at δ = 0.01.
let target = AccuracyTarget::max_error(0.01).failure_probability(0.01)
    .accuracy_metric(AccuracyMetric::Relative);
// Stages 1-2: the exact plan, a per-series KLL (P2) and a per-series DDSketch
// (P3). Stage 1 leaves P3's bound gated on values_non_negative[x].
let candidates = all_candidates("avg by (region) (quantile_over_time(0.99, x[100s]))", &target);
// The data workload: x is a gauge whose samples are declared non-negative.
let mut data = with_metric_types(data(), &[("x", MetricType::Gauge)]);
data.sources.get_mut("x").unwrap().values_non_negative = true;
// Stage 3 binds values_non_negative[x] = 1, so P3's bound is relative 0.01.
let selection = select_plan(&candidates, &[entry_with(target, repeating(10_000))], &data,
                            PlanningModels::builtin())?;
// P3 is valid: selection.accepted["P3"] = [metric relative; accepted relative (DDSketch)].
// With values_non_negative = false, P3 is rejected with AccuracyTargetMissed
// ("the output guarantees a Relative bound None").

Before this PR

  • avg|sum by (region) (quantile_over_time(0.99, x[100s])) over a gauge: the DDSketch candidate was rejected (AccuracyTargetMissed, relative bound unknown), and the gauge could not be declared non-negative.
  • topk(10, sum by (job) (rate(data_total[5m] offset 1h))) over a declared counter: the Count-Min-with-heap candidate was rejected with CountMinWeightsNotProvenNonNegative.
  • topk(3, irate(m[5m])) lowered to Limit(Sort(...)).

After this PR

  • With values_non_negative: true, the DDSketch candidate is valid with a relative-error guarantee through avg and sum, also under offset. Without the declaration it is rejected as before.
  • The offset rate top-k over a counter passes the weight check, and is still rejected over a gauge.
  • topk(3, irate(m[5m])) lowers to a TopK aggregate and plans.

Verification

Regression tests. Each new test, or new case, failed before its fix and passes now.

  • plan-selection a_non_negative_gauge_carries_a_ddsketch_relative_bound_through_avg covers avg, sum and max, plus avg with offset 1h, each with and without the declaration.
  • plan-selection count_min_over_a_counter_derivative_needs_a_declared_counter adds the rate(... offset 1h) case.
  • frontend-promql topk_over_a_per_series_counter_derivative_is_a_heavy_hitter_ranking covers rate, increase and irate.
  • plan-selection a_direct_irate_ranking_plans checks only that topk(10, rate|irate(...)) composes and selects a plan. The heavy-hitter TopK intent is checked by the frontend test above.
  • plan-selection a_quantile_outside_the_unit_interval_is_answered_exactly covers the four probes for φ = −1 and 2 (avg by, a bare quantile, and max by). It checks that no sketch candidate is offered and that the exact plan is selected over a gauge declared non-negative. On the previous head it failed: a sketch candidate was offered.
  • logical-asap-optimization a_quantile_estimate_shares_its_samples_sign_only_inside_the_unit_interval checks the sign rule for φ = −1 and 2. It fails with the φ check removed.
  • types: the SourceFacts JSON round trip includes values_non_negative.

Commands run. Each was run with CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0, which only reduces disk use. All passed.

  • cargo test --workspace --locked at the head: exit 0, 150 test binaries, 0 failures.
  • cargo fmt --all -- --check
  • cargo clippy -p asap-types -p asap-plan-selection -p asap-logical-asap-optimization -p asap-frontend-promql --all-targets --all-features --locked -- -D warnings (re-run for the last commit on asap-plan-selection and asap-logical-asap-optimization, the crates it touches)
  • cargo check -p asap-planner --tests --locked

Not applicable: performance measurements.

Limitations.

  • values_non_negative feeds only the sign statistic. Count-Min weight confirmation still requires a declared counter, so a Count-Min heap over a non-negative gauge's samples stays rejected. That would need a proof variant that does not say "counter", and is left as a follow-up.
  • The review was not independent: the author reviewed the diff.

Closes #772

…s compose through avg (#772)

- SourceFacts::values_non_negative declares every sample of a metric
  non-negative, for a gauge as well as a counter (default false, omitted
  from JSON when false).
- Plan selection binds the sign of a metric's samples
  (values_non_negative[m]) to proven when the metric is declared a
  counter or declared non-negative.
- A quantile estimate over a metric's raw samples now carries that
  metric's sign statistic, so a per-series DDSketch's relative error
  composes through sum and avg once the sign is declared.
- Test: {avg,sum,max} by (region) (quantile_over_time(0.99, x[100s]))
  with x a gauge, with and without the declaration.
- Docs: the new field in the library API and the input/output workflow.
samples_of_metric now passes through an offset-only TimeShift: it moves
the window, not the samples, so their sign is the metric's. With it,
topk(10, sum by (job) (rate(data_total[5m] offset 1h))) over a declared
counter keeps its non-negative Count-Min weights, and a per-series
DDSketch quantile over `x[100s] offset 1h` of a non-negative gauge
keeps its relative bound through avg. An `@` anchor still ends it.
Tests: the offset forms of both acceptance tests.
…773 review)

`topk(k, irate(m[w]))` lowered to an exact sort and limit while
`topk(k, rate(m[w]))` got the heavy-hitter TopK intent. irate now takes
the same direct path as rate and increase (and the matching
direct_counter_ranking check). Tests: the lowering of
topk(3, rate|increase|irate(...)) and a plan-selection check that the
direct irate ranking plans.
@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation enhancement New feature or request promql PromQL front-end lowering (L1→L2) labels Oct 20, 2026
…o sample sign (#772 review)

PromQL answers quantile_over_time(phi, ...) with phi < 0 or phi > 1 as
-Inf or +Inf, which no sketch estimates and the executor's DDSketch and
KLL refuse at run time.
- Pass 1 prunes every sketch candidate of such a quantile with the new
  PruneReason::QuantileOutsideUnitInterval, so only the exact plan is
  offered and selected (this also fixes bare `quantile_over_time(2, x)`
  and `max by (...) (quantile_over_time(-1, x))`, which selected a sketch
  plan before).
- estimates_samples_of follows a quantile estimate only for phi in
  [0, 1], so such an estimate never carries its samples' sign.
Tests: the four reviewer probes plan exactly, also over a gauge declared
non-negative; the sign rule for phi = -1 and 2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation enhancement New feature or request promql PromQL front-end lowering (L1→L2)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Declare non-negative gauges so per-series quantile sketches get a guarantee through avg

1 participant