Repository navigation
feat: rate, increase and irate of a declared counter are non-negative Count-Min weights (#773) - #777
Merged
Merged
Conversation
…ection current_series and its row reference compared sample bits against the literal 0x7ff0_0000_0000_0002; they now read operators::STALE_MARKER, its one definition.
…are non-negative Count-Min weights (#773) - WeightDomain::CounterDerivativeOfMetric { metric }: an open requirement that heap weights are reset-corrected rate/increase/irate values of `metric`'s samples (or plain sums of them). Plan selection confirms it as NonNegative { ResetAwareCounterDerivative } when the data workload declares `metric` a counter, and leaves it unproven otherwise. - The keyed top-k realization over `sum by (...) (rate|increase|irate(m))` records that requirement instead of proving it from the operator alone: over a gauge, a negative sample can give a negative rate, so the Count-Min-with-heap candidate is now rejected with CountMinWeightsNotProvenNonNegative. Over a declared counter it passes the weight check and goes on to the accuracy and cost checks. - The PromQL frontend recognizes `topk(k, sum by (...) (irate(m[w])))` as a heavy-hitter ranking, as it does for rate and increase. - Test: topk(10, sum by (job) (increase|rate|irate(data_total[5m]))) with data_total declared a counter and a gauge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Count-Min top-k of
topk(10, sum by (job) (increase(data_total[5m])))adds each series'increaseas a weight, and Count-Min's bound and the executor both need those weights to be non-negative. Reset correction keepsrate,increaseandiratenon-negative only when the samples themselves are, that is, when the metric is a counter. So whether these candidates are valid depends on the metric's declared type.What
maindid differed from what the issue describes. I checked it with the acceptance test below:increaseandrate: Pass 1 proved the weights non-negative (ResetAwareCounterDerivative) from the operator alone, without reading the metric's type. The Count-Min-with-heap candidate therefore passed the weight check over a counter, which is what the issue asks for, but also over a gauge, where the proof does not hold.irate: the frontend did not lowertopk(k, sum by (...) (irate(...)))to a heavy-hitter top-k, so there was no Count-Min candidate at all.I could not reproduce the rejection the issue reports over a counter on this shape. This PR implements the expected behaviour the issue states: admitted over a declared counter, rejected over a gauge.
What
rate,increaseoriratetop-k are proven non-negative. The Count-Min-with-heap candidate passes the weight check and still goes through the accuracy and cost checks.CountMinWeightsNotProvenNonNegative. The reason names the metric:… (rates or increases of \data_total`, which is not declared a counter)`.topk(k, sum by (...) (irate(m[w])))is lowered to a heavy-hitter top-k, as it already was forrateandincrease.current_series.rsand its row reference compare against the sharedSTALE_MARKERinstead of the literal.How
This follows the existing
WeightDomain::SamplesOfMetricpattern. Logical optimization records an open requirement without reading the data workload, and Stage 3 confirms it from the declared metric types.rate,increaseoriraterecordsCounterDerivativeOfMetric { metric }instead of an unconditional proof. The metric comes from the derivative's input, through the samesamples_of_metricrule.WeightDomain::confirmedturns the requirement intoNonNegative { ResetAwareCounterDerivative }for a declared counter, and intoUnknownOrSignedotherwise. Stage 3's existing Count-Min weight check andwith_confirmed_weights(the plan the executor runs) both call it, so the executor still receives the proof it already keys on.UnknownOrSigned, so no Count-Min candidate is built. Count-Sketch, which needs no sign, is unaffected.Key code interfaces added / modified
crates/types/src/ir/schema/sketch.rsUsage (the test
count_min_over_a_counter_derivative_needs_a_declared_counter):Before this PR
data_totaltopk(10, sum by (job) (increase(...))), Count-Min-with-heaprateirateAfter this PR
data_totalincrease,rateorirate, Count-Min-with-heapCountMinWeightsNotProvenNonNegativeVerification
Regression test.
plan-selectioncount_min_over_a_counter_derivative_needs_a_declared_countercoversincrease,rateandirate, each withdata_totaldeclared a counter and a gauge. Onmainit failed: over a gauge,increase's Count-Min candidate was rejected only withNoAccuracyGuarantee, not for its weights. It passes now.Commands run at the head commit. Each was run with
CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0, which only reduces disk use. All passed.cargo test --workspace --locked: exit 0, 150 test binaries, 0 failures.cargo fmt --all -- --checkcargo clippy -p asap-types -p asap-executor -p asap-plan-selection -p asap-logical-asap-optimization -p asap-frontend-promql --all-targets --all-features --locked -- -D warningscargo check -p asap-planner --tests --lockedNot applicable: performance measurements (no hot-path change).
Limitations.
weighted_topk_bindingis ignored for that reason). This PR changes which candidates pass Stage 3's weight check, not execution.Closes #773