Skip to content

feat(plan-selection): a setting to accept outputs without accuracy guarantees - #799

Merged
zzylol merged 1 commit into
mainfrom
feat/ignore-accuracy-guarantees
Oct 21, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/ignore-accuracy-guarantees

Conversation

@zzylol

@zzylol zzylol commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #774.

Why

The MVP benchmark compares sketch plans for queries such as avg by (region) (quantile_over_time(0.99, data[100s])). Stage 3 has no analytical guarantee for these plans, so it rejects every one of them with NoAccuracyGuarantee, and a deployment that doesn't retain raw data gets no plan at all. The hook AccuracyModel::accepts_output_regardless_of_guarantee already exists. But before this PR there was no built-in model that used it and no JSON setting or stage_pipeline flag to turn it on.

What

  • A built-in IgnoreAccuracyGuarantees accuracy model. It accepts every output regardless of its guarantee. Its satisfies is the built-in rule, and may_accept_regardless_of_guarantee returns true, so the joint search from Joint planning search: stream Stage 1 to Stage 3 and reject permanently invalid query candidates (methods 2 + incumbent) #792 rejects no candidate early for accuracy.
  • PlanSelectionSettings::ignore_accuracy_guarantees (JSON: {"ignore_accuracy_guarantees": true}, default false). When it is set, planning uses IgnoreAccuracyGuarantees instead of the caller's PlanningModels::accuracy. This covers select_plan, e2e_plan and plan_installable.
  • stage_pipeline --ignore-accuracy-guarantees, which also works together with --plan-selection-settings.
  • The report marks each output that was accepted only because of the setting, i.e. the guarantee check would have rejected it:
    • AcceptedAccuracy::guarantee_check_rejection holds the cause (NoAccuracyGuarantee or AccuracyTargetMissed);
    • SelectedPlanReport::queries_accepted_without_guarantee lists the affected queries in the installable plan;
    • guarantee_check_rejection appears in the stage_pipeline JSON.
  • Executability checks still apply: Count-Min over weights not proven non-negative, and exact per-series aggregates the executor has no kernel for.

How

PlanningModels::effective_accuracy() returns IgnoreAccuracyGuarantees when the setting is on, and models.accuracy otherwise. Every read of the accuracy model in plan selection goes through it, including the early-rejection checks from #792. In accuracy_check, the guarantee and evidence check moved into guarantee_check. That check now always runs. When the model accepts the output regardless of its guarantee, the check's result is only recorded. Otherwise the result decides, as before.

Key code interfaces added / modified

crates/plan-selection/src/accuracy.rs

/// Accepts every query output regardless of its guarantee; satisfies = built-in rule.
pub struct IgnoreAccuracyGuarantees; // new
impl AccuracyModel for IgnoreAccuracyGuarantees {
    fn satisfies(&self, guarantee: &AccuracyGuarantee, target: &AccuracyTarget) -> bool; // built-in rule
    fn accepts_output_regardless_of_guarantee(&self, _output: &QueryRoot, _target: &AccuracyTarget) -> bool; // true
    fn may_accept_regardless_of_guarantee(&self) -> bool; // true
}

crates/plan-selection/src/lib.rs

/// Stays. Stage 3's settings, loadable from JSON.
pub struct PlanSelectionSettings {
    /// Cost model. // unchanged
    pub cost_model: CostModel,
    /// Time per op / scanned byte. // unchanged
    pub resource_speed: ResourceSpeed,
    /// Horizon for non-repeating queries. // unchanged
    pub non_repeating_query_horizon_seconds: f64,
    /// Measured per-operator CPU. // unchanged
    pub measured_operation_costs: MeasuredOperationCosts,
    /// Accept every output regardless of its guarantee; default false.
    pub ignore_accuracy_guarantees: bool, // new
}
/// Stays. What Stage 3 accepted for one query.
pub struct AcceptedAccuracy {
    /// Target's metric. // unchanged
    pub target_metric: Option<AccuracyMetric>,
    /// Accepted guarantee's metric. // unchanged
    pub accepted_metric: Option<AccuracyMetric>,
    /// Accepted directly by the model. // unchanged
    pub accepted_regardless_of_guarantee: bool,
    /// When accepted regardless, the cause the guarantee check would have rejected it with; None if its guarantee meets the target.
    pub guarantee_check_rejection: Option<RejectionCause>, // new
    /// Sketches read. // unchanged
    pub sketches: Vec<String>,
}

crates/plan-selection/src/pass/mod.rs

impl<'a> PlanningModels<'a> {
    /// The accuracy model planning uses: IgnoreAccuracyGuarantees when the setting is on, else `accuracy`.
    pub fn effective_accuracy(&self) -> &'a dyn AccuracyModel; // new
}

crates/planner/src/installable.rs

pub struct SelectedPlanReport {
    // selected_candidate_id, cost_per_second, predicted_usage, search: unchanged
    /// Entry indices whose output was accepted only regardless of its guarantee; omitted from JSON when empty.
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
    pub queries_accepted_without_guarantee: Vec<usize>, // new
}

CLI: stage_pipeline ... --ignore-accuracy-guarantees (new).

Usage

// The user's query: the average per group of per-series p99s, rank 0.01, δ 0.01, every 10 s.
let workload = average_of_p99s_workload();
// A deployment that keeps no raw samples.
let capabilities = DeploymentCapabilities { raw_data_retained: false, ..Default::default() };
// The deployment's Stage 3 settings as JSON, with the new switch.
let settings: PlanSelectionSettings = serde_json::from_str(r#"{"ignore_accuracy_guarantees": true}"#)?;
// Stages 1-3 run. Stage 3 asks models.effective_accuracy(), which is IgnoreAccuracyGuarantees,
// so the DDSketch pane plan is valid, and the cheapest plan is selected.
let plan = plan_installable(UserInput::new(&workload, FrontendInput::Promql { histograms: None },
    PlanningModels::builtin().with_deployment_capabilities(&capabilities).with_plan_selection(&settings))).await?;
// The report marks query 0 as accepted without a guarantee.
assert_eq!(plan.selection.unwrap().queries_accepted_without_guarantee, vec![0]);

Without the setting, the same call fails with no valid candidate: exact plans are RawDataNotRetained, and sketch plans are NoAccuracyGuarantee. This flow is the test ignoring_accuracy_guarantees_selects_a_sketch_plan_that_runs.

Before this PR

The only way to use the hook was to write an AccuracyModel in code. JSON-driven runtimes and stage_pipeline couldn't select sketch plans that have no guarantee. The report also couldn't tell an output whose guarantee was met apart from one accepted only by the model.

After this PR

One setting, "ignore_accuracy_guarantees": true, or one stage_pipeline flag, compares candidates on cost alone. The report lists every output that is accepted only because of the setting. With the setting off, nothing changes.

Caveat: DDSketch needs strictly positive inputs at run time, and the executor refuses a sample of 0 or below. When the setting accepts DDSketch over a metric not declared positive, the plan fails at run time on such a sample. The undeclared MVP p99 query selects P5-m1 (DDSketch panes).

Verification

Speed mode: cargo test --workspace --locked --no-fail-fast ran on the PR head. 151 test binaries: 1,799 passed, 2 failed, 36 ignored. The 2 failures are the #795 tests a_non_negative_gauges_merged_ddsketch_panes_carry_the_relative_bound_through_avg and an_undeclared_gauges_merged_ddsketch_panes_get_no_relative_bound_through_avg. They already fail on main at 825504e, with the same assertions in main's own CI test job: P3-m1 vs P5-m1, which looks like a semantic conflict between #792 and #795. This PR doesn't touch them. Main's format-and-lint job also fails, on a clippy type_complexity error in crates/types/src/ir/properties/summary_coverage.rs, which this PR doesn't touch either.

New tests:

  • runtime_install::ignoring_accuracy_guarantees_selects_a_sketch_plan_that_runs: without the setting, planning fails with no valid candidate. With the setting loaded from JSON, a DDSketch plan is selected, the report marks query 0, and the plan runs. Its answers equal the query-time reference and are within 5 of the exact per-group averages of p99s.
  • runtime_install::ignoring_accuracy_guarantees_plans_the_undeclared_p99_query: the MVP p99 query over an undeclared gauge gets a plan with no raw reads, and the report marks query 0.
  • plan_selection::tests::unproven_count_min_weights_reject_even_when_ignoring_guarantees: CmsWithHeap over a gauge is still CountMinWeightsNotProvenNonNegative.
  • plan_selection::tests::ignoring_guarantees_accepts_and_marks_outputs_the_check_rejects: each rejection that NoAccuracyGuarantee or AccuracyTargetMissed causes becomes a valid candidate whose guarantee_check_rejection holds that cause. Outputs whose guarantee meets the target record None.
  • stage_pipeline::ignore_accuracy_guarantees_flag_accepts_outputs_without_a_guarantee: the flag makes the no_accuracy_guarantee candidates valid and marked.

The committed dag-viewer fixture was regenerated. Its only change is the new guarantee_check_rejection key.

…arantees

PlanSelectionSettings::ignore_accuracy_guarantees selects the built-in
IgnoreAccuracyGuarantees model in place of the caller's accuracy model.
Each accepted output records the cause the guarantee check would have
rejected it with, and the installable plan's report lists those queries.
Executability checks still apply. stage_pipeline gains
--ignore-accuracy-guarantees.
@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation enhancement New feature or request optimizer Workload optimization and plan selection labels Oct 21, 2026
@zzylol
zzylol merged commit aceedff into main Oct 21, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation enhancement New feature or request optimizer Workload optimization and plan selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A setting to select candidates without accuracy guarantees

1 participant