[zzylol]

Stack 1/5 for executing the plan ASAPPlanner selects (#509's physical ASAP DAG). Next: #819.

Why

The backend selected its own plans, with its own ERP, SDS, precompute and query engines, and each part pinned a different Planner revision, so none of it ran the plan the Planner selects. The Planner now selects plans end to end on its main; the backend should only supply deployment inputs and execute the result.

What

• Control plane: plans a PromQL PlanningWorkload with the Planner's Stages 0–3 under the backend's deployment inputs and writes plan.json (InstalledPlan): the selected PhysicalASAPDAG plus the query each root answers. A root whose rows cannot carry the full series identity is rejected (PlanError::SeriesIdentity), not planned without it.
• Data plane: loads the plan, accepts Prometheus Remote Write bodies up to Limits::max_compressed_bytes (32 MiB), and forwards every query to Prometheus with its end-to-end headers (Authorization, X-Scope-OrgID, ...) under a timeout (--forward-timeout-seconds, default 120). Until the precompute engine lands (feat(precompute): build stored summary panes from Remote Write samples #820), Remote Write samples are validated, counted and dropped: Prometheus must receive Remote Write itself. Later PRs in the stack answer planned queries from the plan.
• Planner pin: ASAPPlanner main 58999df2 for the four crates the backend uses: asap-types (as planner-types), asap-frontend-promql, asap-logical-asap-optimization, asap-plan-selection. scripts/sync-asapplanner-main.sh and its test update these four pins. asap-executor is not pinned here because no crate uses it yet (and cargo update -p fails for a package absent from Cargo.lock); the first stack PR that uses it adds the pin and its -p line back.
• Examples: examples/planner-layering-example1.json, test(promql): enforce range step parity in production E2E #509 Example 1 in the current workload format (DataWorkload { arrival, sources }, AccuracyTarget, QueryRequirements { accuracy, max_latency_ms }), generated from the Planner's own example definitions.
• Deleted: everything else in the old backend (−176k lines), including crates/, docs/, demos/, tools/, the Python package, .dockerignore and the hadolint hook. The table in Deleted capabilities lists each piece and its replacement; the ones not replaced yet are tracked in Capabilities removed with the old backend that are not replaced yet (#818) #823.

Supersedes #788–#812, #791 and #779.

Deleted capabilities

Every piece of the old backend this PR deletes, and what replaces it. "Deferred" rows have no replacement yet and are tracked in #823. Every path is checked against git diff --name-status main...HEAD, and every deleted file falls under some row.

Deleted	What it did	Replaced by / tracked in
control_plane/src/planner_selection.rs, control_plane/src/physical/workload_cost.rs, control_plane/src/physical/workload_cost/status.rs, control_plane/src/physical/post_asap/cost_model.rs, control_plane/src/physical/deployment_cost/mod.rs, control_plane/src/physical/deployment_cost/tco.rs, control_plane/examples/workload_cost_manifest.rs	Backend-side plan selection and workload cost	ASAPPlanner Stage 3 (select_plan, cost models 1–3)
control_plane/src/physical/erp.rs, tools/empirical-bench, control_plane/examples/calibration_candidates.rs, control_plane/examples/offline_frequency_plan.rs, control_plane/tests/offline_evidence.rs, control_plane/tests/fixtures/erp-kll-measured.json	ERP, measured-cost evidence, and exporting calibration candidates for isolated measurement	ASAPPlanner measured_operation_costs, imported from sketch-bench
control_plane/src/accuracy.rs, control_plane/src/epsilon_alloc.rs	Accuracy checks and error-budget allocation	ASAPPlanner Stage 1 accuracy rules and Stage 3 AccuracyModel
control_plane/src/physical/workload_cost/materialization_candidates.rs, control_plane/src/physical/pane_reuse.rs, control_plane/src/physical/maintained_population.rs, control_plane/src/physical/compiler/windows.rs	Materialization choice, window layouts, pane reuse, maintained populations	ASAPPlanner Stage 2 materialization and Pass 2 sharing
control_plane/src/physical/sketch_catalog.rs, control_plane/src/physical/summary_catalog.rs, control_plane/src/physical/runtime_capability.rs, control_plane/sketch_capabilities.yml, control_plane/sketch_params_default.yml	Which sketches exist and their default parameters	ASAPPlanner sizing; the backend's DeploymentCapabilities
control_plane/src/query_parser, control_plane/src/physical/compiler.rs, control_plane/src/physical/post_asap/lower.rs, control_plane/src/physical/post_asap/matcher.rs, control_plane/src/physical/post_asap/deployment_expr.rs, control_plane/src/physical/post_asap/mod.rs, control_plane/src/physical/post_asap/tests.rs, control_plane/src/physical/realization.rs, control_plane/src/physical/executable_binding.rs, control_plane/src/physical/backend_stage.rs, control_plane/src/physical/mod.rs, control_plane/src/query_plan.rs, control_plane/src/query_plan/query_time.rs, control_plane/examples/compile_workload_artifact.rs	Parsing, lowering and compiling plans, and the backend-facing projection of a planning cycle	ASAPPlanner frontends and executor; the control plane installs plan.json
control_plane/src/physical/deployment_cost/wire.rs, control_plane/src/emit/backend_wire.rs, control_plane/src/emit/mod.rs, control_plane/src/backend_client.rs, control_plane/src/physical/publication.rs, control_plane/src/physical/plan_dot.rs, control_plane/src/types.rs, control_plane/src/workload.rs, control_plane/proto/anyvalue.proto, control_plane/proto/control_plane.proto, control_plane/proto/feedback.proto, data_plane/src/storage_engines/types/installed_precompute_plan.rs, data_plane/src/storage_engines/types/storage_backend.rs, data_plane/src/storage_engines/types/enums.rs, data_plane/src/storage_engines/types/mod.rs	Plan transport between control and data plane, and the data plane's installed-plan indexes	plan.json (InstalledPlan), loaded by the data plane
data_plane/src/storage_engines/types/hot_reload_config.rs	Live plan replacement: generation-consistent install of a new plan into a running data plane (ArcSwap)	Deferred, #823 (the plan is loaded at startup)
control_plane/src/clickhouse.rs, control_plane/src/query_plan/clickhouse_exact.rs, control_plane/examples/compile_clickhouse_workload.rs, control_plane/examples/audit_clickhouse_corpus.rs, control_plane/examples/support/automatic_clickhouse_corpus.rs, control_plane/tests/fixtures/sql_exact_cuts, data_plane/examples/audit_clickhouse_fallback.rs	ClickHouse planning and the ClickHouse corpus audit	Deferred, #823
control_plane/src/opamp, control_plane/proto/opamp.proto, control_plane/src/emit/monitor.rs, control_plane/src/physical/deployment_cost/online.rs, control_plane/src/runtime_samples.rs, control_plane/src/metrics_exposer.rs, data_plane/src/bin	OpAMP control, monitoring (incl. the monitor-coordinator harness), online re-planning	Deferred, #823 (re-planning is out of scope; runtime feedback is ASAPPlanner #250)
control_plane/examples/offline_planner_replay.rs, tools/run-offline-planner-replay.py, tools/o11y-execution, tools/shared-workload, tools/test_shared_panes.py	Offline planner replay, old acceptance and workload replay tooling	control_plane --workload ... --out plan.json plans offline; the correctness e2e test (#822) and the planned benefit harness
control_plane/build.rs	Stamped the Planner and backend revisions into the binaries	The Planner revision pinned in Cargo.toml / Cargo.lock
crates/asap_types	A copy of the Planner's types	ASAPPlanner asap-types, pinned
crates/asap_otel_proto, data_plane/src/drivers/ingest/otel.rs	OTel ingestion with a sketch-metric proto	Deferred, #823
data_plane/src/drivers/ingest/prometheus_remote_write.rs, data_plane/src/drivers/ingest/series_resolver.rs, data_plane/src/drivers/ingest/mod.rs, data_plane/src/drivers/mod.rs	Remote Write ingestion and series routing	Remote Write decoding in this PR (data_plane/src/remote_write.rs; samples counted and dropped); samples selected by the plan's Scan (#820)
data_plane/src/drivers/query/adapters/prometheus_http.rs, data_plane/src/drivers/query/adapters/config.rs, data_plane/src/drivers/query/adapters/traits.rs, data_plane/src/drivers/query/adapters/mod.rs, data_plane/src/drivers/query/fallback/prometheus.rs, data_plane/src/drivers/query/fallback/mod.rs, data_plane/src/drivers/query/servers/http.rs, data_plane/src/drivers/query/servers/mod.rs, data_plane/src/drivers/query/mod.rs, data_plane/src/query_engines/query_forwarding.rs, data_plane/src/query_engines/query_result.rs, data_plane/src/query_engines/request.rs, data_plane/src/utils	Prometheus query API, fallback forwarding, query results and their Prometheus JSON encoding, per-request native context	Prometheus HTTP API and forwarding in this PR; answers from the plan (#821)
data_plane/src/drivers/query/servers/metrics.rs	The data plane's Prometheus /metrics endpoint (query request counts and latency)	Deferred, #823
data_plane/src/drivers/query/adapters/victoriametrics_http.rs, data_plane/src/drivers/query/fallback/victoriametrics.rs	VictoriaMetrics adapter	Deferred, #823
data_plane/src/precompute_engine/window_manager.rs, data_plane/src/precompute_engine/ingest_handler.rs, data_plane/src/precompute_engine/maintenance_runtime.rs, data_plane/src/precompute_engine/output_sink.rs, data_plane/src/precompute_engine/raw_dag.rs, data_plane/src/precompute_engine/group_key.rs, data_plane/src/precompute_engine/metrics.rs, data_plane/src/precompute_engine/config.rs, data_plane/src/precompute_engine/engine.rs, data_plane/src/precompute_engine/mod.rs, data_plane/src/precompute_engine/precompute_engine_design_doc.md	Building stored summaries	The precompute engine (#820)
data_plane/src/precompute_engine/revisions.rs, data_plane/src/precompute_engine/frame_lineage.rs	Late-sample revisions	Deferred, #823 (late samples are dropped and counted)
data_plane/src/precompute_engine/partitioning.rs, data_plane/src/precompute_engine/series_router.rs, data_plane/src/precompute_engine/subdag_scheduler.rs, data_plane/src/precompute_engine/worker.rs	Multi-worker precompute	Deferred, #823
data_plane/src/precompute_engine/erp_observer.rs	ERP observations at runtime	ASAPPlanner measured costs (offline, sketch-bench)
data_plane/src/query_engines/asap_query_engine, data_plane/src/query_engines/mod.rs	Answering PromQL from summaries	The query engine (#821) running the plan's roots
data_plane/src/query_engines/routing	Choosing per query between summaries and raw data (capability matching, freshness probe, storage routing)	ASAPPlanner (the plan says what each query reads)
data_plane/src/query_engines/asap_clickhouse_query_engine	SQL acceleration over ClickHouse	Deferred, #823
data_plane/src/storage_engines/sketch_db/sds.rs, data_plane/src/storage_engines/sketch_db/index, data_plane/src/storage_engines/sketch_db/lifecycle, data_plane/src/storage_engines/sketch_db/current_series.rs, data_plane/src/storage_engines/sketch_db/data, data_plane/src/storage_engines/sketch_db/query, data_plane/src/storage_engines/sketch_db/mod.rs, data_plane/src/storage_engines/mod.rs, data_plane/src/storage_engines/traits.rs, data_plane/src/storage_engines/types/precomputed_output.rs	Stored summaries, index and lifecycle	SDS and SummaryStore (#819)
data_plane/src/storage_engines/sketch_db/persistence	S3 / Gorilla object-store persistence	Deferred, #823
data_plane/src/storage_engines/sketch_db/backfill	Backfill from ClickHouse or Prometheus history	Deferred, #823
data_plane/src/storage_engines/sketch_db/accuracy.rs, data_plane/src/storage_engines/sketch_db/metrics.rs	Runtime accuracy and storage metrics	ASAPPlanner guarantees; experiment metrics come with the benefit harness
data_plane/src/update_sampling	Update sampling	Deferred, #823 (not planned)
data_plane/src/tests, data_plane/tests (old files), data_plane/benches, data_plane/examples/promql, data_plane/examples/repeated_window_comparison.rs, data_plane/examples/sketch_db_diag.rs, control_plane/tests/discovery_snapshot.rs, control_plane/tests/snapshot_version.rs, control_plane/tests/o11y_exact_fallback.rs, control_plane/tests/fixtures/o11y_queries.json	Old tests, benchmarks and examples	New control_plane / data_plane tests (this PR and #819–#822)
scripts/e2e.sh	Manual end-to-end test runner	Deferred, #823
control_plane/.gitignore, data_plane/.gitignore, data_plane/.cargo/config.toml, data_plane/rustfmt.toml, data_plane/LICENSE	Per-crate tooling config and a duplicate license	The root .gitignore, default rustfmt and the root LICENSE
control_plane/Dockerfile, data_plane/Dockerfile, .dockerignore, data_plane/docker-compose.yml.j2, data_plane/query-engine-rust-cli-compose.yml.j2, demos/asapquery	Container images, compose files, demo	Deferred, #823 (with the benefit harness)
tools/autosketch-comparison, tools/erp-figure1, data_plane/examples/autosketch_comparison.rs, data_plane/examples/univmon_erp_artifact.rs	AutoSketch and ERP experiments	sketch-bench (where the AutoSketch evaluation lives)
tools/clickhouse-benefit, tools/o11y-sql-main-eval	SQL evaluation	Deferred, #823
tools/cluster-data-exporter	Google/Alibaba trace exporter	Deferred, #823
docs/01-getting-started, docs/03-how-to-guides, docs/design_docs, docs/developer_docs, docs/evaluation, docs/examples, docs/user_guide, docs/README.md, docs/offline-o11y-final-2026-09-08.md, control_plane/docs	Docs of the old architecture	The design doc for this stack; new docs follow with the stack
pyproject.toml and the Python package	Python tooling	Deferred, #823 (not planned)

How

plan_workload lowers the workload with asap_frontend_promql::lower_promql_query_workload, adds each series' identity to PromQL roots with with_promql_series_identity (a failure is returned as PlanError::SeriesIdentity, as the Planner's executor lowering does), and runs asap_plan_selection::run_stage1_to_stage3 with PlanningModels::builtin() plus the backend's DeploymentCapabilities and PlanSelectionSettings. The backend never chooses: it takes Stage 3's Selection::plan (the selected candidate's timed roots), expands each PaneChain into its panes with expand_pane_chains (the executor's compiler rejects unexpanded chains), and exports it with compile_physical_asap_workload.

run_stage1_to_stage3 is used instead of the asap-planner facade's e2e_plan because e2e_plan returns PlanOutput, whose DAG export resets every node to query time, and because the facade pulls in the SQL frontend (DataFusion), which a PromQL-only backend does not need.

The backend declares query_time_results_retained: false: the data plane does not implement MatAtQueryKeptAcrossEvaluations, so the Planner never offers it.

The data plane's write route sets DefaultBodyLimit::max(limits.max_compressed_bytes), replacing axum's 2 MB default, so bodies up to the Remote Write limit reach remote_write::decode and larger ones get 413. Forwarding copies every request header except hop-by-hop headers (Connection and the headers it names, Keep-Alive, Proxy-*, TE, Trailer, Transfer-Encoding, Upgrade), Host, Content-Length, and Accept-Encoding (only the response's Content-Type is copied back, so the response must not be compressed). The reqwest client's timeout covers sending the request and reading the whole response; a timeout is a 502.

Key code interfaces added / modified

// control_plane/src/lib.rs
pub struct InstalledPlan {
    /// PromQL text of each root of `dag`, in root order.
    pub queries: Vec<String>,
    /// The selected candidate's DAG, with every `PaneChain` expanded into
    /// the panes the executor runs.
    pub dag: PhysicalASAPDAGDocument,
}
impl InstalledPlan {
    /// The DAG validates and has one root per query.
    pub fn validate(&self) -> Result<(), PlanError>;
    /// Reads and validates a plan written by `save`.
    pub fn load(path: &Path) -> Result<Self, PlanError>;
    /// Writes the plan as JSON.
    pub fn save(&self, path: &Path) -> Result<(), PlanError>;
}

/// What the backend gives the Planner about its deployment.
pub struct DeploymentInputs {
    /// What the data plane can execute and keep (Stages 1–3).
    pub deployment_capabilities: DeploymentCapabilities,
    /// How Stage 3 prices candidates: cost model, resource speed, and any
    /// measured operation costs.
    pub plan_selection: PlanSelectionSettings,
}
/// `Default`: `backend_capabilities()` with `PlanSelectionSettings::default()`
/// (cost model 1, illustrative unit costs).
impl Default for DeploymentInputs;

/// Stages 0–3; returns the plan Stage 3 selected.
pub fn plan_workload(workload: &PlanningWorkload, inputs: &DeploymentInputs)
    -> Result<InstalledPlan, PlanError>;

/// Every summary the Planner can build (`supported_summaries: None`), raw
/// samples kept in Prometheus, no query-time result kept across evaluations.
pub fn backend_capabilities() -> DeploymentCapabilities;

pub enum PlanError {
    /// The workload is not PromQL.
    UnsupportedLanguage(QueryLanguage),
    /// `PlanningWorkload::validate` failed.
    Workload(String),
    /// Query `i` lowers to a standalone scalar, which the data plane cannot answer.
    ScalarRoot(usize),
    /// The PromQL frontend failed.
    Lowering(String),
    /// Query `query`'s rows cannot carry the full series identity.
    SeriesIdentity { query: usize, message: String },
    /// Stage 1 or Stage 3 failed (including no valid candidate).
    Planning(String),
    /// Pane-chain expansion, export or document validation failed.
    InvalidDag(String),
    /// The DAG's root count differs from the query count.
    RootCount { roots: usize, queries: usize },
    /// Reading or writing `plan.json` failed.
    File { path: String, message: String },
}

// data_plane/src/lib.rs
/// Prometheus's default query timeout (two minutes).
pub const DEFAULT_FORWARD_TIMEOUT: Duration;
impl DataPlane {
    /// `forward_timeout` bounds each forwarded query, from sending the
    /// request to reading the whole response.
    pub fn new(queries: Vec<String>, prometheus_url: String, forward_timeout: Duration) -> Self;
}
/// `/api/v1/write` (body limit `limits.max_compressed_bytes`),
/// `/api/v1/query[_range]` (forwarded), `/-/ready`.
pub fn router(state: Arc<DataPlane>) -> Router;

control_plane CLI: --workload, --deployment-capabilities (Planner DeploymentCapabilities JSON), --measured-operation-costs (Planner MeasuredOperationCost list JSON), --out.
data_plane CLI: --plan, --prometheus-url (its help states that Remote Write samples are counted and dropped until the precompute engine lands), --forward-timeout-seconds (default 120), --listen.

Before this PR

The backend selected its own plan with its own cost model and engines, on Planner revisions that differed per crate.

After this PR

control_plane --workload examples/planner-layering-example1.json --out plan.json
wrote plan.json (2 queries, 13 nodes)

The plan is the one the Planner selected (for Example 1, P1: both queries exact at query time). The data plane installs it, counts and drops Remote Write samples, and forwards queries, with their auth and tenancy headers, marked x-asap-execution: forwarded. scripts/sync-asapplanner-main.sh updates the four Planner pins and Cargo.lock successfully.

Usage

# Workload: #509 Example 1 (two dashboard panels over http_requests_total).
WORKLOAD=examples/planner-layering-example1.json
# Where the control plane writes the selected plan.
PLAN=plan.json
# Plan with the default deployment inputs.
cargo run -p control_plane -- --workload "$WORKLOAD" --out "$PLAN"
# -> wrote plan.json (2 queries, 13 nodes)

# Prometheus that answers forwarded queries. It must also receive Remote
# Write itself: the data plane counts and drops samples for now.
PROMETHEUS=http://localhost:9090
# Serve Remote Write (/api/v1/write) and the query API on 8088.
cargo run -p data_plane -- --plan "$PLAN" --prometheus-url "$PROMETHEUS" --listen 127.0.0.1:8088 &
# Any query is forwarded with its end-to-end headers; the response says so.
curl -s -D - -o /dev/null -H 'Authorization: Bearer t' -H 'X-Scope-OrgID: tenant-a' \
  'http://127.0.0.1:8088/api/v1/query?query=up' | grep x-asap
# -> x-asap-execution: forwarded

Verification

• python3 -m unittest discover -s scripts/ci -p 'test_*.py': 5 tests pass. The fake cargo now rejects a -p spec that is not in the repository's Cargo.lock (exit 101, like real Cargo), and a new test runs the sync script against the repository's own Cargo.toml; it failed with package ID specification `asap-executor` did not match any packages before the pin was dropped.
• scripts/sync-asapplanner-main.sh run for real on a scratch copy of the tree: rewrote the four pins and updated Cargo.lock to ASAPPlanner main (synchronized ASAPPlanner dependencies to f9dd41e0...).
• cargo fmt -p control_plane -p data_plane -- --check; cargo check --workspace --locked; cargo clippy --workspace --all-targets --locked -- -D warnings; cargo test --workspace --locked: all pass (control plane 4 tests, data plane 4 unit + 5 server tests).
• Tests written first and seen failing before each fix: a Remote Write body between 2 MB and 32 MiB got 413 (now 204), and a body over 32 MiB gets 413; Authorization and X-Scope-OrgID were not forwarded (now forwarded, while Host and Keep-Alive are not); a failed series-identity rewrite was swallowed (now query 0: series identity: ...). A forwarded query slower than the timeout gets 502.
• Usage run on this PR's tree against a stand-in Prometheus: x-asap-execution: forwarded, and the stand-in received Authorization: Bearer t and X-Scope-OrgID: tenant-a.
• node scripts/ci/test_planner_merge.js was not run locally (local Node v12 has no node:fs); MVP CI runs it.