Skip to content

feat(precompute): event-time barrier for Remote Write producers - #839

Merged
zzylol merged 1 commit into
mainfrom
feat/watermark-barrier
Oct 21, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/watermark-barrier

Conversation

@zzylol

@zzylol zzylol commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Base: #824 (stored outputs built in a QueryRun). Ports #814 to the plan-executing engine.

Why

A pane closes only when a later sample of a routed metric passes its end (plus the allowed lateness). When a series stops sending, its last pane stays open, and every planned query reading that window is forwarded. Remote Write gives the producer no way to say "everything up to T is written". #814 added that statement to the old backend; this PR adds it to the engine that executes the Planner's plan.

What

  • Endpoint: POST /api/v1/precompute/watermark?time_ms=T. It answers {"barrier_ms": B}, the barrier in force: T capped at the wall clock, or a higher earlier barrier. A missing or non-integer time_ms gets 400 bad_data with the reason, and a stopped engine gets 500. Input stays open.
  • Ordering: the barrier is queued behind every write already acknowledged, and the response is sent once it has run.
  • Effect: every pane with end <= T is closed, keeping the left-open (end − width, end] convention. The last pane of a series that stopped sending is built, and a planned query reading it is answered by the plan.
  • Cap: T is capped at the wall clock, like the sample watermark.
  • Repeats: a repeated or lower T does nothing.
  • Overflow: closing reuses the expired-pane skip and checked arithmetic, so T = i64::MAX terminates and builds only the retained panes (at most seven for Example 4b).
  • Later samples: a sample that arrives later for a pane the barrier closed (one ending at or before T) counts as late. A later sample in a pane still open is accepted.
  • Several producers: send the barrier only after all of them have written through T; the README says so. The backend keeps no producer roster.
  • Design doc: for the "idle shards" open decision, this is the producer-side answer. A wall-clock idle close stays open.
  • Later in the stack: with sharded ingestion workers, the barrier advances every worker.

Key code interfaces added / modified

// data_plane/src/precompute.rs
impl Precompute {
    /// `advance_watermark_at` the wall clock.
    pub fn advance_watermark(&mut self, barrier_ms: i64) -> i64;
    /// Closes every pane with end <= min(barrier_ms, now_ms); a repeated or
    /// lower barrier does nothing. Returns the barrier in force.
    pub fn advance_watermark_at(&mut self, barrier_ms: i64, now_ms: i64) -> i64;
}
// data_plane/src/engine.rs
impl EngineHandle {
    /// Queues the barrier behind acknowledged writes and waits until it has run.
    pub async fn advance_watermark(&self, barrier_ms: i64) -> Result<i64, IngestError>;
}

Before this PR

Example 4b without raw data, with one series that stops at 69:45 and 15 s allowed lateness: the pane (60, 70 min] stays open, and quantile_over_time(0.99, latency_ms[60m]) at 70 min is forwarded.

After this PR

After POST /api/v1/precompute/watermark?time_ms=4200000, the same query at 70 min is answered by the plan, and later writes are still accepted.

Verification

  • data_plane/tests/answers.rs: barrier_serves_the_window_of_a_stopped_series. It fails before this PR, because the endpoint did not exist. Before the barrier the query is forwarded; after it, the plan answers. Input is still accepted. barrier_ms is in the response. A missing, non-numeric or fractional time_ms gets 400 bad_data with the reason.
  • data_plane/tests/precompute.rs:
    • barrier_closes_the_last_pane_of_a_stopped_series
    • barrier_is_monotonic_and_capped_at_the_wall_clock
    • barrier_at_i64_max_terminates
    • input_stays_open_after_a_barrier
  • cargo test --workspace --locked passes.

@zzylol zzylol added enhancement New feature or request area: data plane Runtime ingestion, storage, and query-serving data paths labels Oct 20, 2026
@zzylol
zzylol force-pushed the feat/kept-query-time-results branch from 8827771 to d1feb90 Compare October 20, 2026 23:26
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from 80650bd to 90cd86e Compare October 20, 2026 23:26
@zzylol
zzylol force-pushed the feat/kept-query-time-results branch from d1feb90 to 9bfa31f Compare October 20, 2026 23:46
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from 90cd86e to 7fbd02a Compare October 20, 2026 23:46
@zzylol
zzylol marked this pull request as ready for review October 20, 2026 23:46
@zzylol
zzylol force-pushed the feat/kept-query-time-results branch from 9bfa31f to 185f7db Compare October 20, 2026 23:53
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from 7fbd02a to 3cb7b7e Compare October 20, 2026 23:53
@zzylol
zzylol force-pushed the feat/kept-query-time-results branch from 185f7db to a3fa8e0 Compare October 20, 2026 23:56
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from 3cb7b7e to defe86d Compare October 20, 2026 23:56
@zzylol
zzylol force-pushed the feat/kept-query-time-results branch from a3fa8e0 to 5780d83 Compare October 21, 2026 00:09
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from defe86d to 4987a39 Compare October 21, 2026 00:10
Ports #814 to the plan-executing engine.

POST /api/v1/precompute/watermark?time_ms=T: the producer states that
every sample at or before T has been written. The request queues an
AdvanceWatermark command behind every write already acknowledged and
answers {"barrier_ms": B}, the barrier in force, once it has run: every pane with end <= T is closed, keeping
the left-open (end - width, end] convention, so the last pane of a
series that stopped sending is built and its window served from the
plan without a later sample.

- T is capped at the wall clock, like the sample watermark.
- A repeated or lower T does nothing; the answer is the barrier in force.
- The closing reuses the expired-pane skip and checked arithmetic, so
  T = i64::MAX terminates.
- Input stays open; a sample arriving later for a pane the barrier closed
  counts as late.
- A missing or non-integer time_ms gets 400 bad_data with the reason.
- With several producers, the barrier must follow all of their writes
  through T; the README says so.
@zzylol
zzylol changed the base branch from feat/kept-query-time-results to main October 21, 2026 00:17
@zzylol
zzylol force-pushed the feat/watermark-barrier branch from 4987a39 to 0a5c014 Compare October 21, 2026 00:17
@zzylol
zzylol merged commit 887af13 into main Oct 21, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: data plane Runtime ingestion, storage, and query-serving data paths enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant