[zzylol]

Stack: #821 → #824 → #839 → this PR (deferred; see Status).

Why

One engine thread ran ingestion, pane builds and every query, because the Planner's DAG types used to hold Rcs. A long query held up ingestion and every query behind it. ASAPPlanner main (pinned ecea897f) now compiles plans that are Send + Sync, so the data plane can run ingestion and queries on many threads over one compiled plan.

What

• Ingestion workers (--ingestion-workers, default one per core). Remote Write series are sharded by a hash of their label set; each worker owns its shard's open panes.
  • Global watermark = the minimum over the workers that have seen a sample of their newest routed raw sample, capped at the wall clock, minus the allowed lateness; or the newest barrier if later.
  • The worker that moves the watermark past a pane end closes the pane: under every worker's lock it takes each worker's samples of the pane and runs the definition's build once. A pane never waits for an idle worker, and a group spanning workers is built whole.
  • The event-time barrier (feat(precompute): event-time barrier for Remote Write producers #839) runs after every worker has run the writes acknowledged before it, and closes every worker's panes.
  • A write is admitted whole or refused (503) by the existing queue bound, then sent to every worker, so a retry never doubles samples.
• Query threads (--query-threads, default one per core). One query per thread; queries no longer run on an ingestion thread. Before reading, a query waits until every worker has run every write acknowledged before it.
• Compile once. Ingestion and QueryEngine are compiled when the plan is installed and shared through Arc.
• SDS: SummaryStore holds Arc summary instances under a RwLock. An instance is inserted once and never rewritten (insert returns the stored one); readers clone the Arc under the read lock. Query-run summaries (StoredOutputBuild::QueryRun) go to the same store from any query thread.
• jemalloc (tikv-jemallocator) is the data plane's global allocator.

How

• precompute::Shared is what ingestion threads share: the compiled Ingestion, each worker's open panes (Mutex), each worker's newest sample, the barrier, per-definition first pane and built-through end, the SDS and the counters. Shared::ingest_at(worker, …) takes a shard, then closes panes. Precompute runs all workers on the caller's thread, for tests.
• QueryEngine::evaluate takes &self; its counters are atomics (kept_stats()).
• Right after startup a worker with no sample yet does not hold the watermark back, so the first worker to take a request may close a pane before another takes its part; the allowed lateness covers that.

Before this PR

One engine thread: a planned query waited behind ingestion and every other query; the SDS was owned by that thread.

After this PR

Ingestion runs on N workers and queries on M threads over one compiled plan; queries read summary instances concurrently by Arc.

Verification

cargo test --workspace --locked passes (fmt, clippy and the benefit harness were not run for this draft). New tests:

• sds::store_shares_instances_and_never_rewrites_them, sds::store_is_read_concurrently;
• precompute::a_pane_closes_once_every_shard_has_passed_it (minimum watermark over 2 shards), precompute::a_barrier_closes_every_shards_panes, precompute::sharded_panes_answer_like_one_shard (4 shards against 1, Example 4b's KLL and Example 1's rates and sums);
• answers::answers_concurrent_queries_like_one_at_a_time (32 concurrent queries, 4 workers, 4 query threads), answers::a_query_reads_the_panes_of_the_writes_before_it (8 runs, no wait between write and query).

Existing tests changed only where the API did: kept_stats is a method, and SummaryStore::get returns an Arc.

Status: deferred (MVP is single-threaded)

Concurrency is deferred for the MVP. This branch stays at a70c2382, stacked on the old #839 head 7fbd02a6. These review findings are still open:

• H1 (High). Right after startup a pane can be built from only some shards. The watermark is the minimum over workers that have already taken a sample, so a worker whose first shard is still queued does not hold it back. Fix (prototyped locally, not pushed): try_ingest records each shard's oldest routed sample as queued for its worker before sending, and that holds the watermark until the worker has taken it. Add a test with real worker threads (1, 2 and 4 workers, several runs) asserting zero late samples and answers equal to query_time_reference. The "allowed lateness covers that" claim in the code doc is wrong and must go.
• M1. Pane builds hold every worker's lock. Take the samples out and advance built_through under the locks, then build and insert after releasing them.
• M2. An idle shard pins the global watermark. Add a per-worker idle timeout (lateness plus one pane, on wall-clock time relative to the other workers) and document it in the README next to the barrier.
• L1. Concurrent evaluations can both build the same QueryRun output. Count built and stored once, and single-flight the build.
• L2. A failed send leaves samples_queued too high for the shards that were not sent.
• L3. Use the query_run names throughout instead of "kept".
• L4. Add a Usage section.