feat(asap-tools): feed the sketchdb arm in the ClickHouse runner

[milindsrivastava1997]

Part of #874 (2 of 3). Stacked on #899; review only its commits.

Problem: experiment_run_clickhouse.py refused sketchdb with dataset.paced_feeder: its engine ingested dataset.local_data_file, which a fed config doesn't have.

Fix:

• In a fed run, sketchdb starts the engine on remote-write ingest (streaming.remote_write.base_port/path) instead of JSON.
• Once the engine is ready, _start_feed recreates the tables and starts the feed to ASAPQuery only; then set_data_start, as baseline does.
• Fed sketchdb feeds no ClickHouse copy and sets forward_unsupported_queries: false. The issue proposed a shared feed so forwarded queries had data, but a ClickHouse feeder during sketchdb would load the node its CPU and latency are measured on, which baseline never pays, and the generated arm (feat(asap-tools): sketchdb arm for sketch-bench ClickHouse workloads #901) runs only planned queries. An unplanned query now fails visibly instead. _feed_sinks holds this choice. Unfed (JSON) sketchdb runs still forward.
• Queries wait no flow.steady_state_wait: the aligned groups' starting delay covers it. The mode waits out the feed before syncing, so feeders write their stats.
• A fed dataset has no dataset.precompute to build the planner's input from, so fed sketchdb needs controller.prebuilt_output_dir; this is checked before ClickHouse starts. No ClickHouse discovery is used.
• The feed's alignment reads a mode that runs every query group, not whichever mode is listed first (sketchdb may skip trailing groups, fix(asap-tools): give sketchdb its own SQL in ClickHouse runs #899).

Not covered by tests: the wiring inside main(); it needs a real run (see #874).

Tests: tests/test_experiment_run_clickhouse.py (fed sketchdb needs a prebuilt plan; each mode's feed sinks; sketchdb's queries follow its own feed's start).

🤖 Generated with Claude Code

[milindsrivastava1997]

Code review findings (Claude Code).

[milindsrivastava1997]

The feed alignment just above (~652–657: the LCM of the aligned groups' intervals and the align_to_interval / query_time_offset checks) reads only experiment_modes[0]'s client config. Its comment, 'Every mode's client config has the same query timing', stopped being true once sketchdb can skip trailing groups (#899).

Example: a config with experiment: [{mode: sketchdb}, {mode: baseline}]. The LCM then covers only the structured groups. If a PromQL group has an interval no structured group has (15 s vs 10 s), baseline's feed start isn't a multiple of 15 s, and its floored eval windows start before data_start and score partial windows. An invalid offset in a group only baseline runs also passes the pre-start check. Suggest using the union of groups across all modes, or baseline's config.

[milindsrivastava1997]

Fixed in 7866734: the alignment and aligned-group checks now read the client config of a mode that runs every group (any mode but sketchdb).

[milindsrivastava1997]

In fed sketchdb runs this starts two feeder processes per metric on the measured node (engine remote-write plus ClickHouse INSERT); baseline feeds ClickHouse only. Sketchdb's monitored CPU and latency therefore include competition baseline never sees, which biases the comparison against ASAPQuery. The ClickHouse feed is needed only for forwarded queries, and the generated sketchdb arm (#901) runs only structured groups, which ASAPQuery answers itself.

Separately: this sink choice duplicates paced_feeder_sinks(mode, ...) in experiment_utils/services/paced_feeder.py, so the rule 'sketchdb also feeds the query engine' now lives in two places. Suggest reusing that helper.

[milindsrivastava1997]

Fixed in 6c58a8b: a fed sketchdb run now feeds only ASAPQuery and sets forward_unsupported_queries: false, so it doesn't pay for a ClickHouse feeder that baseline never runs. _feed_sinks holds the choice, with tests. On the duplication: paced_feeder_sinks builds Prometheus remote-write sinks, so it isn't reused here.

[milindsrivastava1997]

Stale comment: the section header above (~line 772) still says --- sketchdb mode: precompute engine + JSON ingest + ClickHouse fallback ---, which contradicts this branch setting ingest_json_config = None in fed runs.

[milindsrivastava1997]

Fixed in 7866734.