[iscekic]

bot: Sweep v2 for the offline benchmark: agentic container runs, time to first token (TTFT), long-context cases, and a separate frontier suite. This PR is stacked on #7510.

Change

• sweep:offline --suite efficient|frontier selects DECIDER_CASES or the new FRONTIER_CASES.
• Agentic cases run through the real bench container (--container-url, --cli-api-url, KILO_TOKEN). --skip-agentic leaves them out.
• Direct requests stream, and record ttftMs plus total latency. Summaries and the held-out scoreboard report p50TtftMs.
• Long-context cases: deterministic, seeded generators build a synthetic repository, an agent transcript, and logs and configs, at about 32k, 96k and 160k tokens. The facts the answer needs are spread across the context, and the generator computes the answer.
  • Efficient tier: 36 cases, 3 per route.
  • Frontier tier: 24 harder cases at 160k tokens.
• Frontier cases: 270 cases, 15 per route, all json_equal, with answers computed by node oracles. They target multi-file semantics, concurrency, numeric edge cases and constrained optimization.

Result: efficient sweep with long-context cases (44 platform pairs, held-out)

Live table picks against picks rebuilt from the train split:

Metric	Change
Accuracy	+18.7%
Estimated production cost per request	−37%
p50 latency	−21%

• In this run TTFT covers only the newly streamed rows. A full streamed re-sweep is running and will report TTFT for every case.
• Agentic routes are not covered: the container path needs a funded Kilo token.

Rollout note

• The long-context cases change the efficient decider dataset hash, so the next platform drain re-measures every pair. Long-context cases cost much more per case than short ones.

Verification

• VM: contracts 110, auto-routing 275, auto-routing-benchmark 441 tests pass. Typecheck passes, and type-aware lint is clean.
• Long-context tests cover deterministic materialization, sizes within ±15% of target, and oracle answers that match their checks.
• Real sweep: 42,768 case results, not truncated.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The only incremental change since the last review is a formatting-only reflow of the stream tests in sweep.test.ts; all prior findings remain resolved and no new defects were introduced.

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	0

Files Reviewed (1 file)

• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts - 0 issues (prettier-only reflow of the null-error and [DONE] cancellation tests; no behavioral change)

Note: this incremental review covers only changes after commit 4700323b5e (commit 9b5df91c7 "style(auto-routing): format"). The three prior findings (surrogate oracle, "error": null, [DONE] termination) were fixed in 4700323b5e and verified resolved.

Previous Review Summaries (2 snapshots, latest commit 4700323)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 4700323)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental changes resolve all three prior findings and introduce no new defects: the Unicode oracle now preserves UTF-16 surrogate halves, "error": null chunks are no longer treated as failures, and the stream reader stops and cancels at [DONE] instead of blocking on an open connection.

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	0

Files Reviewed (4 files)

• services/auto-routing-benchmark/src/datasets/frontier-cases/refactoring.ts - 0 issues (oracle surrogate fix verified)
• services/auto-routing-benchmark/src/datasets/decider-cases.test.ts - 0 issues (regression test valid)
• services/auto-routing-benchmark/src/scoreboard/sweep.ts - 0 issues (!= null check and [DONE] termination fix verified)
• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts - 0 issues (new tests valid)

Previous review (commit 4d01e77)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

A corrupted benchmark oracle in frontier-cases/refactoring.ts can only be matched by a wrong answer, and the new streaming reader in scoreboard/sweep.ts mishandles OpenRouter "error": null frames and does not stop reading at [DONE].

Overview

Severity	Count
CRITICAL	1
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

CRITICAL

File	Line	Issue
services/auto-routing-benchmark/src/datasets/frontier-cases/refactoring.ts	713	unicode-key-identity oracle encodes '𝌆'.split('') as two U+FFFD replacement characters instead of the UTF-16 code units \ud834/\udf06, so json_equal rejects a correct answer and accepts an incorrect one.

WARNING

File	Line	Issue
services/auto-routing-benchmark/src/scoreboard/sweep.ts	53	chunk.error !== undefined throws on a valid chunk carrying "error": null, discarding the answer/usage and retrying; repo precedent uses != null.
services/auto-routing-benchmark/src/scoreboard/sweep.ts	84	Reader loop exits only on socket close, so a stream left open after [DONE] blocks until the 170s fetch timeout and drops the received answer.

Files Reviewed (25 files)

• services/auto-routing-benchmark/README.md - 0 issues
• services/auto-routing-benchmark/scripts/offline-sweep.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/decider-cases.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/decider-cases.test.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/frontier-cases/agentic-execution.ts - spot-checked
• services/auto-routing-benchmark/src/datasets/frontier-cases/debugging.ts - spot-checked
• services/auto-routing-benchmark/src/datasets/frontier-cases/implementation.ts - spot-checked
• services/auto-routing-benchmark/src/datasets/frontier-cases/index.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/frontier-cases/investigation.ts - spot-checked
• services/auto-routing-benchmark/src/datasets/frontier-cases/planning-design.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/frontier-cases/refactoring.ts - 1 issue
• services/auto-routing-benchmark/src/datasets/long-context/cases.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/generators/common.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/generators/index.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/generators/logs.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/generators/repo.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/generators/transcript.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/index.ts - 0 issues
• services/auto-routing-benchmark/src/datasets/long-context/long-context.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/compare.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/schema.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/sweep.ts - 2 issues

Note: the frontier-cases/* dataset oracles are large and data-heavy; those files were spot-checked (case ids verified unique across all files, taxonomy/tier verified, sample oracles re-derived by hand) rather than exhaustively recomputed.

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 31K · Output: 2.1K · Cached: 195.6K}

_{Review guidance: REVIEW.md from base branch igor/auto-efficient-retune}