[iscekic]

bot: Add an offline held-out sweep that measures a routing-table change before it ships, and record the result for this series. This PR is stacked on #7504. Merge #7492, #7493, #7496 and #7504 first.

Change

• pnpm sweep:offline --pairs pairs.json --catalog live|file --reps N --concurrency N --max-usd N --out sweep.json [--resume sweep.json]
  • Runs every non-agentic decider case against each (model, variant) pair through OpenRouter, with the catalog variant's request fragment. Mechanical checks and the Jev judged grader score the answers.
  • A stable hash of the case id splits each route about evenly into train and heldOut.
  • Spend is capped by --max-usd. The run checkpoints regularly and resumes from the checkpoint.
• scoreboard build --sweep sweep.json [--rebuild-from-sweep] fills quality.heldOut:
  • Without --rebuild-from-sweep: the held-out accuracy, estimated production cost and p50 latency of the given table's first picks.
  • With --rebuild-from-sweep: rebuilds the table from train only, with the current ranking policy (production cost estimate and Wilson bound), then evaluates its first picks on heldOut.

Result: all 44 platform pairs, 2 repetitions

Live table picks against picks rebuilt by this series' policy from the new cases. The metrics are weighted by the production route mix and measured on held-out cases only.

Held-out metric	Change
Accuracy	+16.6% (about 0.83 → 0.96)
Estimated production cost per request	−56%
p50 model latency	−28%

• 14 of 15 non-agentic routes had held-out coverage for both picks. Agentic routes need the CLI container and are measured by the normal platform drain.
• The sweep cost a few dollars. The live table ranks on the old easy cases, so this result combines the new cases and the new policy; that combination is what ships.

Rollout

• After feat(auto-routing): harder decider cases, Jev-graded checks and Wilson ranking #7504 deploys, the new dataset changes the decider dataset hash. The platform drain then re-measures every pair through the CLI container and republishes the table. That is the normal pipeline. Its spend scales with pairs × cases × repetitions.
• To turn on the cost policy, set productionTokenProfile and switchHorizonTurns in the benchmark config (feat(auto-routing): rank by estimated production cost and switch only when it pays #7496). lowConfidenceThreshold stays off until production confidence data is reviewed.

Verification

• VM typecheck passes for contracts, auto-routing, auto-routing-benchmark and web.
• VM tests: contracts 110, auto-routing 259, auto-routing-benchmark 347. Lint is clean.
• The full sweep ran end to end, including a resume from a checkpoint after a restart: 39,600 case results, not truncated.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental diff keeps unknown-cost routes in --rebuild-from-sweep by letting those candidates compete at the route's maximum known request-cost estimate (zero when the whole route lacks costs), and rewrites the sweep tests to use error strings a real artifact persisted; both prior suggestions are resolved and no new issues were found in the changed lines.

Files Reviewed (4 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/src/scoreboard/build.test.ts
• services/auto-routing-benchmark/src/scoreboard/build.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts

Verified: the measured filter no longer drops avgCostUsd === null candidates, the imputed per-task cost reproduces estimateRequestCostUsd equal to the max known estimate (build.ts:78-112), held-out cost still excludes unknown-cost picks rather than imputing them (build.ts:187-206), and the new fixtures produce the asserted values (unknown pick at accuracy 1, benchmarkPerTaskUsdWeighted 0.29, estimatedRequestUsdWeighted 0.3). The removed parameterized error strings were not real persisted error values, so their removal is not a coverage loss.

Previous Review Summaries (2 snapshots, latest commit 47712fd)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 47712fd)

Status: 2 Issues Found | Recommendation: Merge with minor follow-ups

Executive Summary

The incremental fix correctly resolves all three prior findings (null grader scores, infra-error averaging, and re-ranked held-out picks); the only remaining items are two low-severity consistency/test-quality suggestions.

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	2

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/src/scoreboard/build.ts	70	Rebuild drops candidates/routes with unknown cost from every metric, asymmetric with the held-out path.
services/auto-routing-benchmark/src/scoreboard/sweep.test.ts	27	Fabricated legacy error string (and grader_failed, line 24) gives no real legacy regression coverage.

Previous findings re-verified

All three prior findings are fixed at 47712fde70 and dropped:

• Null grader scores now persist as score: null and are excluded by summarizeSweep (schema allows null; --resume/scoreboard build --sweep parse). Verified in schema.ts, sweep.ts, offline-sweep.ts.
• Errored/infra cases are excluded from graded cases and counted in errors; only empty answers score 0. Verified.
• Held-out evaluation now reads routes[route]?.[0], so it evaluates the re-ranked first pick. Verified.

Verification notes

Producer (offline-sweep.ts) emits a non-null score only with error in {null, usage_missing, OpenRouter response is missing answer content}, exactly the allow-list in summarizeSweep; cost accounting still adds model spend at most once per job across retries. No CRITICAL/WARNING issues found in changed lines.

Files Reviewed (7 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/scripts/offline-sweep.ts
• services/auto-routing-benchmark/src/scoreboard/build.ts - 1 issue
• services/auto-routing-benchmark/src/scoreboard/build.test.ts
• services/auto-routing-benchmark/src/scoreboard/schema.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts - 1 issue

Fix these issues in Kilo Cloud

Previous review (commit 48255d2)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

A grader failure writes a null result score that contradicts OfflineSweepSchema, silently producing a sweep file that --resume and scoreboard build --sweep cannot parse.

Overview

Severity	Count
CRITICAL	1
WARNING	0
SUGGESTION	2

Issue Details (click to expand)

CRITICAL

File	Line	Issue
services/auto-routing-benchmark/scripts/offline-sweep.ts	269	gradeDeciderOutput returns score: null on judge failure (or budget exhaustion in the judge wrapper), but the result schema requires a non-nullable number; the checkpoint is written with "score": null and the next --resume/scoreboard build --sweep fails to parse it.

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/scripts/offline-sweep.ts	51	summarize averages score over errored results (counted as 0), biasing route accuracy used by the rebuild ranking and held-out reporting.
services/auto-routing-benchmark/src/scoreboard/build.ts	150	With --rerank-profile (no rebuild), heldOut.* evaluates table.routes[0] while benchmark metrics evaluate the re-ranked routes[0], so the held-out pick can differ from the scoreboard's own first pick.

Files Reviewed (8 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/package.json
• services/auto-routing-benchmark/scripts/offline-sweep.ts - 2 issues
• services/auto-routing-benchmark/scripts/scoreboard.ts
• services/auto-routing-benchmark/src/scoreboard/build.test.ts
• services/auto-routing-benchmark/src/scoreboard/build.ts - 1 issue
• services/auto-routing-benchmark/src/scoreboard/compare.ts
• services/auto-routing-benchmark/src/scoreboard/schema.ts

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 67.5K · Output: 19K · Cached: 935.3K}

_{Review guidance: REVIEW.md from base branch igor/auto-efficient-quality}