feat(auto-routing): scoreboard for Auto Efficient latency, cost and quality

[iscekic]

bot: Add one repeatable scoreboard for kilo-auto/efficient on three axes: router latency, cost and quality. This PR does not change routing. The next PRs in this series use the scoreboard to prove their changes.

Change

• /decide logs timings for each phase in the auto_routing_decision event: coding plan, settings, capabilities, table, preparation, cache read, classifier and total.
• The gateway logs auto_routing_decide_client with the time to /decide from the gateway side. It samples 1% of successful calls and logs every failure.
• pnpm --filter auto-routing-benchmark scoreboard:prod collects production aggregates from Axiom and the read replica. It reads credentials only from the environment.
• pnpm --filter auto-routing replay sends cold, new-turn and exact-repeat sessions to a running worker and writes the latency distributions.
• pnpm --filter auto-routing-benchmark scoreboard build|compare joins the live routing table, the classifier winner, the production metrics and the replay. compare --relative-only prints deltas without absolute values, for PR bodies.

Findings from the first baseline

• Model time to first token is about an order of magnitude larger than the router overhead. Latency work must cover model choice, not only /decide.
• Most /decide requests are classification cache hits. On the hit path, round trips to the Durable Object and KV take most of the time.
• The routing-table accuracy weighted by route mix is close to 1.0. The golden decider cases no longer separate the candidates, so quality needs harder cases.
• The classifier costs a negligible share of the cost per request.

Verification

• VM typecheck passes in auto-routing, auto-routing-benchmark and web-shared.
• VM tests: auto-routing 207 passed. auto-routing-benchmark 299 passed, plus 7 new scoreboard tests. The gateway decision client passes 14 Jest tests.
• Lint passes in both worker packages.
• A real scoreboard:prod run wrote every non-timing field. The timing fields stay null until this PR deploys. A fallback gives the baseline worker overhead.
• A real replay against the local worker on a VM ran 108 requests with 0 errors. The worker log carried all 8 timings keys.
• scoreboard build --table live loaded the published table. It loaded candidate latency from the registry provenance and the classifier route accuracy.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental commit (51f9cbd) fixes the previously reported estimator bias and I found no new correctness, security, or reliability issues; the new sampling weight matches the worker's logging semantics in decide.ts.

Files Reviewed (4 files)

• services/auto-routing-benchmark/scripts/scoreboard-prod.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/production.ts - 0 issues (prior totalClassified finding fixed)
• services/auto-routing/src/classifier-config.ts - 0 issues

Verification Notes

• estimateClassifiedMetrics now weights each sampled: true row by 1 / sampleRate and each always-logged (sampled: false) row by 1, so the reported route mix, cache-hit rate and classifier cost are reconstructed over the full classified population without depending on the /decide request count. This matches decide.ts:289-291 (successes logged with probability successSampleRate) and decide.ts:333 (sampled: !alwaysLog).
• The script reads the same KV key (decision_log_sample_rate) the worker uses via getDecisionLogSampleRate, with a warned fallback to the shared exported default and a --sample-rate override.
• Unit tests cover uniform weighting, no-sampled rows, null cache/cost exclusion, all-successes-sampled (rate 1), disabled sampling (rate 0, no division by zero), and the empty-population case.

Previous Review Summaries (2 snapshots, latest commit 9c17577)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 9c17577)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The review-fix commit resolves all five prior findings; the one new issue is that the classified-population estimate in production.ts treats non-decision /decide requests (400s and coding-plan early returns) as classified, inflating sampledWeight and biasing the weighted route mix toward sampled rows.

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/src/scoreboard/production.ts	20	totalClassified = requestCount - unsampledNonClassified counts /decide requests that never emit a decision log (invalid_json, invalid_envelope, coding_plan_default), inflating sampledWeight and biasing routeMix, cacheHitRate and classifierPerRequestUsd toward the sampled (non-switched) subpopulation.

Files Reviewed (6 files)

• packages/web-shared/src/lib/ai-gateway/auto-routing-decision.ts - 0 issues (prior invalid_response finding fixed)
• services/auto-routing-benchmark/scripts/scoreboard-prod.ts - 0 issues (prior kilo-auto/balanced and sampled == true findings fixed)
• services/auto-routing-benchmark/src/scoreboard/build.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.ts - 0 issues (prior empty-routeMix finding fixed)
• services/auto-routing-benchmark/src/scoreboard/compare.ts - 0 issues (prior replay-errors axis finding fixed)
• services/auto-routing-benchmark/src/scoreboard/production.ts - 1 issue

Fix these issues in Kilo Cloud

Previous review (commit 7e51b68)

Status: 5 Issues Found | Recommendation: Address before merge

Executive Summary

The scoreboard's production pipeline can silently emit a fully-null weighted benchmark set when the observed route mix is empty, and its Postgres and Axiom halves measure different request populations; the remaining findings are lower-severity observability and classification nits.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	3

Issue Details (click to expand)

WARNING

File	Line	Issue
services/auto-routing-benchmark/src/scoreboard/build.ts	22	Empty production routeMix ({}) is treated as a real mix, so all weighted benchmark metrics silently become null; reachable when the Axiom worker query fails.
services/auto-routing-benchmark/scripts/scoreboard-prod.ts	220	Postgres metrics filter on auto_model = 'kilo-auto/efficient' while the Axiom /decide metrics include the kilo-auto/balanced alias, so the two halves of the scoreboard measure different populations.

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/scripts/scoreboard-prod.ts	10	j.sampled == true excludes always-logged switch decisions from routeMix, cacheHitRate and the latency/cost aggregates.
services/auto-routing-benchmark/src/scoreboard/compare.ts	50	Replay errors is registered on the latency axis, so a reliability-only change summarizes as latency: worse.
packages/web-shared/src/lib/ai-gateway/auto-routing-decision.ts	121	A schema-invalid 200 response is logged as outcome: 'error' with status: 200, conflating validation failures with transport errors.

Files Reviewed (12 files)

• packages/web-shared/src/lib/ai-gateway/auto-routing-decision.ts - 1 issue
• services/auto-routing-benchmark/README.md - 0 issues
• services/auto-routing-benchmark/package.json - 0 issues
• services/auto-routing-benchmark/scripts/scoreboard-prod.ts - 2 issues
• services/auto-routing-benchmark/scripts/scoreboard.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.test.ts - 0 issues
• services/auto-routing-benchmark/src/scoreboard/build.ts - 1 issue
• services/auto-routing-benchmark/src/scoreboard/compare.ts - 1 issue
• services/auto-routing-benchmark/src/scoreboard/schema.ts - 0 issues
• services/auto-routing/package.json - 0 issues
• services/auto-routing/scripts/decide-replay.ts - 0 issues
• services/auto-routing/src/decide.ts - 0 issues

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 47.6K · Output: 19.6K · Cached: 880.4K}

_{Review guidance: REVIEW.md from base branch main}