[iscekic]

bot: Adds two ranking inputs and a candidate screen: effort from classifier complexity, a time-to-first-token (TTFT) term in the value formula, and a report of promising new catalog models. This PR is stacked on #7517.

Change

• Effort from complexity:
  • The routing table can carry effortByComplexity: for each route and model, the variant to use per reasoningComplexity bucket.
  • After the model is picked, the engine swaps only the variant, and only to a pair that passes every filter. The same model keeps its prompt cache.
  • Sticky-immediate turns keep the incumbent's variant.
  • The builder learns each bucket's choice from per-case D1 results, joined to src/datasets/case-complexity.json. The production classifier generated that file. A bucket with fewer than 6 graded cases uses the route-level pick.
• TTFT ranking:
  • When TTFT is known, the latency term is p50TtftMs + LATENCY_TOTAL_WEIGHT × p50LatencyMs, with a weight of 0.15.
  • Candidates publish p50TtftMs, and the policy is value-ttft-prodcost-wilson-v4.
• Candidate screening: screen:candidates samples cases per route for catalog models that are not yet benchmarked. It reports their value against the current first picks. It is a report only and never changes config.
• A D1 migration adds routing_tables.effort_by_complexity.

Held-out result (fully streamed sweep, 44 platform pairs, 2 repetitions)

Rebuilt picks against the live table's picks, weighted by the production route mix:

Policy	Accuracy	Estimated cost	p50 latency	p50 TTFT
#7510/#7517 policy (no TTFT term)	+14.4%	−37.7%	−28.1%	+11.1%
This PR, weight 0.25	+14.9%	−37.7%	−28.3%	+8.4%
This PR, weight 0.15 (committed)	+17.1%	−32.2%	−4.7%	−7.4%

• At 0.15 every axis improves over the live table. Weights between 0.15 and 0.18 give the same picks; at 0.2 the picks flip and TTFT gets worse than live, so 0.15 sits in the middle of the stable range.
• Caveat: I tuned this one value on held-out data, which adds a small selection bias. The shadow table (feat(auto-routing): shadow routing table telemetry #7515) can confirm it on production traffic before rollout.
• Effort from complexity adds +0.5% accuracy at about the same cost. Most buckets do not reach the minimum of 6 graded cases, so they fall back to the route-level pick.

Verification

• VM: contracts 115, auto-routing 289, auto-routing-benchmark 468 tests pass. Typecheck passes, and type-aware lint is clean.
• The production classifier labeled all decider cases with no errors after fix(auto-routing): keep classifier prompt prefixes on Unicode boundaries #7516.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental review of 441b639..3578284 (6 files) found no new issues: the concurrent screen runtime, shared budget reservations, resume reconciliation, and grader reserve logic are internally consistent and match the updated README.

Files Reviewed (6 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/scripts/screen-candidates.ts
• services/auto-routing-benchmark/src/scoreboard/screen-runtime.ts
• services/auto-routing-benchmark/src/scoreboard/screen-runtime.test.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.ts
• services/auto-routing-benchmark/src/screen-candidates.test.ts

Verification notes

• createScreenBudget.reserve admits only when spent + reserved + amount <= maxUsd, rejects requests larger than the remaining budget when nothing is outstanding, and waits otherwise; every settle decrements once and clears reservedUsd when the last reservation settles, so no deadlock or unbounded waiter growth.
• runScreenWorkerPool drains in-flight jobs after a worker throws, then rethrows the first rejection, so the finally checkpoint still runs.
• screenCandidateJobs yields each (pair, case) exactly once; models with fewer variants skip out-of-range rounds via the if (benchCase) guard.
• restoreScreenMeasurements enforces a bijection between resumed pairs and current measurements, rejects duplicates/incompatible results, and restores completed cases and costUnknown exclusions before the pool runs.
• recordScreenModelCost fallback and the pre-request reservation use the same prompt/completion token estimate (max_tokens or 8192), so they cannot diverge.
• sweep.ts beforeRequest/onAttemptComplete are optional and preserve the prior return shape; error ??= 'Sweep budget exhausted before request' still short-circuits the first-attempt case in main while preserving a real retry error.
• Not re-raised: the routing-table-builder.ts:143 TTFT WARNING is outside this increment and was dispositioned by the author as an intended offline-only boundary; the decision-engine, label-case-complexity, and screen-candidates findings are author-fixed.

Previous Review Summaries (4 snapshots, latest commit 441b639)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 441b639)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental review of 6c03fab..441b639 (4 files) found no issues: isolating unknown-cost screening models while charging catalog token prices, the added tokenUsage plumbing, and the budget-reservation fallback are internally consistent and match the updated README.

Files Reviewed (4 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/scripts/screen-candidates.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.ts
• services/auto-routing-benchmark/src/screen-candidates.test.ts

Verification notes

• recordScreenModelCost marks costUnknown only when a response has content or a transport/body-read failure, charges promptTokens × promptUsdPerToken + completionTokens × completionUsdPerToken when both counts are present, and reserves max(0, remainingUsd) otherwise; the main loop then skips that model's remaining pairs (screen-candidates.ts:472) so the estimate is applied at most once per model.
• Unknown-cost models cannot be recommended twice over: their results carry score: null (grading is skipped at screen-candidates.ts:499) so pricedCases !== cases, and recommend is additionally gated with !spend.costUnknown (screen-candidates.ts:452).
• readSweepCompletion returns tokenUsage only when present, so the existing exact-shape toEqual assertions in sweep.test.ts are unaffected; the extended usage zod schema strips unknown keys and is not a breaking change.
• The two new tests reproduce the isolate-and-charge path and the reserve-remaining-budget fallback with correct arithmetic (0.0012, 0.0032, 0.25).
• The current routing-table-builder.ts:143 TTFT WARNING is on a file outside this increment and was already dispositioned by the author as an intended offline-only boundary; the decision-engine, label-case-complexity, and screen-candidates findings are now outdated and author-fixed.

Previous review (commit 6c03fab)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental review of c8c8f28..6c03fab (2 files) found no issues: lowering LATENCY_TOTAL_WEIGHT from 0.25 to 0.15 is a self-consistent tuned-constant change documented in the PR body, the README matches, and -v4 is new in this PR so no stale published table results.

Files Reviewed (2 files)

• packages/auto-routing-contracts/src/routing-table.ts
• services/auto-routing-benchmark/README.md

Verification notes

• LATENCY_TOTAL_WEIGHT is exported and consumed only in rankValue (routing-table.ts:226); the TTFT branch is the sole consumer, and no hardcoded 0.25 weight reference remains anywhere in the contracts or benchmark packages.
• No test asserts a specific value for the weight, so the retune does not break the suite.
• RANKING_POLICY_VERSION stays value-ttft-prodcost-wilson-v4; the base branch used -v3, so v4 is introduced by this PR and no already-published table is left stale by retuning the weight within v4.
• README text was updated in lockstep with the constant.
• The open TTFT-on-publish-path WARNING on routing-table-builder.ts:143 is unchanged by this increment and was already dispositioned by the author as an intended production boundary (offline sweep/screen only).

Previous review (commit c8c8f28)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental review of b025cc9..c8c8f28 (11 files) found no new issues: the switch-engine comparisons, the SSE reader cleanup, and the labeling atomic-publish fix are correct, and the earlier TTFT concern is now documented as an intended production boundary.

Files Reviewed (11 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/scripts/label-case-complexity.ts
• services/auto-routing-benchmark/scripts/screen-candidates.ts
• services/auto-routing-benchmark/src/datasets/decider-cases.test.ts
• services/auto-routing-benchmark/src/datasets/frontier-cases/refactoring.ts
• services/auto-routing-benchmark/src/label-case-complexity.test.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.test.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.ts
• services/auto-routing-benchmark/src/screen-candidates.test.ts
• services/auto-routing/src/decision-engine.test.ts
• services/auto-routing/src/decision-engine.ts

Verification notes

• decision-engine.ts: picked/incumbentPick are now used for the cost-horizon and best_accuracy comparisons, matching the variants actually served; applyComplexityEffort cannot change the model, so model: picked.model is equivalent to freshPick.model. The eligibility gate still reads the raw incumbentCandidate, but that call is unchanged context and predates this PR.
• scoreboard/sweep.ts: the unconditional reader.cancel() is safe on a closed stream, !done gating only drops post-[DONE] data, and chunk.error != null only changes behavior for an explicit null (the intended fix).
• label-case-complexity.ts: failure checkpoints and the final publish are atomic temp-file renames; the committed dataset is no longer clobbered, and the README matches the new --resume contract.
• screen-candidates.ts: screenRecommendation now enforces the documented per-variant "beats two distinct routes" rule.
• frontier-cases/refactoring.ts: ['\ud834','\udf06'] is the correct UTF-16 split of U+1D306 and matches the new assertion.

Previous review (commit b025cc9)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

The new TTFT ranking term is inert on the publish path because model_summaries never carries p50TtftMs, so the -v4 policy bump republishes with unchanged ordering; the complexity-effort remap also feeds stale switch inputs in the decision engine.

Overview

Severity	Count
CRITICAL	0
WARNING	3
SUGGESTION	1

Issue Details (click to expand)

WARNING

File	Line	Issue
services/auto-routing-benchmark/src/routing-table-builder.ts	143	p50TtftMs: s.p50TtftMs is always undefined, so the TTFT ranking term and the -v4 policy bump do not change published ordering
services/auto-routing/src/decision-engine.ts	381	Switch-worthiness is evaluated against the unmapped freshPick while a remapped (possibly costlier/less accurate) variant is served
services/auto-routing-benchmark/scripts/label-case-complexity.ts	55	Failure checkpoint writes partial labels non-atomically over the committed case-complexity.json

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/scripts/screen-candidates.ts	398	recommend unions beaten routes across variants instead of applying the documented per-variant two-route threshold

Verification notes

• Confirmed mapSummaryRow (src/db.ts) omits p50TtftMs and model_summaries has no TTFT column, so production summaries built via getSummariesForRuns never populate it.
• Confirmed applyComplexityEffort can return a different variant than freshPick while the switch conditions still read freshPick.

Files Reviewed (35 files)

• packages/auto-routing-contracts/src/benchmark.ts
• packages/auto-routing-contracts/src/classifier/system-one-classifier.test.ts
• packages/auto-routing-contracts/src/classifier/system-one-classifier.ts - clean (surrogate handling correct)
• packages/auto-routing-contracts/src/routing-table.test.ts
• packages/auto-routing-contracts/src/routing-table.ts
• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/migrations/0015_brief_monster_badoon.sql
• services/auto-routing-benchmark/migrations/meta/0015_snapshot.json
• services/auto-routing-benchmark/migrations/meta/_journal.json
• services/auto-routing-benchmark/package.json
• services/auto-routing-benchmark/scripts/label-case-complexity.ts - 1 issue
• services/auto-routing-benchmark/scripts/offline-sweep.ts
• services/auto-routing-benchmark/scripts/scoreboard.ts
• services/auto-routing-benchmark/scripts/screen-candidates.ts - 1 issue
• services/auto-routing-benchmark/src/admin.test.ts
• services/auto-routing-benchmark/src/auto-decider-sync.test.ts
• services/auto-routing-benchmark/src/custom-routing-table.test.ts
• services/auto-routing-benchmark/src/custom-routing-table.ts
• services/auto-routing-benchmark/src/datasets/case-complexity.json
• services/auto-routing-benchmark/src/db-schema.ts
• services/auto-routing-benchmark/src/db.test.ts
• services/auto-routing-benchmark/src/db.ts
• services/auto-routing-benchmark/src/profile-runs.test.ts
• services/auto-routing-benchmark/src/routing-table-builder.test.ts
• services/auto-routing-benchmark/src/routing-table-builder.ts - 1 issue
• services/auto-routing-benchmark/src/run-process-job.test.ts
• services/auto-routing-benchmark/src/run.ts
• services/auto-routing-benchmark/src/scoreboard/build.test.ts
• services/auto-routing-benchmark/src/scoreboard/build.ts
• services/auto-routing-benchmark/src/scoreboard/schema.ts
• services/auto-routing-benchmark/src/scoreboard/sweep.ts
• services/auto-routing-benchmark/src/screen-candidates.test.ts
• services/auto-routing/src/decide.ts
• services/auto-routing/src/decision-engine.test.ts
• services/auto-routing/src/decision-engine.ts - 1 issue

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 64.2K · Output: 29.7K · Cached: 811.1K}

_{Review guidance: REVIEW.md from base branch igor/auto-efficient-sweep-v2}