Skip to content

feat(auto-routing): benchmark-routed kilo-auto/frontier behind a flag - #7519

Merged
iscekic merged 105 commits into
mainfrom
igor/auto-frontier-router
Oct 14, 2026
Merged

iscekic merged 105 commits into
mainfrom
igor/auto-frontier-router

Conversation

@iscekic

@iscekic iscekic commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

bot: Put kilo-auto/frontier on the benchmark pipeline: the same router as efficient, with a frontier-only pool, its own harder tests and its own routing table. It ships behind a flag that defaults to off. This PR is stacked on #7518.

Change

  • AutoRouter = 'efficient' | 'frontier'. /decide takes an optional router, which defaults to efficient.
  • Isolated per router:
    • routing table (routing_table_v1 stays efficient; routing_table_frontier_v1 is new);
    • benchmark config, registry, runs and publish;
    • engine identity, which includes the router and its dataset hash;
    • sticky, classification and classifier-usage state.
  • Shared: the classifier, billing, privacy and denied-model handling. Frontier ignores owner pools and coding-plan default shortcuts.
  • Frontier benchmarks run FRONTIER_CASES (feat(auto-routing): sweep v2 with TTFT, long-context and frontier cases #7517).
  • The cron and daily sync drain both routers. Concurrency budgets are per router, inside the shared container cap.
  • Gateway: with the PostHog flag kilo-auto-frontier-routing on, kilo-auto/frontier asks /decide for a frontier decision. When the flag is off, the decision is null or the table is missing, the static FRONTIER_MODE_TO_MODEL mapping applies unchanged.
  • Admin: the benchmarks page has a router selector.
  • D1 migration 0016: adds the router columns. I hand-edited it: Drizzle's SQLite table rebuilds selected a router column that the old tables do not have, which would fail on deploy. They now select the literal 'efficient'.

Verification

  • VM: contracts 119, auto-routing 303, auto-routing-benchmark 513 tests pass. Web gateway and admin Jest pass 239 tests. Typecheck passes, and type-aware lint is clean.
  • Migration test: applied every earlier migration to a fresh SQLite database, seeded config, decider and profile rows, then applied 0016. All rows survive with router = 'efficient'.

Frontier sweep result (16 model/effort pairs across 8 frontier models, held-out frontier cases, agentic excluded)

Baseline: today's static mapping, with Sonnet at medium effort for code-style modes and Opus at medium for planning and asking. It is compared with picks rebuilt from the train split by this router's policy:

Metric Change against the static mapping
Accuracy +49%
Accuracy per dollar +24%
Estimated cost per request +20%
p50 latency +38%
p50 TTFT +22%
  • The frontier cases separate frontier models: the static mapping scores about 0.56 on them.
  • The policy optimizes quality per dollar inside the frontier pool, so it pays somewhat more for much higher accuracy. A cost cap is the lever if latency or spend must stay flat.
  • Keep the flag off until the platform drain measures the frontier pool through the CLI container (agentic routes included) and publishes routing_table_frontier_v1.

# Conflicts:
#	services/auto-routing-benchmark/src/scoreboard/build.test.ts
#	services/auto-routing-benchmark/src/scoreboard/build.ts
…eculative

# Conflicts:
#	services/auto-routing-benchmark/src/decider-judge.ts
@iscekic
iscekic added this pull request to stack #7532 October 14, 2026 11:57
…weep-v2

# Conflicts:
#	services/auto-routing-benchmark/README.md
#	services/auto-routing-benchmark/scripts/offline-sweep.ts
#	services/auto-routing-benchmark/src/datasets/decider-cases.test.ts
#	services/auto-routing-benchmark/src/datasets/decider-cases.ts
#	services/auto-routing-benchmark/src/scoreboard/build.test.ts
#	services/auto-routing-benchmark/src/scoreboard/build.ts
#	services/auto-routing-benchmark/src/scoreboard/compare.ts
#	services/auto-routing-benchmark/src/scoreboard/schema.ts
#	services/auto-routing-benchmark/src/scoreboard/sweep.test.ts
#	services/auto-routing-benchmark/src/scoreboard/sweep.ts
…router

# Conflicts:
#	services/auto-routing/src/classifier-usage.test.ts
#	services/auto-routing/src/decision-cache.ts
…olicy-v3

# Conflicts:
#	services/auto-routing-benchmark/README.md
#	services/auto-routing-benchmark/scripts/offline-sweep.ts
#	services/auto-routing-benchmark/src/scoreboard/sweep.ts
…router

# Conflicts:
#	services/auto-routing/src/decide.ts
#	services/auto-routing/src/model-capabilities.ts
Base automatically changed from igor/auto-efficient-policy-v3 to main October 14, 2026 21:40
…uter

# Conflicts:
#	packages/auto-routing-contracts/src/routing-table.test.ts
#	services/auto-routing-benchmark/README.md
#	services/auto-routing-benchmark/migrations/meta/_journal.json
#	services/auto-routing-benchmark/scripts/scoreboard.ts
#	services/auto-routing-benchmark/src/routing-table-builder.test.ts
#	services/auto-routing-benchmark/src/routing-table-builder.ts
#	services/auto-routing-benchmark/src/run.ts
#	services/auto-routing-benchmark/src/scoreboard/build.test.ts
#	services/auto-routing-benchmark/src/scoreboard/build.ts
#	services/auto-routing-benchmark/src/scoreboard/schema.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants