[svonava]

Adds a self-contained examples/train-a-router workflow for learning when a cheaper answer model is sufficient and when a stronger model is worth its cost. On the family-separated BBEH final test, GPT-6.1 Sol reaches 80.8% accuracy and GPT-6 Luna 55.6%, giving the router a substantial correctness gap to learn.

The walkthrough establishes always-model baselines, scores unchanged Laya, adapts it with LoRA, compares embedding neighbors and two small logistic classifiers, learns request-specific answer costs, and routes a new question. The example includes a one-page guide, explicit GPU/CPU requirements, SIE Cloud and self-hosted encoder paths, answer collection/grading, and an offline interactive cost–quality report. No GEPA or earlier example assets are required.

Representative results from policies selected on calibration before final-answer collection:

Policy	Final accuracy	Loss vs Sol	Answer cost / 10k	Answer-cost savings
Always Sol	80.8%	0.0 points	$70.77	0.0%
Always Luna	55.6%	25.2 points	$12.23	82.7%
Unchanged Laya	68.6%	12.2 points	$41.82	40.9%
Laya + LoRA	78.3%	2.6 points	$56.80	19.7%
Embedding + learned cost, 80% calibration cost cap	79.9%	0.9 points	$58.16	17.8%
Embedding + learned cost, 60% calibration cost cap	74.8%	6.1 points	$41.89	40.8%
Text-predicted task + learned cost	80.1%	0.7 points	$58.31	17.6%

Rows use the 80% calibration answer-cost cap unless specified. It is an illustrative budget, not an accuracy-loss ceiling or a guarantee of final savings. The complete curves retain all 14 method/cost variants and simple controls. The Cloud encoder adds $0.81 per 10,000 requests, making the two embedding totals $58.97 / 16.7% savings and $42.70 / 39.7% savings. Application/native hosting and one-time training remain separate. Competitive text controls prevent a claim that a larger routing model is universally better.

The 3,918 retained questions are split 1,969 training / 589 development / 385 calibration / 975 final. Identified source/context families stay together. Development selects LoRA duration, neighbor count, classifier regularization and cost-model settings; selected models are refitted from scratch on training + development. All 8,450 policy rules and 56 presentation choices were frozen before final answers were collected. The 0.9-point embedding accuracy-loss estimate has a paired 95% interval of −0.3 to 2.1 points, not a guaranteed sub-one-point loss. The predeclared 973-question source-format sensitivity preserves the policies and barely changes the headline accuracies.

Large artifacts are public on Hugging Face at immutable revision 86dd7494a4c2396bc9ffa890f994b9722610b357: four standard question/outcome tables, all 7,836 selected model answers, embeddings, fitted heads, original/adapted Laya weights, full reports and provenance. The manifest SHA is 4fd8a49d760384ad745c1bd49b3ee61cb58fd8d98c71cef6552ff6dbfda66b55. Download/replay requires no API keys. SIE credits cover fresh embeddings; live Sol/Luna answers use OpenAI credits. Teacher-only labels and a locally served alternative answer pair are explicitly unvalidated extensions.

Validation completed:

• Exact 32-file scientific source closure, artifact hashes and policy identities verified; independent ML/leakage and numerical audits passed.

• 44 example/source tests, Ruff and whitespace checks passed. A clean Python 3.12 Modal install of all dependency groups and original Laya CPU scoring passed earlier in this preparation.

• H100 LoRA selection, fresh refit, export/reload parity, and all held-out scores completed. Three development-selected passes avoid the deterioration observed with longer training.

• Shared 90-question measurements: resident LoRA averages 78 ms on H100 and 3.98 s on 8-core CPU; the Cloud Qwen encoder averages 204 ms, with roughly 0.05 ms for the classifier or 2.4 ms including learned cost prediction. Timing scopes and separate busy-resource cost projections are documented.

• Final clean Python 3.12 replay at commit 6b2602ba0ee3a00099bbbaff5832c0ae3b5bab2a passed: anonymous public artifact download, offline result recomputation, fresh CPU embedding/control/cost fits, calibration and evaluation. All 14 methods / 8,450 policy results match exactly; this replay reused the previously completed H100 native scores. No model API calls or offline network attempts occurred. The complete worker took 353 seconds including downloads and report checks.

• All three interactive reports passed isolated Chromium checks for method selection, cost views, uncertainty, search, pagination and the metadata-only control. Six screenshots were visually reviewed; no page/console errors or external requests occurred.

• The public live answer command passed for both embedding and LoRA routers. One production SIE embedding call and two selected Sol answer calls returned the correct answer, with one physical attempt each. Dependency installation included all public groups; no public source files changed during validation.

The self-host setup now pulls the public SIE v0.10.1 CUDA default image by digest and selects the Flash adapter for the same pinned 4B checkpoint. It requires no source build. Fresh vectors are used for a fresh fit and calibration; supplied vectors remain the exact scientific replay path.

At commit 87c7cd23edf3233b02d4b1cafee58c9e56f6b46a, the exact image, profile, server arguments, health check and smoke passed on Modal H100 as UID/GID 1000 with an empty checkpoint cache. All 589 development vectors matched the measured source Flash vectors exactly. Docker Compose orchestration and host port mapping were not run on GPU; the Compose configuration passed static validation. Four focused encoder tests, Ruff and whitespace checks passed.

A separate matched H100 comparison completed 6,624 timed requests without failures, with three primary trials per adapter. It supports retaining SGLang as the catalog default: Flash is an available published-image option, not a general speed improvement. The differing workload/cache histories do not establish a memory winner. All 15 numerical and frozen-policy comparison pairs replayed independently; all 20 public evidence files passed anonymous download and hash verification. Pinned adapter measurements and release-image receipts.

Repository CI at the scientific-replay commit 6b2602ba completed on 2026-10-22: Python offline unit tests and all other checks passed except inherited gateway-deny Rust advisories RUSTSEC-2026-0381/0382/0383 (which also fail the aggregate Required check). That tested base and PR head have identical Cargo.lock bytes. This example does not modify those dependencies; the separate repair is PR #843. Prior CI run. Repository-wide checks must also complete on the new self-host update 87c7cd23; the earlier CI result is not a result for that new head.

Walkthrough · One-page guide · Pinned artifacts

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}