feat(examples): train cost-aware routers on BBEH outcomes

examples/request-routing/README.md

@@ -1,4 +1,7 @@
-# Route requests without a generation call on every message
+# Route requests to handlers
+
+This example chooses a handler for a request. To train a new decision function
+from reviewed examples, use [Train a router](../train-a-router/README.md).
 
 For a single native classification call with your own handler labels, see
 the [direct routing example](./direct). The recorded cascade below uses a

examples/request-routing/cost/README.md

@@ -1,4 +1,9 @@
-# Send each request to a cheap or a strong LLM
+# Historical cost-routing study
+
+This is a separately installable study of cheap/strong LLM routing, with its own
+locked environment and archived answer costs. It is independent of the parent
+handler example and the new [Train a router](../../train-a-router/README.md)
+learning example. The figures below describe the dated recorded study.
 
 ## What this shows
 

examples/request-routing/score.py

@@ -329,7 +329,10 @@ def recorded_inputs(directory):
             require(
                 type(recorded["conf"]) in (int, float)
                 and math.isfinite(recorded["conf"])
-                and math.isclose(confidence, recorded["conf"], rel_tol=1e-6, abs_tol=1e-7),
+                # Float32 BLAS reductions differ across CPUs (e.g. Apple Accelerate
+                # versus OpenBLAS). Observed max absolute drift on the pinned 3,000
+                # cases is 3.8e-6. Route order and the threshold branch stay exact.
+                and math.isclose(confidence, recorded["conf"], rel_tol=1e-5, abs_tol=1e-6),
                 "recorded_front_confidence",
             )
             require((confidence >= THRESHOLD) == (recorded["conf"] >= THRESHOLD), "front_threshold")

examples/train-a-router/.gitignore

@@ -0,0 +1,21 @@
+# Local environments and generated experiment outputs.
+.venv/
+__pycache__/
+.pytest_cache/
+.ruff_cache/
+*.pyc
+data/
+my-router/
+published-results/
+my-results/
+answers/
+fresh-encoding/
+local-features/
+fresh-answers/
+fresh-grades/
+fresh-cpu-router/
+bbeh-source/
+calibration.html
+runs/
+reports/
+outputs/

examples/train-a-router/LABELS.md

@@ -0,0 +1,85 @@
+# Where the router’s labels come from
+
+This example labels **model outcomes**, not whether a question looks “easy” or “hard.” For every training question, Luna and Sol each produce an answer independently. A deterministic grader checks each answer against the BBEH reference. The resulting labels are:
+
+| Luna correct? | Sol correct? | What the router learns |
+|---|---|---|
+| Yes | Yes | The cheaper answer can be enough |
+| No | Yes | Paying for Sol can recover accuracy |
+| Yes | No | The stronger model is not always the better choice |
+| No | No | Paying more does not necessarily solve this question |
+
+Both correctness labels are retained. The model also learns expected costs from actual token usage. Neither router sees a reference, an answer, a correctness label, or the test cost when processing a new question.
+
+## If you have reference answers
+
+Use them to check real outputs from the two models you intend to serve. This is the validated route in this example. It avoids assuming that a teacher can predict which model will solve a question without seeing their answers.
+
+BBEH provides reference answers. The supplied recordings were collected with `gpt-6-luna` and `gpt-6.1-sol`, medium reasoning, a 16,384-output-token limit and the same question-only JSON answer contract. A format or quality failure is retained as a failure; it is not regenerated until it looks correct. Transport attempts are tracked separately, including unresolved cost reservations.
+
+The grader normalizes source-validated equivalent answer formats and preserves the raw upstream score too. For example, a unique option letter plus its matching text can be equivalent to the letter alone. It does not use the expected answer to repair a model’s response. The primary and source-format sensitivity results are both supplied.
+
+## If you do not have reference answers
+
+A teacher model can **judge the answers produced by both candidate models**. A useful protocol is:
+
+1. Collect representative questions and both candidate answers. Keep related questions together when splitting the data.
+2. Give a teacher the question, any authoritative context and each anonymized answer. Use a concrete correctness rubric and allow “uncertain.” Randomize answer order if asking for a comparison.
+3. Retain its two judgments, confidence/uncertainty and explanation. Avoid reducing every pair to a forced winner: both answers can be right or wrong.
+4. Check a separate sample with people, executable tests or authoritative references. Measure teacher errors and disagreements before treating its labels as truth.
+5. Train using the accepted labels; choose settings on development data and policies on calibration data. Keep an independently checked test set for the money–quality comparison.
+
+A frontier API model or a model served through SIE can be the teacher. Its usefulness depends on the task and the quality of its judgments, not only parameter count. If Sol itself judges Sol versus Luna, correlated errors and self-preference can distort the result; an independent judge and an audited sample are preferable.
+
+**Teacher-only labeling is not a measured result of this BBEH example.** We have not established that Qwen 27B, GLM Flash or another frontier model produces equivalent labels here. The current results should not be presented as evidence of that. Teacher labeling is a way to extend the workflow, with its own validation cost.
+
+## Collect fresh Sol/Luna answers
+
+The published recordings are enough for the complete training exercise. The following optional commands make a new paid collection with the same questions and settings. They keep the old recordings intact. Run them from this example folder.
+
+```bash
+# Enter your OpenAI key and inspect the training/development collection plan without making calls.
+export OPENAI_API_KEY='your-openai-api-key'
+uv run --frozen --group collection python router.py collect \
+  --data data --phase learn --output fresh-answers/learn --budget-usd 50
+```
+
+```bash
+# Make the planned training/development calls, retaining physical attempts and their costs.
+uv run --frozen --group collection python router.py collect \
+  --data data --phase learn --output fresh-answers/learn --budget-usd 50 --execute
+```
+
+```bash
+# Fetch the exact public benchmark source and grade the new answers against its references.
+uv run --frozen python -X utf8 source/fetch.py --output bbeh-source
+uv run --frozen --group collection python -X utf8 grade_v2.py \
+  --prepared data/source --collection fresh-answers/learn \
+  --official bbeh-source/bbeh/evaluate.py --pair pair.json --output fresh-grades/learn
+```
+
+The budget is a ceiling, not the expected bill. The collector reserves room for in-flight requests and unresolved executions, and stops rather than silently exceeding it. An identical plan can resume in the same output directory: completed responses are reused and unresolved attempts retain their reservations. Inspect the receipts first. A different plan cannot overwrite that run.
+
+Use `--phase calibration` for a separate calibration collection. Final collection additionally requires `--frozen PATH`, pointing to a policy bundle made before those final answers. When repeating the published experiment, `data/reference/frozen-frontiers.json` is the supplied unchanged bundle:
+
+```bash
+# Plan a fresh final-answer repeat for the already frozen reference policies; add --execute to dispatch it.
+uv run --frozen --group collection python router.py collect \
+  --data data --phase final --output fresh-answers/final --budget-usd 25 \
+  --frozen data/reference/frozen-frontiers.json
+```
+
+These commands create new answer and grade directories; they do not replace the downloaded evidence or silently change a prepared training run. For example, train a separate CPU embedding router directly from the newly graded answers:
+
+```bash
+# Prepare a new CPU study and fit neighbors/classifiers using the fresh training and development labels.
+uv run --frozen python -X utf8 embedding/recipes.py prepare --source data/source --pair pair.json --run fresh-cpu-router
+uv run --frozen --group outcomes python -X utf8 embedding/recipes.py fit \
+  --run fresh-cpu-router --features data/features \
+  --train-grades fresh-grades/learn/grades-train.json \
+  --development-grades fresh-grades/learn/grades-development.json
+```
+
+This lower-level study is separate from the README’s published-data reproduction. Direct scripts use `python -X utf8` so multilingual inputs and JSON files use UTF-8 regardless of the host locale; `router.py` sets this mode automatically, including for its child processes. `native/bbeh_native.py` also accepts explicit grade paths; its prepared plan must bind the new acquisition protocol. Preserve source, answer-pair, grading and acquisition bindings together; changing models or prompts is a new experiment.
+
+For production feedback, collect outcomes on a representative sample of **both** routes. Labeling only the arm chosen by the current router hides the alternative’s performance and can reinforce its mistakes. New labels can update the neighbor collection, refit the small embedding heads or support another LoRA adaptation. Re-evaluate before replacing a serving policy.

examples/train-a-router/QUICKSTART.md

@@ -0,0 +1,64 @@
+# Train a cost-aware router
+
+**Goal:** send requests to Luna when it is sufficient and pay for Sol when its extra accuracy is worthwhile. The supplied BBEH experiment compares unchanged Laya → LoRA adaptation → embedding neighbors → a small embedding classifier, with an optional learned cost estimate. On 975 held-out questions, an embedding classifier with learned cost estimates saved 16.7% with a 0.9-point accuracy loss; another preselected policy saved 39.7% with a 6.1-point loss. Both savings figures include the Cloud encoder fee.
+
+**You need:** Python 3.12 and `uv`; an NVIDIA GPU for the full LoRA exercise (validated on H100 80 GB). Replay and classifier training run on CPU. SIE Cloud can supply Qwen3-Embedding-4B vectors; OpenAI supplies live Sol/Luna answers. Cached vectors and answers are included, so replay needs neither API key.
+
+**1. Set up and inspect the evidence.** Training/development/calibration/test contain 1,969/589/385/975 questions. Related question families stay in one split. Accuracy measures the chosen answer; savings compare its cost with always-Sol. An accuracy change from 80% to 78% is a 2-point loss. Step numbers match the [full walkthrough](README.md).
+
+```bash
+# From your repository clone, install dependencies and download the pinned data and decision-model weights.
+cd examples/train-a-router
+uv sync --frozen --group training --group outcomes
+uv run --frozen python router.py fetch --output data --group core --group weights
+uv run --frozen python router.py replay --data data --output published-results
+uv run --frozen python router.py prepare --data data --run my-router
+```
+
+Open `published-results/explorer/index.html` for the full cost–quality curves.
+
+For the **CPU embedding exercise**, install only `--group outcomes`, fetch only `--group core`, and run the following after `prepare`. Then continue at steps 4–5. This uses recorded Laya comparisons; it does not perform LoRA training.
+
+```bash
+# Supply the saved Laya predictions needed for comparison during an embedding-only refit.
+uv run --frozen python router.py reuse-native --data data --run my-router
+```
+
+**2–3. Try and adapt Laya on GPU.** Start with unchanged weights, then let development data choose the LoRA training duration. On H100, comparing eight training passes took about **51 minutes**; the selected three-pass refit and checkpoint export took another **26 minutes**. Allow **77 minutes total**, plus setup and scoring time.
+
+```bash
+# Score unchanged Laya, train LoRA, and save input-only predictions for both versions.
+uv run --frozen --group training python router.py base --data data --run my-router --device cuda
+uv run --frozen --group training python router.py fit-native --data data --run my-router
+uv run --frozen --group training python router.py score --data data --run my-router --method native --device cuda
+```
+
+**4–5. Fit embedding routers on CPU.** The large Qwen encoder stays fixed. Neighbors average outcomes of similar questions; two small classifiers learn whether Luna and Sol will answer correctly.
+
+```bash
+# Fit neighbors, classifiers and simple controls, then score held-out question embeddings.
+uv run --frozen --group outcomes python router.py fit-embedding --data data --run my-router
+uv run --frozen --group outcomes python router.py score --data data --run my-router --method embedding
+```
+
+**6. Estimate cost and choose your tradeoff.** Cost varies by question, so retain both the original and learned cost estimates.
+
+```bash
+# Fit cost predictors, freeze the calibration choices, and open the calibration explorer.
+uv run --frozen --group outcomes python router.py fit-cost --data data --run my-router
+uv run --frozen python router.py calibrate --data data --run my-router
+uv run --frozen python router.py frontier --run my-router --output calibration.html
+```
+
+Open `calibration.html` and record a **method and its candidate ID together** before viewing test results. For the headline approach, choose `embedding__textcost`; candidate IDs are specific to a method and run. There is no required quality-loss threshold.
+
+```bash
+# After selecting your calibration policy, evaluate the frozen choices against the published test answers.
+uv run --frozen python router.py evaluate-refit --data data --run my-router --output my-results
+```
+
+Open `my-results/explorer/index.html` to assess your choice. This is a reproduction using published test answers, not a new blind evaluation.
+
+**7. Try a live request.** Add a SIE key for an embedding route and an OpenAI key for the selected answer. Run `router.py answer` with both the method and candidate copied from calibration for the same run. [The live command](README.md#7-route-a-new-question) uses `embedding__textcost`. Laya routes need no embedding call; after the CPU shortcut, use the supplied `data/reference` run and weights for live Laya inference.
+
+To regenerate vectors, use [SIE Cloud](README.md#use-sie-cloud-or-your-own-encoder) or [your own SIE server](self-host/README.md). See [labels and collection](LABELS.md) for a new paired answer run. The [README](README.md) includes measured latency, hardware requirements, uncertainty and the effect of router fees.