Skip to content

docs: record run-088 Urdu stage-1 results in WO11 #407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Nov 3, 2026
Merged

docs: record run-088 Urdu stage-1 results in WO11 #407

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions TODO.all-sys/11-urd-oracle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,38 @@
# Urdu WO65 — stage 1 + 2 (staged)
# Urdu WO65 — stage 1 + 2

Launch after B3 frees the a100s — full command pair recorded in
TODO.sota/65 (arms urd-diac-1.0 + urd-diac-base-1.0 over bucket
rababa-urdu-cc174b96@data-diacrit, D_RUN=run-088-urd-oracle, 200K x
300B subset). Stage 2 = train_oracle_student_hf.py adapted
(google/byt5-base). Gate CER < 3.57, target < 3.
## STAGE 1 COMPLETE (2026-11-03): oracle CER 2.62 vs arms 3.4 / 8.66

run-088-urd-oracle executed 2026-11-02/03. Arms: urd1 =
urd-diac-1.0-fp32.zip (sha c299ea44…, ~20 min at 183 rows/s) and urd2
= urd-diac-base-1.0-fp16.zip (sha 6818469c…, ~3h at 18 rows/s), over
data-diacrit/train.jsonl (573K rows; 486,444 usable after the ≥5-char
and ≤300-byte filters; subset 200,000, seed 42).

Consolidation (job 6ae91046c78863f9c6310f25): 137,270/200,000 rows
were exact ties under urd1 (arm2 skipped); urd2 decoded the remaining
62,730. relabel stats: `{"rows": 200000, "disagreement": 0.1045,
"arm_cer": {"urd1": 0.034, "urd2": 0.0866}, "arm_eval_rows": {"urd1":
200000, "urd2": 62730}, "oracle_cer": 0.0262, "picks": {"urd1":
183451, "urd2": 16549}}` — the oracle beats the better arm 3.4 → 2.62
(a 23% relative gain; urd2 contributes 16,549 picked rows).

Launch notes (durable): the `@revision` bucket-mount tag resolves
EMPTY — mount the plain directory path and point D_ROWS_JSONL at
rababa-urdu-cc174b96/data-diacrit/train.jsonl inside it. A
D_ARM_ONLY primary job exits without consolidating; run one final
consolidation-only job (no D_ARM_ONLY) after the second arm lands.

## STAGE 2 RUNNING (2026-11-03)

train_oracle_student_hf.py, O_RUN=run-088-urd-oracle, O_EPOCHS=3,
google/byt5-base (job 6ae933d7c78863f9c6312b44, a100, 24h). The
script gained an O_EVAL_JSONL epilogue (greedy CER on a {src,tgt}
jsonl) for non-Arabic oracle runs — train
PR interscript-train feat/urd-oracle-eval-jsonl; eval set =
data-diacrit/val.jsonl.

GATE: student CER < 3.57 (urd-vocalized-1.0 baseline); target < 3.
On gate green: export IMF artifact + parity (≥500 single-row greedy)
+ certify — the VERSION IS THE OWNER'S. Parts contract and publish
gates per README; never LLM teachers for the labels (the oracle is
the shipped arms).
Loading