Skip to content

docs(WO64): r8 stage-2 final verdict — negative at both checkpoints #348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 22, 2026
Merged

docs(WO64): r8 stage-2 final verdict — negative at both checkpoints #348

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions TODO.sota/64-b3-arabic-sota.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
# WO64 — B3: Arabic to external SOTA (DER < 2.0, WikiNews < 8)

**Stage-2 verdict (2026-10-22, INTERIM — oepoch-0 control pending):**
the r8 noisy-student continuation finished all 36,321 steps (loss
0.001-0.003) but scores **DER-CE 7.13** on the SadeedDiac-25 protocol
(1,200 paragraphs, interscript-ml-tools scoring) — worse than the v1
lineage (teacher 2.29, rung-2.0 4.82, rung-2.1 4.57) and far off the
< 2.0 target. Suspected cause: the continuation overfit the
oracle-relabeled targets (suspiciously low train loss + eval
regression). oepoch-0 (end of continuation-epoch 1) eval queued as the
control; the r8 student line ships NOTHING at this number.
**Stage-2 verdict (2026-10-23, FINAL — both checkpoints measured): NEGATIVE.**
The r8 noisy-student continuation finished all 36,321 steps (loss
0.001-0.003) but scores **DER-CE 7.13 (oepoch-1) / 6.96 (oepoch-0)**
on the SadeedDiac-25 protocol (1,200 paragraphs, interscript-ml-tools
scoring) — worse than the v1 lineage (teacher 2.29, rung-2.0 4.82,
rung-2.1 4.57) and far off the < 2.0 target. The two checkpoints
scoring the same kills the overfit hypothesis: the continuation
simply did not inherit the teacher's quality — the oracle-relabeled
targets are the bottleneck (same lesson as the Thai arm: low train
loss is not evidence when the targets are). The r8 student line ships
NOTHING; its durable infrastructure (self-contained decode, bucket
sync, chain dispatch) is the salvage. B3's SOTA path: the v1 lineage
numbers + VRR (both published) + a data-quality lever on the oracle
targets if the chain is ever revisited — the r6-morph knowledge-
injection template, not noisy-student relabeling.

Status: IN_PROGRESS (r8 stage 2 training 2026-10-22, stage-3 SadeedDiac eval auto-chained; VRR closure = WO81)
Goal: SadeedDiac DER < 2.0 (canonical 2.29), WikiNews multiref WER < 8
Expand Down
Loading