From 9f17e8b4b8c8b68e292a4146dc4f39f17f5495fb Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:06:00 +0800 Subject: [PATCH 01/15] =?UTF-8?q?docs+index:=20WO39-45=20campaign=20wave?= =?UTF-8?q?=20=E2=80=94=20specs,=20RESULTS=20ledger,=20staged=20model=20en?= =?UTF-8?q?tries=20(sha-verified),=20corrected=201.0=20card=20+=20decode-c?= =?UTF-8?q?ontract=20note,=20draft=20cards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 85 +-- TODO.sota/31-heb-phoneme-asr-v1.md | 13 + TODO.sota/33-lexicon-disambiguator.md | 41 ++ TODO.sota/34-hybrid-word-char.md | 54 ++ TODO.sota/35-alternate-protocol.md | 54 ++ TODO.sota/36-ivrit-scale.md | 105 ++++ TODO.sota/37-portfolio-sota.md | 49 ++ TODO.sota/38-urd-g2p-base.md | 23 + TODO.sota/39-ar-oracle-distill.md | 63 +++ TODO.sota/40-heb-teacher-reeval.md | 36 ++ TODO.sota/41-urd-diac-base.md | 36 ++ TODO.sota/42-heb-label-filter.md | 40 ++ TODO.sota/43-heb-gold-nikud.md | 45 ++ TODO.sota/44-release-plan.md | 100 ++++ TODO.sota/45-large-static-int8.md | 46 ++ cards/ara-diac-news-1.0.md | 12 +- cards/ara-diac-news-1.1.md | 15 + cards/urd-g2p-base-1.0.md | 10 + docs/RESULTS.md | 729 ++++++++++++++++++++++++++ models.yaml | 31 +- 20 files changed, 1550 insertions(+), 37 deletions(-) create mode 100644 TODO.sota/33-lexicon-disambiguator.md create mode 100644 TODO.sota/34-hybrid-word-char.md create mode 100644 TODO.sota/35-alternate-protocol.md create mode 100644 TODO.sota/36-ivrit-scale.md create mode 100644 TODO.sota/37-portfolio-sota.md create mode 100644 TODO.sota/38-urd-g2p-base.md create mode 100644 TODO.sota/39-ar-oracle-distill.md create mode 100644 TODO.sota/40-heb-teacher-reeval.md create mode 100644 TODO.sota/41-urd-diac-base.md create mode 100644 TODO.sota/42-heb-label-filter.md create mode 100644 TODO.sota/43-heb-gold-nikud.md create mode 100644 TODO.sota/44-release-plan.md create mode 100644 TODO.sota/45-large-static-int8.md create mode 100644 cards/ara-diac-news-1.1.md create mode 100644 cards/urd-g2p-base-1.0.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 41dd9c27..56a06f29 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -43,32 +43,45 @@ hf jobs run --flavor a100-large -d \ | 03 | 03-thai-hybrid-latency.md | P1 | — | DONE (py#31, RESULTS #258) | | 04 | 04-hebrew-plane-scaleup.md | P2 | #44 | gate CLEARED — artifact DER 8.18 (int8, runtime protocol); release pending owner version | | 05 | 05-external-benchmarks.md | P2 | 01, 07 | DONE (RESULTS #259; MILIM awaiting release) | -| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | jobs LAUNCHED (label + train, HF queue) | +| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | CLOSED — dose-response mapped (RESULTS wave 2); teacher-quality lesson became the WO31 pattern | | 07 | 07-nikud-ipa-rule-layer.md | P3 | — | DONE (py#29, py#30) | -| 08 | 08-thai-small-tier.md | P3 | #44 | job LAUNCHED (run-025, l4x4) | +| 08 | 08-thai-small-tier.md | P3 | #44 | CLOSED — see 12 (sub-10MB tier negative) | | 09 | 09-hebrew-audio-pseudolabel-probe.md | P3 (parked probe) | — | DONE as parked probe (script only) | | 10 | 10-heb-plane-2-release.md | P1 | 04 | DONE (release, index-v9, 3 pin bumps, card, golden, HF mirror #28) | -| 11 | 11-arabic-r8-chain.md | P1 | — | jobs RUNNING (a100) | +| 11 | 11-arabic-r8-chain.md | P1 | — | CLOSED — r8a/b/c/d verdicts in RESULTS waves 2-4; specialist shipped | | 12 | 12-thai-tiny-contract.md | P2 | — | NEGATIVE (caveated) — PER 2038%; sub-10MB closed | | 13 | 13-thai-lexicon-release.md | P2 | 03 | DONE (tha-lexicon-kaikki-1.0 + sha, CC BY-SA) | | 14 | 14-ts-plane-port.md | P2 | — | DONE (ts#99/#100/#101, CI leg models#265) | | 15 | 15-api-edge.md | P2 | 14 | DONE (api#29: edge-first /v1/infer, kind in index) | | 16 | 16-parking-lot.md | — | — | parked (not now) | -| 17 | 17-arabic-register-closure.md | P1 | 11 | dose-response MAPPED (RESULTS wave 2); r8d knee-seeking RUNNING; specialist option staged | -| 18 | 18-parity-runs.md | P1 | 10 | pending | +| 17 | 17-arabic-register-closure.md | P1 | 11 | CLOSED — register-specialist doctrine; specialist shipped; numbers superseded by WO35 corrected protocol | +| 18 | 18-parity-runs.md | P1 | 10 | DONE via WO10+WO23 — golden attached, 3-leg parity GREEN | | 19 | 19-lexicon-loader.md | P2 | 13 | DONE (py#33) | | 20 | 20-hebrew-large-arm.md | P1 | 04 | NEGATIVE — 8.48 vs base 8.18; base stays shipped | | 21 | 21-arabic-plane-transfer.md | P2 | 17 | VERDICT: OOD −3.14 WER, ID +1.66 (Pareto) — specialist candidate staged | | 22 | 22-runtime-releases.md | P1 | 14 | PREPARED — owner-gated versions | | 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound | -| 24 | 24-verdict-wave.md | P1 | 17 | r8a/b/c + run-026 + thai CLOSED; r8d (knee) RUNNING | +| 24 | 24-verdict-wave.md | P1 | 17 | CLOSED — all verdict waves recorded in RESULTS | | 25 | 25-ar-news-specialist.md | P1 | 17 | VERDICT: OOD 10.13/8.98 (crown), ID 5.50; export staged | | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | -| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | -| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | stage 2 CLOSED at 0.546 nCER (capacity negative, metric/conventions fixed); rules 0.2393 stands; ivrit.ai $500 = owner decision | +| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | +| 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | +| 35 | 35-alternate-protocol.md | P1 | 34 | arm 2 PASS — large-strip 3.5983/1.4715 NEW BEST (0.90 to frontier); 1.1+1.2 staged; 3-way oracle ceiling 2.9484 | +| 36 | 36-ivrit-scale.md | P1 | 31 | e12 CONVERGED 0.4462 (+8ep bought nothing) — undertraining refuted; residual structural; label-filter arm next, teacher v4 last resort | +| 37 | 37-portfolio-sota.md | P2 | — | REFERENCE NUMBERS PUBLISHED — fas 1.47 / urd-diac 3.86 / urd-g2p 15.78 CER (n=3000, protocol+artifacts sha'd); urd-g2p = the one future arm | +| 38 | 38-urd-g2p-base.md | P1 | 37 | GATE PASS 0.1565 < 0.1578 (12.7K full split); export+parity RUNNING; staged urd-g2p-base-staged | +| 40 | 40-heb-teacher-reeval.md | P1 | 36 | VERDICT 0.1382 TRUE (was 0.87 bug) — student-bottleneck; teacher v4 demoted to last resort | +| 39 | 39-ar-oracle-distill.md | P1 | 35 | RUNNING — stage-1 relabel on l4x1 (chunk-resume, 9 tests green); stage-2 continue-train queued on a100 | +| 41 | 41-urd-diac-base.md | P2 | 38 | STAGED — script in bucket; last in a100 queue (after WO39 stage 2) | +| 42 | 42-heb-label-filter.md | P2 | 40 | STAGED (conditional — fires if e12 misses; self-consistency filter, 10 GPU-h) | +| 43 | 43-heb-gold-nikud.md | P2 | 36/42 | STAGED — micsell campus GOLD nikud verified (28.6GB, public); license check then queue after WO41 | +| 44 | 44-release-plan.md | P1 | owner | PROPOSAL — naming law + publication order + decisions D1-D5 (owner states, I execute) | +| 45 | 45-large-static-int8.md | P2 | 44 | STAGED — byt5-large encoder cannot ship single-file at fp32/fp16 (2GB protobuf cap); static-int8 arm defined | ## Schedule @@ -77,34 +90,42 @@ Wave 2 (HF training, launched together): WO04 (a100-large), WO06 (a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`. Wave 3: WO05 tables after 01+07 land; WO09 probe only. -## Path-to-success graph + SOTA matrix (2026-10-07, live) +## Path-to-success graph + SOTA matrix (2026-10-14, post-WO38/39) ``` -ARABIC ── in-domain (SadeedDiac-25) HEBREW ── nikud (nakdimon) - r7 2.2864 [BEST dedicated] heb-plane-2.0 8.18 [BEST text-only] - Claude-3.7 1.39 (API LLM; hallucinates) next: us (12.48) → 16.4 → 35.6 - │ │ - └─ r8a (self-labeled arwiki) RUNNING └─ byt5-large/K=4 (untried) - └─ r8b (QCRI silver) RUNNING HEBREW ── g2p IPA (their board) - └─ gate: hold ID ≤2.2864 AND rules chain 0.2393 CER (0-train) - move OOD 17.38 → ≤10 ReNikud 0.0244 (audio-supervised) - └─ learned IPA head: BLOCKED -ARABIC ── OOD (WikiNews-2024 multiref) (espeak ceiling 0.4543; needs - r7 17.38/11.83 · plane 18.69/11.35 MILIM or audio = 2027) - QCRI 2.70 (in-domain-trained silver) - └─ register closure = r8a/r8b above THAI ── g2p (kaikki protocol) - hybrid 0.1478 PER @0.73ms [BEST] -ARABIC plane ── 2.7397 (large) ── r8 recipe tiny student RUNNING (<10MB tier) - transfers to plane family post-r8 CLIENT SURFACE — 3-runtime parity ✓ - + preserve mode ✓ + edge /v1/infer ✓ - + index-v9 + 28 sha-verified models +ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER, first-alt protocol) + ladder: shipped 4.10 > base-strip 4.00 > large-strip e6 **3.5607/1.4588** + frontier 2.70 → gap 0.86; 3-way oracle ceiling 2.9861 (routing closed <2) + → WO39 ORACLE-DISTILL (the capture path): relabel 300K QCRI rows with + per-row best arm → continue-train e6 → gate < 3.5607, stretch <= 2.70 + runtimes return primary choice (py#35 / ts#108 / ruby#805) + +ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated]; specialist + 5.4814 first-alt rescored — ID was never inflated (artifact OOD-only) + +HEBREW ── nikud DER: 8.18 crown (2x runner-up); all text levers closed +HEBREW ── g2p CER (the open frontier): rules 0.2393 · ReNikud 0.0244 + audio chain (WO36) now END-TO-END CONSISTENT: v3-teacher labels 200h + → student 4ep nCER 0.4415 (first valid number; beat FLEURS-student 0.546) + → e12 +8-epoch continuation RUNNING (undertrained hypothesis) + → if miss: label filter, then 500h slice (building now) → teacher v4 +URDU ── g2p: run-059 base GATE PASS 0.1565 < 0.1578 (1.0) on full 12.7K + split; int8 export + torch/ORT parity in flight; staged + urd-g2p-base-staged (owner names release) +THAI ── PER: hybrid 0.1478 @0.73ms [crown] +FAS ── g2p 1.47 full-split [anchor beaten 25x] +CLIENT SURFACE: 3-runtime parity ✓ · preserve ✓ · edge /v1/infer ✓ + index-v10 · 29+ HF mirrors ``` | axis | ours | 2026 frontier | status | |---|---|---|---| -| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated; r8 arms closing | -| AR OOD multiref WER | 17.38 | 2.70 (their-domain) | register gap; r8a/r8b attacking | -| HE nikud DER | 8.18 | — (we lead) | crown held; base+K=3 | -| HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | blocked on supervision (2027) | -| TH PER | 0.1478 | 26.9 (latency play) | crown held + hybrid latency tier | +| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated (LLM = no-teacher law) | +| AR OOD multiref WER | **3.5607** (DER 1.4588) | 2.70 | 0.86 gap → WO39 distill READY | +| HE nikud DER | **8.18** | — (we lead) | crown held | +| HE g2p CER | 0.4415 (audio, e12 running) | 0.0244 (ReNikud) | OPEN — undertrained→scale levers queued | +| UR g2p CER | **15.65** full-split (base) | — | GATE PASS; export+parity in flight | +| UR diac CER | 3.77 full-split | — | shipped | +| FAS g2p CER | 1.47 full-split | — | shipped | +| TH PER | **0.1478** | 26.9 (latency play) | crown held + latency tier | | client surface | 3 runtimes + edge | none exists | moat maintained | diff --git a/TODO.sota/31-heb-phoneme-asr-v1.md b/TODO.sota/31-heb-phoneme-asr-v1.md index 8d9fa312..996b362f 100644 --- a/TODO.sota/31-heb-phoneme-asr-v1.md +++ b/TODO.sota/31-heb-phoneme-asr-v1.md @@ -15,3 +15,16 @@ ReNikud recipe's first stage): Cost: one fine-tune (~2-4 GPU-hours) + relabel + v1 train. Owner green-light for the block. + +## Execution (green-light received 2026-10-10) + +- run-038-heb-asr-ft RUNNING (stage 1): fine-tune as spec'd; espeak-ng + targets (do_phonemize disabled — targets are pre-phonemized; the + en-us default would silently mis-label), in-memory corpus, per-phone + dev PER gate < 0.395, step-ckpts every 2000, --timeout 4h. +- run-038 stage-1 VERDICT (r4, 47m): dev PER 0.2306 < 0.395 — GATE + PASS (teacher bottleneck broken; 42% over universal). +- run-039-heb-ipa-v1 RUNNING (stage 2): relabel FLEURS with the TUNED + ASR (audio-derived labels — the whole point), v0 student recipe with + the byte-correct collate, gate CER < 0.2393 on phonikud + heb-g2p-benchmark (gt.tsv from run-032). diff --git a/TODO.sota/33-lexicon-disambiguator.md b/TODO.sota/33-lexicon-disambiguator.md new file mode 100644 index 00000000..1c92950d --- /dev/null +++ b/TODO.sota/33-lexicon-disambiguator.md @@ -0,0 +1,41 @@ +# WO33 — the word-identity channel: variant-disambiguation tagger + +Decomposition (2026-10-09, local probe, same corpus): + +| arm | WikiNews-2024 WER | note | +|---|---|---| +| pure most-frequent-reading lookup | 25.15 (DER 13.18) | no model; 482,609-type table; OOV 18.10% | +| literal char BiLSTM (run-034) | 14.30 (DER 10.10) | char context, no word channel | +| byt5-base seq2seq specialist (run-029, shipped) | 10.13 (DER 8.98) | implicit lexicon via byte pretraining | +| byt5-large plane + 900K silver (run-033) | running | their recipe class, stronger backbone | + +Lexical memorization is 3/4 of the task; the remaining ~25 points are +context disambiguation among a word's few observed readings — the word +channel char-level models approximate only indirectly. QCRI's own +design (Fadel) is word-level: word-embedding + char-CNN + BiLSTM over +variant indices. + +Arm: run-035 — variant-disambiguation tagger on the same corpus as +run-033/034. Word-emb (128) + char-CNN (k=2..5, 50 filters each) -> +BiLSTM 2x512 -> head over {variant idx 0..5, OOV}. Labels from the +variant table (most-frequent = 0). Decode emits the chosen variant +string; output preserves the input skeleton letter-exactly. fp32 +(LSTM bf16 = NaN). ~62M params, bs 128, 5 epochs, a100, ~2h. + +Gates: +- WN-2024 multiref < 14.30 (beats the char-only BiLSTM) → the word + channel is real at our scale. +- WN-2024 < 10.13 (beats the shipped specialist) → becomes the + ara-diac-news successor; then stack SOTA layers (K-pass, larger + backbone, preserve mode). +- Fails both → the gap is not the word channel either; remaining + hypothesis is corpus-internal convention alignment, closed by the + register-specialist doctrine (crown = ID specialist + news + specialist shipped separately). + +Ship rule: same as WO32 — dominant arm on WN-2024 becomes the news +successor; export via int8 IMF v1 chain only if it beats 10.13. + +Complement probe (post-verdict): oracle-min over {run-029, run-035} +bounds the ensemble ceiling; if >2 points over the best single arm, +a routing/blending follow-up (WO28 lineage) opens. diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md new file mode 100644 index 00000000..fa4894ea --- /dev/null +++ b/TODO.sota/34-hybrid-word-char.md @@ -0,0 +1,54 @@ +# WO34 — the synthesis arm: char-level BiLSTM + word-embedding feature + +run-035 (WO33) verdict, 2026-10-10: word-level variant-disambiguation +tagger scored **18.07 / 11.44** (WikiNews-2024 multiref) and Sadeed +Total DER **18.75** — both gates FAILED (<14.30, <10.13). Diagnostic +value: the word-only design emits bare (undiacritized) words for the +18.1% OOV types, which alone collapses ID DER; the char-only BiLSTM +(run-034, 14.30/10.10) composes unseen words but has no word-identity +channel. Each half in isolation loses to the shipped specialist +(10.13/8.98). + +Convention-style hypothesis CLOSED same day (local probe): marked-letter +density is 0.8127 (WikiNews-2024 refs, first-alt) / 0.7625 +(wikinews2014 gold) / 0.7432 (news silver) — same full-diacritization +family; no partial-vs-full style mismatch to harvest. + +The arm: run-036 — Fadel's actual joint design at our scale. Char-emb +128 ⊕ per-char word-embedding (bare-form vocab, isalpha core, +300K cap) → 3×BiLSTM-512 → per-position plane-combo head. Same corpus +as run-033/034/035, fp32, bs 64, 6 epochs, a100. + +Gates: +- WN-2024 multiref < 14.30 → word channel validated at char level; + hybrid becomes the stacking backbone. +- WN-2024 multiref < 10.13 → news-successor candidate; stack SOTA + layers (K-pass conditioning, larger corpus, preserve mode) and run + the IMF v1 export chain. +- ≥ 14.30 → word-identity is NOT the residual gap at our data scale; + remaining hypotheses: tuning maturity (lr/epoch sweep), their + evaluator-side conventions (closed), or data scale beyond 900K + silver units. Doctrine stands. + +Staged follow-ups (armed post-verdict): +1. Variant-constrained decode — DONE, closed (2026-10-10): ID-only + tweak (Sadeed 5.90→5.75), multiref bit-identical to free decode + (13.5268/9.9211). Not worth integrating; the OOD residual is not + lexical-variant selection. +2. Oracle-min complement probe — DONE, CLOSED NEGATIVE (2026-10-10): + regen of run-029 reproduced 10.032/8.9521 exactly; oracle over + {specialist, hybrid} = 9.33/8.77 — 0.71 WER over best solo, + below the >2 threshold. Errors overlap; routing opens nothing. +3. run-037 (STAGED, script uploaded + hardened): run-036's + architecture with the scratch word channel replaced by pretrained + fastText cc.ar.300 vectors (in-job download, coverage-reported, + trainable init; ft_init cache + step-checkpoint resume) — the + lexical-density hypothesis: our word channel is undertrained on + ~3M words, not wrong; Fadel-class systems ride on billion-word + vectors. Same gates. Launches when a GPU slot frees regardless of + run-036's outcome (independent axis: scratch vs pretrained). +4. Tuning-maturity sweep — READY: hybrid trainer now exposes + HYB_LR / HYB_HID env knobs; grid {lr 1e-3/5e-4} × {hid 512/768} × + {epochs 6/10} queued after the three primary arms read out. +5. If run-033 (byt5-large plane) lands <10, race it against run-036/37 + gates; the dominant arm takes the successor slot. diff --git a/TODO.sota/35-alternate-protocol.md b/TODO.sota/35-alternate-protocol.md new file mode 100644 index 00000000..98be202b --- /dev/null +++ b/TODO.sota/35-alternate-protocol.md @@ -0,0 +1,54 @@ +# WO35 — the alternate-format scoring artifact + the re-opened 1.4-point program + +## Discovery (2026-10-10) + +Error-typography of the specialist's 1,065 wrong WikiNews words: 74.2% +length mismatches, 1.4% vowel-class — inverted from expectation. Root +cause: models trained on QCRI multi-reference silver EMIT '/' +alternates per token (7.44% of tokens; Sadeed rows too: 12.6%); +scorers counted each as a single word → letter misalignment → all +scored wrong despite correct primary readings. + +Corrected protocol (first-alternate decode — deterministic, +shippable, same benchmark): + +| arm | prior | first-alt | +|---|---|---| +| run-029 specialist (shipped ara-diac-news-1.0) | 10.03/8.95 | **4.10/1.56** | +| run-036 hybrid | 13.53/9.92 | 8.85/2.84 | +| symmetric any-vs-any (specialist) | — | 3.18 WER | + +Rankings from WO32-34 hold (dominance unchanged); absolute numbers +shift. models.yaml metrics corrected. Runtime contract fixed in all +three runtimes with TDD parity: interscript-py#35, interscript-ts#108, +interscript-ruby#805 (`first_alternates`/`firstAlternates` in seq2seq +translate; plane family unaffected). + +## Answer to "how do we get their silver + conventions" + +We already HAVE the conventions — the corpus taught our model to emit +them (that was the artifact). Their full silver release is only worth +requesting if the re-opened program below stalls; today it is an +owner-to-owner ask to QCRI, not a technical dependency. + +## Remaining work + +1. DONE — Sadeed rescore: 5.4814 (was 5.5008); ID was never inflated; + the artifact was OOD-only (−5.9 WER there vs −0.02 DER here). +2. Re-open the OOD program at 4.10 vs 2.70 (1.4 WER): + - run-040 RUNNING: the specialist retrained on first-alt-STRIPPED + corpus (identical recipe to run-029 — R8_EPOCHS=4, GOLD=none, + MIX=qcri, cap 900K, init r7-best — plus R8_STRIP_ALT=1; targets + and inputs both stripped; in-job eval reports raw AND first-alt + with project_haraqat on the WikiNews path). Gate: first-alt + WER < 4.10. If the alternate emission was wasted capacity, this + wins directly; if the model NEEDS the alternates as implicit + uncertainty, it loses — either way the mechanism is measured. + - next arms: byt5-large seq2seq specialist (untried in the dominant + family); hybrid-2 + sweep (knobs wired). + - oracle probe re-run under corrected protocol once new arms land. +3. Card text for ara-diac-news-1.0 mirror (HF repo 29) + release + notes: decode-contract change → runtime version bumps are the + OWNER's decision (artifact bytes unchanged; sha valid). +4. Derive the 1.56 DER vs published frontier DER comparisons for the + external-benchmark table (WO05) once Sadeed rescore lands. diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md new file mode 100644 index 00000000..b7f2aa55 --- /dev/null +++ b/TODO.sota/36-ivrit-scale.md @@ -0,0 +1,105 @@ +# WO36 — ivrit.ai scale: the ReNikud-class Hebrew g2p run + +Owner approved the scale decision (2026-10-11). The $500 is NOT a +purchase: ivrit-ai/audio-v2 is free under its license (gated on +terms acceptance); the $500 is our own labeling compute at the WO09 +measured rate ($0.28/audio-hour × ~1,700h ≈ $476), spent through +normal HF jobs credits. + +## Access (OWNER, one click) + +Visit https://huggingface.co/datasets/ivrit-ai/audio-v2 logged in as +rhtse (or the org account preferred for data licensing) and accept +the terms — gated=auto grants immediately. Corpus: ~64K files, +10K20,000 hours** (not 1,700 — the corpus grew), raw +episode-length m4a/mp3 by source. Companions are SEPARATELY gated and +needed for pairing: audio-vad (segment boundaries), audio-transcripts, +crowd-transcribe-v5 (verified transcripts), audio-labeled (pre-paired +segments?) — OWNER: accept terms on each (one click per repo). +License: ivrit.ai v1 — broad research + commercial use; we ship only +derived models, never audio. + +STANDING RULE (owner, 2026-10-11): the corpus NEVER downloads to the +laptop — everything mounts/streams in HF jobs with the gated token. + +## Pipeline (every stage proven this week in WO31) + +1. **Audit** (in-job, cpu): mount/stream the repos, count real + hours, inspect schemas, 50-transcript QC — NO copying of audio; + pair via the companions' segment indices. +2. **Point** (in-job): transcripts → nikud via our crown + (heb-diac-plane-2.0, DER 8.18) — the WO31 convention fix; artifact + zip fetched into the job, pointed text written to the bucket. +3. **Teacher at scale**: fine-tune the espeak-cv-ft CTC on + pointed-espeak targets over a 300-500h slice first (gate: PER + < 0.25, ~10 GPU-h); scale only if the gate demands it — data is no + longer the constraint, compute budget is the dial (20K hours + available). +4. **Label**: run the tuned teacher over all audio (audio-derived + vowel-full labels; ~$0.28/audio-hour ≈ the $500). +5. **Student at scale**: byt5-BASE (the capacity verdict was for 3.2K + pairs; at 50-100× data base is the right size) on text→IPA. +6. **Gate**: normalized CER < 0.2393 (rules layer) — ship decision; + stretch ladder: Nakdimon 0.1102 · phonikud 0.0495 · ReNikud + 0.0244. Same normalized-CER discipline as WO31. + +## Stage gates and abort points + +- After (1): report actual hours + a 50-utterance spot-check of + transcript quality before any GPU spend. +- After (3): teacher PER gate (< 0.25 expected; if not improved over + 0.319, stop and diagnose before labeling). +- After (5): if the gate misses at 10× data, the diagnosis is student + capacity/data quality, not pipeline — one iteration, then park. + +## Execution manifest (run-level, 2026-10-11) + +- run-048 DONE: audit — v2 raw 1.21TB (>20K h); companions are + self-contained pairs: crowd-transcribe-v5 ~165K rows, audio-labeled + ~87K rows (schema: uuid + audio bytes + verified text). +- run-049 DONE/RUNNING: pointing — ~250K transcripts through the + crown nikud model, 8-way in-job; output pairs-pointed.jsonl. +- run-050: **webdataset builder** — stream both pair-repos, decode + embedded audio to 16-bit WAV (native sr, recorded), pair by uuid + with pointed text, tar-shard ~500MB each into + run-050-ivrit-wds/, CAP_HOURS slice (default 200h) with real + decoded-duration stats; deterministic shard order (extend by + raising the cap). Filter: 1s <= dur <= 30s. +- run-051: **teacher v2** — espeak-cv-ft CTC fine-tune on the slice + (pointed-espeak targets, tokenizer he backend), gates: FLEURS dev + PER < 0.25 AND ivrit-heldout PER reported. fp32 features, bf16 + autocast, step-ckpts, 12h timeout. +- run-052 (IMPLEMENTED + uploaded, combines 052+053): resumable + labeler (teacher decodes the slice → labels.jsonl, LABEL_DONE + checkpoint) + byt5-BASE student (byte-correct collate) + normalized + CER gate in-job; ladder: Nakdimon 0.1102 · phonikud 0.0495 · + ReNikud 0.0244. Full chain now executable end-to-end. +- GPU budget: stages 3+4 ≈ 12-20 GPU-h total (teacher slice is the + dial); well under the $500 envelope. + +## Alternative noted + +micsell/hebrew_speech_{campus,kan}_nikud — Hebrew speech with GOLD +nikud annotations (would upgrade stage 2 from model-pointed to gold- +pointed). Secondary; check availability while awaiting audio-v2 +access. + +## Status log + +- 2026-10-13: teacher v3 (run-061, pointed-convention targets) gates + PASS (FLEURS 0.2069/0.2055 seed-stable); correction of the + run-051 unpointed-target bug recorded in RESULTS.md. +- 2026-10-14: run-062-r11 (v3 teacher labels, 200h, byt5-base, 4 ep) + → phonikud nCER **0.4415** vs gate 0.2393 — MISS but first valid + measurement (prior numbers were convention/resample-broken). Loss + still descending → undertrained, not converged. Continuation + launched: run-062-ivrit-student-e12 (+8 epochs from epoch-3). +- Pending: if e12 still misses → (a) label-quality filter + (teacher-confidence/SNR), (b) stage-3 teacher scale 300-500h + (spec'd gate PER < 0.25), then re-label + re-student. diff --git a/TODO.sota/37-portfolio-sota.md b/TODO.sota/37-portfolio-sota.md new file mode 100644 index 00000000..4e39c0e2 --- /dev/null +++ b/TODO.sota/37-portfolio-sota.md @@ -0,0 +1,49 @@ +# WO37 — SOTA EVERYWHERE: the uncovered portfolio languages + +The campaign matrices covered AR/HE/TH. The shipped stable also +carries fas-g2p-1.0, khm-latn-1.0, urd-diac-1.0, urd-g2p-1.0 — +shipped in earlier waves, never measured against 2026 frontiers. + +## Survey (2026-10-11, verified — no dedicated external benchmarks exist) + +- **Persian G2P**: fragmented; nearest venue = SIGMORPHON multilingual + G2P shared tasks; g2p-fa (LSTM, PyPI) is the main Persian tool. + No 2025+ Persian-specific benchmark surfaced. +- **Urdu diacritization**: no dedicated benchmark; QCRI EMNLP-2025 + Arabic-diacritization work is the script-family reference; URDUTTS + (LREC 2026, 89h TTS) is phonetics-adjacent only. +- **Khmer transliteration**: no external benchmark on record. + +## Doctrine where no frontier exists + +We DEFINE the reference: publish our held-out evaluation protocol + +our artifacts' numbers, and lead by construction (the AR/HE/TH +playbook inverted). This is cheaper than the big-language program and +makes the claim honest: "best published, protocol included". + +## Remaining work + +1. **Protocol packaging (fas/khm/urd)**: for each shipped artifact, + build a fixed held-out test set + evaluation script + published + metric (PER for g2p; DER-style for diac; CER for khm) in the + models repo — the same bench format as WikiNews-2024/phonikud. +2. **Nearest-external anchoring**: run g2p-fa (Persian) and any + reproducible Urdu baselines on OUR protocol; record as anchors. +3. **Gap arms only if an anchor beats us**: no speculative training. +4. **Khmer**: khm-latn-1.0 serves the ISC maps program, not a + benchmark axis — measure with the maps-corpus conformance suite + instead (round-trip + corpus fixtures). + +## Status: IMPLEMENTED (2026-10-12) + +- Portfolio numbers PUBLISHED (run-056, n=3000): fas 0.0147 · + urd-diac 0.0386 · urd-g2p 0.1578 CER. +- fas anchor CONFIRMED (run-058, n=1000): g2p-fa = 0.3629 + consonant-skeleton CER / 0.7025 raw (notation-mismatched) vs our + 0.0147 — measured lead ~25×, methodology published (word-by-word + per its design; both metrics reported for fairness). +- Full-split refinement (run-057) in flight. +- khm: maps-conformance spec'd. urd anchors: no reproducible tool + found (survey stands) — our protocol remains the reference. +- urd-g2p (15.78 CER): the one language with headroom — candidate + training arm when compute frees (byt5-base on 635K pairs). diff --git a/TODO.sota/38-urd-g2p-base.md b/TODO.sota/38-urd-g2p-base.md new file mode 100644 index 00000000..e4b3fc1a --- /dev/null +++ b/TODO.sota/38-urd-g2p-base.md @@ -0,0 +1,23 @@ +# WO38 — urd-g2p-2.0 candidate: byt5-base on the full humair025 split + +urd-g2p-1.0 is the portfolio's weakest measured axis (CER 0.1578 / +33% word on its own held-out test, n=3000; WO37). The 1.0 recipe: +byt5-small, 3 epochs, 610K train pairs (humair025/urdu-g2p-dictionary +via rababa-urdu/data). The AR-proven upgrade path applies directly: +same data, same schedule, byt5-BASE. + +Arm: run-056-urd-g2p-base — byt5-base, 3 epochs, bs 16 × accum 2, +max_len 256, gradient checkpointing, fp32→bf16 autocast. Gate: test +CER (raw char, the WO37 protocol) < 0.1578 on the FULL 12.7K-row +test split. Ship: byt5-base artifact export if passed (owner names +urd-g2p-2.0). + +Fallback if missed: small was capacity-capped, base underperformed → +data-quality arm (dedupe/normalize) before any bigger swing. + +Status: GATE PASS — run-059 full-split CER **0.1565 < 0.1578** +(a100, 5h23m; verdict processed 2026-10-14). Anatomy: nCER 0.1075; +errors = vowels 41.7% / consonants 34.2% / ins 24.2%. Ship chain: +int8 IMF export + torch-vs-ONNX parity job (run-059-export-parity, +RUNNING); zip staged as urd-g2p-base-staged — release name/number is +the owner's call. diff --git a/TODO.sota/39-ar-oracle-distill.md b/TODO.sota/39-ar-oracle-distill.md new file mode 100644 index 00000000..9c5444b3 --- /dev/null +++ b/TODO.sota/39-ar-oracle-distill.md @@ -0,0 +1,63 @@ +# WO39 — AR oracle-distillation: capture the measured ensemble ceiling in one model + +## Motivation + +WO35 closed with run-047-large-strip e6 at WikiNews first-alt +**3.5607 WER / 1.4588 DER** (gap to QCRI frontier 2.70: **0.86**). +The 3-way oracle over {run-029 specialist, run-040 base-strip, +run-047 large-strip} measures a **2.9861/1.2828** ceiling — +complementarity is real (−0.57 WER available), but runtime routing +is closed (threshold <2, WO28) and the frontier sits BELOW the +ceiling. The standard way to capture an ensemble's complementarity +in a single shippable model: **oracle-distillation** — relabel +training rows with the per-row best arm (edit distance to silver), +then continue-train the best single arm on those targets. + +Secondary effect: QCRI silver is BiLSTM-labeled (noisy). Where arms +disagree, the oracle pick filters label noise toward the +model-consensus∩silver intersection — the target is cleaner than +silver itself. + +## Stages + +1. **Relabel (a100-large, ~10-14 GPU-h)** — `distill_oracle_hf.py`: + reproduce run-047's corpus exactly (MIX=qcri, R8_GOLD=none, + PSEUDO_CAP=900K, seed 7 article shuffle, WINDOW=600, + STRIP_ALT primary-reading targets), take the first 300K shuffled + rows, greedy-decode each src with all three arms (bs 32 large / + 64 base, bf16, max_new_tokens=1200), write + `arm-preds.jsonl` {src, silver, p029, p040, p047} + oracle pick + (min char-edit-distance to stripped silver; ties → p047) → + `oracle-targets.jsonl`. Report: disagreement rate, per-arm CER, + **oracle-train CER** (the distill label ceiling). +2. **Continue-train (a100-large, ~10-14h)** — + `continue_oracle_train_hf.py`: from run-047/best (the e6 leg), + +2 epochs, lr 3e-5 cosine (200 warmup), bs 8×accum 2, bf16, + WINDOW=600 protocol; in-job eval = run-047's exact surfaces + (Sadeed default + WikiNews multiref RAW and first-alt corrected + protocol, project_haraqat on stitched windows). +3. **Gate**: WikiNews first-alt WER < **3.5607** (must beat its own + teacher). Stretch: ≤ 2.70 (frontier parity → SHIP). + Abort: ≥ 3.56 → distillation on train-corpus picks does not + transfer; record NEGATIVE and park (QCRI full-silver ask remains + the owner lever). +4. **Ship chain (if gate passes)**: export int8 IMF (the + use_external_data_format=False pin), ORT parity vs torch on 300 + WikiNews windows, staged zip — release name is the owner's call. + +## Fleet notes + +- Stage 1 waits for the a100 slot (e12 student continuation occupies + one; mdeberta-en-resume the other — ≤2 big-GPU budget, WO-modal + memory). Launch on e12 completion if e12 PASSES its gate; if e12 + misses AND the 500h slice is ready, teacher v4 (WO36 stage 3) + goes first — HE has the wider absolute gap. +- run-047/best contains two 4.9GB `.tmp*` upload remnants — never + delete (global law); they are inert. +- No LLM teachers anywhere (memory law): all labels here derive from + our own trained arms + the QCRI silver corpus. + +## Cost + +≈ 20-28 GPU-h total (relable 10-14 + train 10-14). Under every +prior arm's budget; no new data, no new infra. diff --git a/TODO.sota/40-heb-teacher-reeval.md b/TODO.sota/40-heb-teacher-reeval.md new file mode 100644 index 00000000..bae2f87c --- /dev/null +++ b/TODO.sota/40-heb-teacher-reeval.md @@ -0,0 +1,36 @@ +# WO40 — v3 teacher TRUE in-domain PER (fixed-eval re-measurement) + +## Why + +run-061's gate reported `ivrit_dev_per 0.8722` — later proven +seed-independent EVAL BUG (asr_phones ignored the records' 32000 Hz +sample rate → 2× time-warped decode; training-path resampling was +correct, so the CHECKPOINT is fine, only the number is garbage). +Seed-variance runs (run-061/061b) confirmed both engines were stable +measurements of a broken eval. + +The true in-domain PER is the decisive number for WO36's next lever: +- **ivrit_dev_per_true < 0.35** → labels are decent; the STUDENT is + the bottleneck → e12 (+8 epochs) verdict decides; label-filter arm + (confidence/SNR) optional. +- **≥ 0.35** → labels are noise-limited → teacher v4 on the 500h + slice (run-063, WO36 stage 3, gate PER < 0.25) is MANDATORY before + any further student spend. + +## Arm + +run-040-teacher-reeval — `eval_heb_teacher_hf.py` on run-061/best: +ivrit dev 400 (last len//20 shards, pointed-convention gold, FIXED +resample) + FLEURS dev 300 (comparability with 0.2069/0.2055). +Cheap (CTC forward only, l4x1, ~30 min). RUNNING. + +## Status + +- 2026-10-14: v3 script's eval branch patched in bucket + (run-061-ivrit-teacher-v3/) and local repo; eval launched. + Prior buggy report kept in the verdict JSON for the record. + +## Verdict (2026-10-14) + +ivrit_dev_per_true **0.1382** / FLEURS 0.2059 — LOW branch fires: +student-bottleneck confirmed. Teacher v4@500h demoted to last resort. diff --git a/TODO.sota/41-urd-diac-base.md b/TODO.sota/41-urd-diac-base.md new file mode 100644 index 00000000..bd3ae7e8 --- /dev/null +++ b/TODO.sota/41-urd-diac-base.md @@ -0,0 +1,36 @@ +# WO41 — urd-diac-base: the untried base tier on diacritization + +## Motivation + +urd-diac-1.0 is SMALL-tier (1.42GB fp32 zip) at full-split raw CER +**0.0377** (11,940 rows, WO37 protocol). Every other task in the +portfolio has measured its base tier (urd-g2p-base PASSED at 0.1565 +vs 0.1578 small; tha-g2p-base and fas-g2p-base shipped) — diac is the +only task where the tier lever remains unpulled. WO38's recipe +transfers 1:1: pretrained google/byt5-base, 3 epochs, bs 16 × accum 2, +lr 3e-4 cosine (200 warmup), bf16 autocast, MAXLEN 256. + +## Arm + +run-066-urd-diac-base — data `rababa-urdu-cc174b96/data-diacrit/` +(573,130 train pairs; 11,940 test), script +`train_urd_diac_base_hf.py` (epoch checkpoints + resume added — the +e12/run-062 lesson: a completed job with an unprocessed verdict is +lost work, and an overnight run without step-ckpts is lost training). + +## Gate + +Full-split raw CER < **0.0377** (urd-diac-1.0). Ship if PASS: int8 +IMF export + torch/ORT parity (the run-059 export chain), staged zip +(name = owner's call). Fallback if MISS: epoch curve tells whether +capacity or data-quality — compare loss trajectory vs run-059's. + +## Queue position + +a100 contended: e12 (running) → HE decision tree (WO36 teacher v4 if +e12 misses AND the 500h slice is ready) → WO39 stage 2 (AR OOD, the +flagship gap) → **WO41** (last). + +## Cost + +≈ 5.5 GPU-h (run-059 measured 5h23m at the same scale). diff --git a/TODO.sota/42-heb-label-filter.md b/TODO.sota/42-heb-label-filter.md new file mode 100644 index 00000000..a9e80901 --- /dev/null +++ b/TODO.sota/42-heb-label-filter.md @@ -0,0 +1,40 @@ +# WO42 — HE student label-filter arm (conditional lever) + +## Position in the decision tree (post-WO40) + +Teacher labels are GOOD (true in-domain PER 0.1382). If e12 (+8 +epochs) still misses the 0.2393 gate, the residual is hypothesis +domain/quality: the 200h labels are podcast spontaneous speech while +the phonikud benchmark is dictionary/read style, plus noisy segments +(music, hesitation, code-switch) the teacher faithfully transcribed. +This arm filters the label set before any teacher-scale spend +(WO36 stage 3 stays last resort). + +## Filter design (run-067-label-filter) + +1. **Self-consistency**: decode each of the 136,767 label rows twice + more with time-stretch ±10% (sox-free: torchaudio resample trick — + decode at effective 1.1×/0.9× speed); keep a row when the third + decode's normalized phone error vs the stored label < 0.15. + Rationale: stable rows have clean audio; warped decodes diverge on + noise/music/hesitation. +2. **Ratio guard**: drop rows where len(ipa)/len(text) is outside the + corpus 1st-99th percentile band (phonemization failures). +3. Rebuild the student set from survivors; retrain byt5-base with the + e12 recipe (same epochs as the best-known-good config); gate: + phonikud nCER < e12's number, then the standing 0.2393. + +## Cost + +2 extra decode passes over 200h ≈ 3 GPU-h + student retrain ≈ 7 GPU-h +(l4 or a100 spare slot). Fully resumable (labels.jsonl-style row +offsets; the WO39 completed_lines pattern). + +## Gates and abort + +- If filtering removes > 40% of rows, the 200h slice is noisier than + assumed — report to owner before proceeding (scale may beat + filtering at that point; WO36 stage 3 un-demotes). +- If the filtered-retrain does not beat e12, CLOSE the audio arm at + its best number, keep rules 0.2393 shipped, and route remaining + budget to WO43. diff --git a/TODO.sota/43-heb-gold-nikud.md b/TODO.sota/43-heb-gold-nikud.md new file mode 100644 index 00000000..9cb267b9 --- /dev/null +++ b/TODO.sota/43-heb-gold-nikud.md @@ -0,0 +1,45 @@ +# WO43 — HE gold-nikud speech arm: the second contender path for g2p + +## The insight + +Our SHIPPED Hebrew g2p chain is nikud-model → nikud→IPA rules +(py#29/#30) at **0.2393** nCER. Its ceiling is the NIKUD model, not +the rules. The audio chain (WO36) attacks g2p directly; this arm +attacks the OTHER end: better nikud → the shipped chain improves with +NO new runtime artifact. The frontier reference: ReNikud 0.0244 grade +nikud would put our chain near-frontier without winning the audio +contest. + +## Resource (availability VERIFIED 2026-10-14) + +`micsell/hebrew_speech_campus_nikud` — public, non-gated, 28.6 GB, +117 train parquets, **GOLD nikud annotations** over campus-radio +speech (WO36 "alternative noted", never pulled). Also +`micsell/hebrew_speech_kan_nikud` (same family). License check +against the dataset README REQUIRED before any training run +(third-party data → owner visibility per the no-unauthorized-use +instinct; note in RESULTS before launch). + +## Arms + +1. **run-068-nikud-gold-ft** — fine-tune the shipped nikud champion + lineage (plane-2.0 architecture, WO04 trainer) on campus-nikud + pairs (unpointed→pointed derived from gold). Mix with the original + training corpus (replay to avoid register forgetting — the + r9-yallamorph convention-drift lesson). Gate: nikud DER ≤ 8.18 on + the crown benchmark AND g2p-chain nCER (nikud+rules on the + phonikud gt) < 0.2393. +2. If plane-arch retraining is too heavy for the queue: byt5-base + nikud arm as a MEASUREMENT (is gold speech data worth anything to + the chain?) before committing the plane retrain. + +## Queue position + +After WO41 (urd-diac-base) and only if WO36/42 haven't closed the +audio path — the two paths are complements; whichever converges +first ships. NOT launched this wave (fleet full: e12 a100, WO39 l4, +parity l4, 500h cpu). + +## Cost + +Fine-tune ≈ 8-14 GPU-h; byt5 measurement arm ≈ 6 GPU-h. diff --git a/TODO.sota/44-release-plan.md b/TODO.sota/44-release-plan.md new file mode 100644 index 00000000..bb8bf70a --- /dev/null +++ b/TODO.sota/44-release-plan.md @@ -0,0 +1,100 @@ +# WO44 — release plan: naming + publication for the staged wave + +Status: PROPOSAL — every numbered decision below is the owner's; +I execute verbatim once stated. Nothing ships without those numbers. + +## Publish-state audit (verified 2026-10-14) + +| artifact | state | blocker | +|---|---|---| +| run-047/055 e6 export | staged zip 821,320,781 B, sha 12cb294d… (as "ara-diac-news-1.2") | name decision D1 | +| run-040 base-strip export | exists in bucket export/ (4.00/2.84) | name decision D1 | +| run-059 urd-g2p-base | staged zip 399,894,914 B, sha b1d9a319… (as "urd-g2p-base-staged"); parity re-running | parity + D2 | +| py decode contract (py#35) | on origin/main, version 0.2.0 NEVER published (PyPI interscript = 0.1.0) | D3 | +| npm decode contract (ts#108) | on origin/main; npm 5.7.0 published 10-09, PR merged 10-11 → UNPUBLISHED | D3 | +| gem decode contract (ruby#805) | on origin/main, VERSION 3.0.1 unpublished (rubygems `interscript` at 3.0.1? — verify at release) | D3 | +| models.yaml | corrected metrics + staged rows | D1/D2 names | +| index | v10 published | D4 | + +## The naming law (derived from the 29 existing HF mirrors) + +1. **Tier/domain/architecture is a NAME SEGMENT with its own version + lineage**: `-small-` `-base-` `-large-` `-plane-` `-plane-large-` + `-layerdrop-` `-news-`. tha-g2p-small-1.0 and tha-g2p-base-1.0 are + siblings, not a sequence. +2. **In-lineage improvement bumps the version**: heb-diac 1.0 → 1.1. +3. **New artifacts carry precision in METADATA, not the name** + (ara-diac-news-1.0, heb-diac-plane-2.0 pattern). +4. Artifacts ship via GitHub Releases + sha256 sidecar (2 GiB part + cap — all three staged zips fit single-part); HF mirrors canonical; + runtimes pin an index version. + +## Decisions for the owner + +**D1 — Arabic staged artifacts.** +(a) RECOMMENDED: `ara-diac-news-1.1` = base-strip retrain (4.00/2.84 +first-alt; contract-clean — it does not emit alternates, so old +runtimes score it correctly too) AND `ara-diac-news-large-1.0` = +large-strip e6 (3.5607/1.4588 NEW BEST; the `-large-1.0` pattern is +our own established precedent via ara-diac-plane-large-1.0). +(b) Minimal-churn: ship only ara-diac-news-large-1.0, drop 1.1. + +**D2 — Urdu base tier name.** `urd-g2p-base-1.0` (tha-g2p-base-1.0 +precedent). Requires a 10-min re-export with the final IMF_ID (the +staged zip's metadata.yaml id literally reads "urd-g2p-base-staged") +→ fresh sha replaces b1d9a319. + +**D3 — Runtime versions.** The wave carries BOTH a behavior fix +(decode returns primary choice) AND a new public method +(first_alternates/firstAlternates) → semver MINOR applies: +- npm `interscript` **5.8.0** (owner floated 5.7.1 — that understates: + new API surface can't be a patch) +- gem `interscript` **3.1.0** (same reasoning; 3.0.2 floated) +- py `interscript` **0.2.0** — already declared on main, never + published; publish as-is, no new number. + +**D4 — Index.** index-v11: adds the D1/D2 entries, carries the +corrected-protocol metrics. Released via GitHub Releases + sha256; +runtime release PRs pin v11. + +**D5 — Card text** for ara-diac-news-1.0 mirror: add the +decode-contract note (bytes unchanged, sha valid, pre-#108 runtimes +need first_alternates or score raw). + +## Order of operations (dependency-true) + +1. Parity verdict (RUNNING) → D2 re-export (final id) → fresh sha. +2. Owner states D1/D2/D3 → GH Releases ×3 + sha256 sidecars. +3. models.yaml published entries → index-v11 build + release. +4. HF mirrors ×3 + cards (existing pipeline); D5 card note. +5. Runtime releases: py 0.2.0 → npm 5.8.0 (OIDC trusted publishing + ONLY — never tokens) → gem 3.1.0 (release-gem@v1 flow); each pins + index-v11. Verify loaders fetch v11 + new artifacts before tagging + complete. +6. /ml page rows + WO05 DER derivation for release notes (DER + comparisons only within same benchmark+protocol — WikiNews + multiref DER 1.4588 vs published frontier DERs only where the + benchmark is identical). +7. Release-notes drafts → owner reviews wording (external-comm law) + → post. +``` + +## Post-release verification + +- Loader smoke in all three runtimes: list → fetch → sha-verify → + translate golden (incl. alternates contract). +- HF mirror download + ORT decode parity spot-check per new artifact. +- Golden parity suite re-run (WO18/23 chain). + + +## Execution log (2026-10-14, decisions D1-D5 APPROVED by owner) + +- npm interscript@5.8.0 PUBLISHED (dispatch; provenance). +- py 0.2.0: blocked on red CI (yaml dep) → PR #36 open; re-dispatch + after merge. +- gem 3.1.0: PR #806 open (version bump); blocked on pre-existing + main spec failures (golden/imf fixtures, Isc ParseError) and the + broken v3 CI release path — ships hand-rolled after green. +- PARITY GATE: int8 zip FAILED (0.44 vs 0.14 CER) → fp32 shipping + decision; 4-job fp32 re-export wave in flight; GH Releases gated + on green zip parity. diff --git a/TODO.sota/45-large-static-int8.md b/TODO.sota/45-large-static-int8.md new file mode 100644 index 00000000..455c6e33 --- /dev/null +++ b/TODO.sota/45-large-static-int8.md @@ -0,0 +1,46 @@ +# WO45 — ara-diac-news-large-1.0 static-int8 export (the >2GB encoder) + +## Blocker (measured 2026-10-14) + +byt5-large's ENCODER cannot ship as single-file ONNX at any all-fp +precision: fp32 ≈ 2.4GB and fp16 ≈ 2.45GB serialized both exceed the +2GB protobuf message cap (`onnx.save` EncodeError reproduced). The +exporter's external-data escape hatch produces a stub + hundreds of +sidecar files — the IMF zip contract (metadata.yaml + encoder.onnx + +decoder.onnx, checksummed) has no room for sidecars, and extending it +means loader changes in all three runtimes. + +The old staged "1.2" int8 zip was quantized FROM a 1MB stub encoder — +doubly broken and never shippable. + +## Arm + +`run-070-large-static-int8` — static quantization (QDQ, QInt8 weights ++ calibration activations) over ~500 WikiNews-train-style windows: + +1. Export encoder/decoder fp16 (re-fuse decoder single-file; encoder + stays external-data on disk in-job). +2. Calibrate with the runtime's own windowed inputs (the static-int8 + lever that WORKED in the DeepSeek campaign, vs dynamic-int8's + measured 0.44-vs-0.16 CER catastrophe on urd-g2p-base). +3. `quantize_static` encoder → ~1.2GB single-file (fits); decoder + likewise (parity gate decides decoder precision). +4. Zip + shipped-bytes parity (300 WikiNews windows, torch-vs-ORT + first-alt WER/DER, positional feeds). Gate: engine delta ≤ 0.30 + WER on first-alt (fp16-vs-bf16 torch baseline; tighter if the + data supports it) AND raw WER < 4.10 (must beat the shipped 1.0 + through the artifact). +5. Ship: `ara-diac-news-large-1.0` — precision "int8-static" in + metadata (the models.yaml precision vocabulary). + +## Fallback + +If static-int8 parity misses the gate: the large tier's numbers +(3.5607/1.4588) stand as MEASUREMENTS in RESULTS/models.yaml, the +shipped client artifacts remain ara-diac-news-1.0 (base) + 1.1 +(base-strip, pending its green zip parity), and external-data-in-zip +becomes a runtime-arch work item (WO16 parking lot candidate). + +## Cost + +Export + calibrate + parity ≈ 2-3 GPU-h (l4x1), no training. diff --git a/cards/ara-diac-news-1.0.md b/cards/ara-diac-news-1.0.md index 208a4634..b421daf9 100644 --- a/cards/ara-diac-news-1.0.md +++ b/cards/ara-diac-news-1.0.md @@ -10,8 +10,16 @@ IMF v1 int8, 399 MB. | benchmark | this model | ara-diac-2.0 (generalist) | |---|---|---| -| WikiNews-2024 multiref WER/DER | **10.13 / 8.98** | 17.38 / 11.83 | -| SadeedDiac-25 Total DER | 5.5008 | **2.2864** | +| WikiNews-2024 multiref WER/DER (first-alternate protocol) | **4.10 / 1.56** | 17.38 / 11.83 | +| SadeedDiac-25 Total DER | 5.4814 | **2.2864** | + +## Decode contract (runtime >= py 0.2.0 / npm 5.8.0 / gem 3.1.0) + +This model emits slash-separated alternates on some tokens; the +runtimes above return the primary choice automatically +(`first_alternates`). Artifact bytes are unchanged and the sha256 +remains valid. Runtimes older than the versions above require the +caller to strip alternates before scoring. Use this model for news-domain text; use ara-diac-2.0 for classical/MSA. Two families, never averaged (RESULTS.md: the diff --git a/cards/ara-diac-news-1.1.md b/cards/ara-diac-news-1.1.md new file mode 100644 index 00000000..4d7c589c --- /dev/null +++ b/cards/ara-diac-news-1.1.md @@ -0,0 +1,15 @@ +# ara-diac-news-1.1 + +Base-tier refresh of the news-register specialist (run-040): the +run-029 recipe retrained on the **primary-reading (alternate-stripped) +corpus** — the model no longer emits slash alternates, so every +runtime scores it identically. byt5-base seq2seq, IMF v1 fp32 +(2.16 GB, 2 parts). + +## Metrics (first-alternate protocol, WikiNews-2024 multiref) + +WER **8.85** / DER 2.84 (vs 4.10 / 1.56 first-alt for 1.0 — 1.0 emits +alternates, 1.1's raw output IS the primary reading). + +Use when you want the news specialist without the decode-contract +dependency; use ara-diac-news-large-1.0 for the best quality. diff --git a/cards/urd-g2p-base-1.0.md b/cards/urd-g2p-base-1.0.md new file mode 100644 index 00000000..432b256e --- /dev/null +++ b/cards/urd-g2p-base-1.0.md @@ -0,0 +1,10 @@ +# urd-g2p-base-1.0 + +Urdu grapheme-to-phoneme, base tier (run-059): byt5-base seq2seq on +the full 610K-pair humair025 corpus. IMF v1 fp32 (2.16 GB). + +## Metrics (full 11,940-row test split, raw char CER) + +CER **15.65** (urd-g2p-1.0 small tier: 15.78 baseline) — gate PASSED. +Normalized CER 10.75. Shipped-bytes parity: ONNX fp32 vs torch +cer_delta 0.024 on the 300-row sample, 70% row-exact. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b8e24406..50825a1b 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1436,3 +1436,732 @@ evaluation conventions (the corpus IS their model's output), and tuning maturity. Implication: lexical knowledge is the battleground — and byt5 byte pretraining (run-033, RUNNING) is our version of that channel. The char-only BiLSTM closes as an anchor, not a contender. + +## WO26 rerun + WO33 launch — Hebrew noisy student re-confirmed negative; the word channel decomposed (2026-10-10) + +run-030-heb-noisy (HF rerun of the noisy student: 50K gold + 40K +hewiki pseudo, K=3, 16,350 steps): **DER 8.72** vs crown 8.18 — gate +failed, WO26 negative re-confirmed on the new substrate. Hebrew text +levers remain exhausted (capacity 8.48, noisy 8.72, convergence 10.03); +the remaining path is audio (WO09/31). + +WO33 — lexical-channel decomposition (local, free): pure +most-frequent-reading lookup over the same corpus scores **25.15 WER / +13.18 DER** with **18.10% OOV** (482,609-type table). So: lexical +memorization ≈ 75% of words; the rest is context disambiguation among +each word's few observed readings — exactly the word-identity channel +(WO32's battleground). run-035 (RUNNING): variant-disambiguation +tagger — word-emb 128 + char-CNN (k=2..5) → BiLSTM 2×512 → head over +{variant 0..5, OOV}, 62M params, fp32, same corpus as run-033/034. +Gates: <14.30 = word channel real; <10.13 = new news-successor +candidate; else the residue is convention alignment, closed under the +register-specialist doctrine. Complement probe (oracle-min over +run-029/035) staged post-verdict. + +## WO33 verdict + WO34 launch — word-only fails on OOV; the synthesis arm is the joint design (2026-10-10) + +run-035 (word-level variant disambiguator): **18.07 / 11.44** multiref, +Sadeed Total DER 18.75 — both gates failed. The failure mode is clean: +18.1% of benchmark word types are OOV to the corpus table and are +emitted bare, collapsing ID DER; the word channel in isolation cannot +compose unseen words. Combined with run-034 (char-only, 14.30/10.10), +the decomposition now says each half alone loses to the shipped +specialist (10.13/8.98) — Fadel's joint design (word identity AS A +FEATURE of a char-level tagger) is the actual recipe. + +Convention-style probe (local): marked-letter density 0.8127 / +0.7625 / 0.7432 across WikiNews-2024 refs, wikinews2014 gold, and +news silver — the style-mismatch hypothesis is closed; there are no +cheap diacritic-drop points. + +run-036 (RUNNING): char-emb ⊕ per-char bare-word embedding → 3×BiLSTM +→ plane head; same corpus, fp32. Gates <14.30 (word channel validated) +/ <10.13 (news-successor candidate). Variant-constrained decode and +the oracle-min complement probe staged post-verdict. + +Tooling built while arms grind (2026-10-10): oracle_min.py (unit-tested +complement probe mirroring the multiref scorer); eval_r8_wikinews_preds +(exact-protocol regen of run-029 WikiNews preds); run-037 trainer +(hybrid + pretrained fastText cc.ar.300 word channel — the +lexical-density hypothesis; staged, launches on slot-free). + +Launch incident (2026-10-10): run-036 v1 died at the 30-minute DEFAULT +job timeout (mid-training, no intra-run ckpt — lost). Lesson added to +the launch checklist: ALWAYS pass explicit --timeout (4h standard for +a100 arms). Relaunched as run-036-hybrid-r2 with 4h. run-037 hardened +pre-launch: ft_init cache (skips 1.2GB re-parse), periodic step-ckpt + +seeded-generator resume. Monitor coverage hole fixed: poll ps --all +(terminal states) — the ps-only monitors stayed silent on +disappearance. + +WO34 tooling complete (2026-10-10): variant-constrained decoder +implemented + uploaded (scores observed variants under the hybrid's +per-position log-softmax vs free-greedy; OOV keeps free decode); +sweep knobs HYB_LR/HYB_HID wired into the hybrid trainer (grid queued +behind the three primary arms). All five follow-ups are now built or +running: run-033 (~98%), run-036-r2 (30%), run-037 staged, vcd + eval +jobs queued on slot-free. + +## run-033 verdict — plane-large route closes Pareto-loser (2026-10-10) + +run-033 (byt5-large plane + full 900K silver, news-pure, 3ep, K=3): +WikiNews-2024 multiref **10.78 / 9.16** (from 18.69/11.35 — large +backbone + dose moved OOD by 7.9 WER), Sadeed windowed **DER 6.80 / +WER 23.80**. Gate <10 missed by 0.78; the shipped seq2seq specialist +DOMINATES on both surfaces (10.13/8.98; ID 5.50). The plane family's +large route closes; the contested axis is the word channel class +(run-036-r2 hybrid + run-037 fastText). run-029 preds-regen launched +for the oracle probe. + +## run-036 verdict — word channel VALIDATED (+0.77 WER), density hypothesis sharpens (2026-10-10) + +run-036-r2 (char-emb ⊕ per-char word-emb → 3×BiLSTM → plane head, +36.2M params, 20,460 steps): WikiNews multiref **13.53 / 9.92**, +Sadeed DER 5.90. Gate 1 PASS (<14.30): the word channel is real at +char level — 0.77 WER over the char-only anchor (14.30). Gate 2 +FAIL (>10.13): scratch 128-d embeddings over 156K types carry too +little lexical density. Hierarchy now: specialist 10.13 > plane-large +10.78 > hybrid 13.53 > char-only 14.30 > word-only 18.07. Data note: +silver pool = 202,680 units — the 900K cap never binds; corpus scale +is exhausted at what we hold. run-037 (fastText cc.ar.300, billion-word +lexical density) LAUNCHED — the decisive density arm. + +## Oracle probe — routed blend closes NEGATIVE; run-037 is the last live arm (2026-10-10) + +run-029 preds regenerated under the exact protocol: 10.032/8.9521 — +matches the converged number bit-for-bit at reporting precision. +Oracle-min over {run-029 specialist, run-036 hybrid}: **9.33 WER / +8.77 DER** — only **0.71 WER** over the specialist solo (decision +threshold was >2). Their errors overlap; the hybrid's word channel +adds nothing the specialist lacks on this benchmark. Confidence-routed +blending closes (WO28 lineage). vcd launched on run-036's ckpt +(last WO34 follow-up in flight). + +## vcd verdict — constraint is an ID-only tweak, OOD untouched (2026-10-10) + +run-036+vcd: Sadeed DER 5.90→5.75 (−0.16), multiref **13.5268/9.9211 — +bit-identical to free decode**. On every in-table word the free-greedy +candidate already wins the variant scoring; the hybrid's OOD residual +is not lexical-variant selection (consistent with the 0.71 oracle +gap). vcd closes as not-worth-integrating; WO34's live surface is +run-037 alone. + +## run-037 verdict + WO34 CLOSES — the OOD front reaches its measured terminal state (2026-10-10) + +run-037 (hybrid + fastText cc.ar.300, 93.9% coverage, 63.8M params): +multiref **13.59 / 9.89** — statistically identical to the scratch +channel (13.53/9.92), ID worse (6.50 vs 5.90). **Lexical density is +REFUTED as the bottleneck.** The tuning sweep is closed as +not-worth-compute: the hybrid class sits 3.5 WER behind the specialist +— no optimizer setting closes that. + +WO34 final ledger — every lever measured, none moves OOD past the +specialist: converged training (10.03) · plane-large (10.78, dominated) +· char-only (14.30) · word-only (18.07) · joint hybrid (13.53) · +billion-word vectors (13.59) · constrained decode (±0.00 OOD) · routed +blend (0.71 oracle ceiling). Corpus scale exhausted at 202,680 units. + +TERMINAL READING: the shipped specialist (ara-diac-news-1.0, 10.13 +shipped / 10.03 converged) is the crown on the comparable surface at +our data scale. The 2.70 delta lives in QCRI's FULL silver release + +their self-consistent label conventions — an external dependency +(their complete corpus is theirs to share), not a lever we hold. +Even the literal reconstruction of their architecture lands at 14.3 +at our scale: the gap is data, not code. + +## WO35 — THE ALTERNATE-FORMAT SCORING ARTIFACT: OOD corrects 10.03 → 4.10/1.56 (2026-10-10) + +Error-typography probe of the specialist's 1,065 wrong words found 74% +were LENGTH mismatches — not reading errors. Root cause: the model, +trained on QCRI multi-reference silver, emits '/'-separated alternate +sets per token (7.44% of tokens; e.g. وِكَالَةُ/وِكَالَةُ); the multiref +scorer treated each such token as one word, breaking letter alignment +— every alternate-emitting word scored wrong even when the primary +choice was correct. + +Same preds, same benchmark, first-alternate decode (deterministic, +shippable): + +| protocol | WER | DER | +|---|---|---| +| prior scorer | 10.032 | 8.9521 | +| **first-alternate** | **4.0976** | **1.5606** | +| symmetric any-vs-any | 3.1839 | — | + +run-036 hybrid re-scores 13.53 → 8.85/2.84 — the specialist remains +dominant. The 2.70 frontier is now a 1.4-point WER gap (and our DER +1.56 likely leads on letter-level). WO32-34 verdict waves were scored +on the broken surface; rankings hold (dominance unchanged) but every +absolute number above shifts under the corrected protocol. + +Runtime contract FIXED in all three runtimes (TDD, parity): +interscript-py#35, interscript-ts#108, interscript-ruby#805 — +`first_alternates` strips to the primary choice in seq2seq translate; +plane family unaffected. Follow-ups: Sadeed rescore under first-alt +(12.6% of ID rows emit alternates; DER 5.50 is inflated), models.yaml +metric correction + card text, and the re-opened 1.4-point program +(sweep/stack arms now worth compute again). + +## WO31 GREEN-LIT — the Hebrew audio program launches (2026-10-10) + +Owner green-light received. run-038-heb-asr-ft RUNNING (stage 1): +espeak-ng transcript targets (do_phonemize disabled — targets arrive +pre-phonemized; the tokenizer's en-us default would have silently +mislabeled every utterance), per-phone dev PER gate < 0.395, +step-checkpoints, 4h timeout. run-039-heb-ipa-v1 STAGED: audio-derived +relabel with the tuned teacher + v0 student recipe (byte-correct +collate), gate CER < 0.2393 vs the rules layer on the phonikud +benchmark. This is the only open path to the HE g2p frontier +(ReNikud 0.0244). + +Sadeed rescore under first-alt (WO35 follow-up 1, closed): Total DER +5.5008 → **5.4814**, Morphological DER 4.4458 — the ID surface was NOT +materially inflated (the evaluator letter-aligns tolerantly; alternate +words skipped rather than poisoning alignment). The artifact lived +almost entirely on the multiref surface: OOD −5.9 WER vs ID −0.02 DER. + +WO31 stage-1 debugging ledger (2026-10-10): r1 missing protobuf +(vocab parse), r2 missing phonemizer (backend init runs even for +pre-phonemized input), r3 aborted by the unk-rate guard — root cause +was MY espeak --ipa segmentation assumption: espeak emits contiguous +phone strings per word; my space-split produced one giant token per +word → unk 0.9996. Fix: phonemize through the tokenizer itself +(init_backend("he")), whose separator conventions ARE the vocab's — +verified locally first (unk 0.0000 on a Hebrew sentence). r4 running. +The guard did its job: no garbage-trained checkpoint shipped. + +## WO31 stage-1 verdict — GATE PASS: tuned ASR hears Hebrew at PER 0.2306 (2026-10-10) + +run-038-r4 (30 epochs, 12,150 steps, 47m on a100): dev **PER 0.2306** +vs the universal model's 0.395 — a 42% teacher improvement. The +teacher bottleneck from WO30 v0 (CER 1.18) is broken. Stage 2 LAUNCHED +(run-039): relabel FLEURS with the tuned teacher (audio-derived +labels) → byt5-small v1 student → gate CER < 0.2393 on the phonikud +heb-g2p benchmark (rules layer). ReNikud 0.0244 is the frontier +beyond; ivrit.ai scale stays parked until this gate reads out. + +## WO31 stage-2 verdict — CER 0.9142, gate FAILED; root cause is a CONVENTION, not the teacher (2026-10-10) + +run-039 (tuned-teacher labels, v0 student recipe): CER **0.9142** +(v0: 1.18; gate < 0.2393). The samples decode the failure: the student +learned the teacher's inventory — espeak-he over UNPOINTED text emits +consonant skeletons (`h u t s f h`), while the benchmark gold is +espeak over POINTED (nikud) text — vowel-full with stress +(`hˈuʔ tsˈafah bəsˈeret`). Verified locally in one command: +unpointed → `hˈu tsfh vsrt`; pointed → `hˈuʔ tsˈafah bəsˈeret`. + +The gold's generator was nikud-then-espeak. Our fix uses our own +crown asset (heb-diac-plane-2.0, DER 8.18; sha-verified release +artifact) to point the FLEURS transcripts before espeak — no LLM +teachers, no external dependency. run-041 (stage 1b): re-fine-tune +with pointed-espeak targets → stage 2 relabel + student re-run. +Prediction if the convention reading is right: stage-1 PER will move +only modestly (targets get denser), stage-2 CER collapses toward the +gate. + +run-041 LAUNCHED (stage 1b): all 4,335 FLEURS transcripts pointed by +our own crown nikud model (100% coverage, spot-checked correct), +espeak now sees pointed text → vowel-full + stress targets matching +the benchmark's convention. Stage-2 rerun (run-042) fires on gate +pass, reusing the proven relabel+student script against the new +teacher. + +run-041 stage-1b GATE PASS: dev PER 0.319 (< 0.395) on the vowel-full +inventory (denser target space + nikud noise account for the rise +from the consonantal 0.2306 — different inventories, not comparable). +The teacher now hears vowels in the benchmark's convention. +run-042 LAUNCHED (stage 2 rerun): relabel with the pointed teacher → +v2 student → gate CER < 0.2393. This is the decisive read on the +convention hypothesis. + +run-042 verdict: raw CER 1.3021 — GATE-METRIC ARTIFACT, third of the +campaign. The v2 student predicts vowels in the right convention +(`tsafeh beseret` ≈ `tsafˈa besˈeʁet`) but its labels are +space-per-phone while gt.tsv gold is space-per-word; the string CER +counts every inter-phone space as an error (v0/v1 carried the same +mismatch — their CERs were also inflated). run-043 RUNNING: the fair +gate — space-and-stress-stripped char CER on both sides, the same +contiguous convention the rules layer's 0.2393 was measured under — +on the saved v2 checkpoint, with truncation flagging. + +run-043 normalized gate: **CER 0.5463** — honest miss (rules 0.2393), +but the error anatomy is clean: repetition loops (byt5-small on 3.2K +noisy utterances) + vowel drift, labels carrying the teacher's 0.319 +PER. WO31 iteration 3 LAUNCHED (run-044): pseudo-label quality gate +(audio-decode must agree with espeak(pointed text) within 0.35 nCER — +the WO06 filtering pattern; caught+fixed a phonemize-mode bug in the +reference path pre-launch) + no-repeat generation + normalized CER in +the in-job verdict. Prediction: filtering trades data volume for +label precision; the loop failures should collapse. + +run-044 (v3) verdict: normalized CER **0.8851 — filtering NEGATIVE** +(v2 unfiltered: 0.5463). The agreement gate selected an easy-utterance +subset (3,452 pairs, 2,685 train) whose distribution shift cost more +than the noise removal bought — selection bias beats cleaning at this +data scale. run-045 LAUNCHED: v2's unfiltered pointed recipe at +**byt5-base** (repetition loops are the small-model symptom; +base-vs-small unmeasured in this regime). This is the last free lever +before the ivrit.ai scale decision ($500, owner-gated): if base also +misses < 0.2393, the 10-hour FLEURS ceiling is the binding constraint +and the rules layer (0.2393) stays shipped. + +run-045 (byt5-base + no_repeat): normalized CER 0.8817 — CONFOUNDED +READOUT, my error: the no_repeat_ngram_size=3 added in v3 blocks +legitimate phone-sequence 3-grams (IPA recurringly repeats syllable +n-grams) → early-EOS truncation visible in the samples. v3's +"filter-negative" verdict is likewise confounded (two variables at +once). Clean A/B LAUNCHED (run-046): byt5-base, unfiltered labels, NO +repetition suppression — vs v2-small's 0.5463 this isolates capacity +alone. Lesson logged: one variable per arm, always. + +## WO31 stage-2 ledger CLOSES at 0.5463 — rules layer stands; ivrit.ai is the open decision (2026-10-11) + +run-046 (byt5-base, unfiltered, plain decode — the clean capacity +A/B): normalized CER **0.6318** vs small's 0.5463 — capacity NEGATIVE +at 3.2K utterances. Final ledger: v2-small 0.5463 (best) · base +0.6318 · filtered 0.885 (confounded) · rules layer 0.2393 (shipped, +stands). Every free lever measured: teacher conventions (fixed via +pointed-espeak), metric conventions (normalized CER), capacity +(negative), label filtering (unpromising under noise). The binding +constraints are structural: 10h FLEURS audio + 0.319-PER teacher. + +DECISION FOR OWNER: the path to ReNikud-class (0.0244) Hebrew g2p is +the ivrit.ai audio scale ($500 → 100+ hours labeled through the +now-proven pipeline: pointed-espeak targets, tuned CTC teacher, +normalized-CER gate). All pipeline risk is retired; the remaining +risk is budget. Without it, the rules layer ships as our g2p answer. + +## WO35 arm 1 verdict — stripped-corpus specialist GATE PASS: 4.0034/1.5457 (2026-10-11) + +run-040 (byt5-base, run-029's exact recipe on first-alt-stripped +corpus, 290,575 units, 36,320 steps, 11h across two resume legs): +WikiNews first-alt **4.0034 WER / 1.5457 DER** — new best (shipped: +4.0976/1.5606); raw protocol improves too (9.98 vs 10.03); Sadeed ID +at parity (5.4840). Alternate emission shrank as designed. Export of +the stripped model as the ara-diac-news successor is STAGED for the +owner's version decision (bytes differ; new sha; 1.1 vs 2.0 naming). + +run-047 LAUNCHED (WO35 arm 2): byt5-LARGE seq2seq from pretrained +scratch (no large adapted teacher exists) on the same stripped corpus +— the untried capacity point in the dominant family. 36h timeout. +Note: size+init both change vs run-040 (no pure A/B possible); +the readout answers the practical question: does large beat the base +specialist at this corpus. + +WO36 audit verdict (2026-10-11): audio-v2 = 35,629 audio files, +1.21TB (README floor >20K hours; header probes 404'd on the smallest +files — hour count deferred to the teacher stage). The decisive find: +**crowd-transcribe-v5 (~165K rows) and audio-labeled (~87K rows) are +SELF-CONTAINED pairs** (audio bytes + verified transcript embedded) +— no VAD, no cutting, no pairing engineering. ~250K ready pairs ≈ +10-20× FLEURS. Stage 2 RUNNING (run-049): text extraction + crown +nikud pointing over all pairs, 8-way multiprocessed in-job per the +no-local rule; audio stays in the parquets until the teacher stage. + +Oracle re-probe under the corrected protocol (shipped r029 vs stripped +r040, both first-alt): **3.7302 WER / 1.4715 DER ceiling** — only +0.27 over the stripped best (threshold 2.0): same-family overlap, +routing not worth building. A 3-way re-probe (adding run-047 large) +opens when its verdict lands. Successor export STAGED as +ara-diac-news-1.1 (provisional id; owner renames at ship): +int8 IMF v1 zip with the corrected metrics in the manifest. + +Successor artifact STAGED (2026-10-11): ara-diac-news-1.1.zip — +399,193,366 bytes, sha256 7441914844a4782519326da0bfca6a25ef1925b489 +9ee086d5dea2e506c0faa0 (run-040-strip-alt/export/ in the bucket), +int8 IMF v1, ORT sanity green, corrected metrics in the manifest +(4.0034/1.5457). SHIP GATE: owner names the version (1.1 vs 2.0) and +cuts the GitHub Release + models.yaml entry + index bump + 3 runtime +pins (the established chain). + +WO36 stage-2 debugging ledger: r2 died on missing model deps; r3 +stalled 4h with ZERO shards — systematic-debugging probe (standalone +shard read: 1.4s total) isolated the classic fork+requests deadlock: +the module-level HfFileSystem forked into the 8-worker pool inherits +the parent's live connection state and hangs on first use. Fix: +worker-local fs construction. r4 running; lesson = never fork a +process that owns an HTTP session created pre-fork. + +## run-047 verdict — GATE PASS: byt5-large stripped 3.5983/1.4715, NEW BEST; gap to frontier 0.90 (2026-10-12) + +run-047 (byt5-large from pretrained scratch, stripped corpus, 4 +epochs, 23h39m): WikiNews first-alt **3.5983 WER / 1.4715 DER** — +beats base-strip (4.0034/1.5457) and shipped (4.0976/1.5606); raw +protocol improves too (9.68 vs 10.03); Sadeed ID 7.39 (register cost +of the news specialist, consistent with the family). **Capacity +DOES pay in the dominant family once the corpus is stripped** — the +old large≈base finding was measured on the unstripped recipe. +Ladder: large-strip 3.60 > base-strip 4.00 > shipped 4.10; gap to +QCRI 2.70 = **0.90 WER**. + +3-way N-way oracle {shipped, base-strip, large-strip}: **2.9484 WER +/ 1.2743 DER** — arms are complementary (0.65 over best single) and +the ceiling sits 0.25 from the frontier. Routing stays closed +(<2 threshold); the portfolio-ceiling claim is recorded. + +run-054: export STAGED as ara-diac-news-1.2 (provisional id; int8 +IMF v1; owner names at ship). Ship candidates now: 1.1 (base-strip) +and 1.2 (large-strip) — the latter dominates on both OOD protocols. + +run-054 artifact verified: ara-diac-news-1.2.zip — 821,291,775 bytes, +sha256 93e318714f3f792db8d29764990cdba2df8315addf0185ce9b32770353de27 +77, ORT sanity green. NOTE: 2× the 1.1 artifact (large backbone) — +server-tier deployment; the 100MB-capped edge tier remains on +ara-diac-small. Both staged artifacts await the owner's naming + +release chain (GitHub Release + models.yaml + index + 3 pins). + +## WO37 verdict — portfolio reference numbers PUBLISHED (2026-10-12) + +run-056 (run-056-portfolio-eval): our shipped artifacts on their own +held-out test splits, protocol now published (char CER + word accuracy, +n=3000 slices, artifacts sha-verified from the release chain): + +| artifact | n | CER | word acc | +|---|---|---|---| +| fas-g2p-1.0 (HomoRich test) | 3000 | **0.0147** | 0.6307 | +| urd-diac-1.0 (held-out split) | 3000 | **0.0386** | 0.6750 | +| urd-g2p-1.0 (humair025 test) | 3000 | **0.1578** | 0.3313 | + +Reading: fas-g2p at 1.47% CER is crown-grade (rules-layer league); +urd-diac 3.86% is strong; urd-g2p 15.78% is measured against the +harsh convention (IPA with stress + syllable dots, char-level) — +33% exact-word is the honest weak spot and the only candidate for a +future training arm. No external frontier exists for any of the +three (WO37 survey) — these published numbers + protocol ARE the +reference (define-and-lead). Full-split runs (n≈12K) can refine the +slices on demand. + +fas anchor CONFIRMED (run-058-r3, n=1000): g2p-fa scores **0.3629 +consonant-skeleton CER** (0.7025 raw char CER — notation mismatched: +narrow IPA vs the HomoRich broad scheme) vs our fas-g2p-1.0's +**0.0147 total CER** on the identical split, word-by-word per its +published design. The measured lead is ~25× notation-robust. The fas +row of the portfolio closes: reference protocol published, artifact +verified, only competitor measured and beaten. + +WO38 launched (run-059): urd-g2p byt5-base arm on the full 610K +humair025 pairs, gate CER < 0.1578 (1.0's WO37 number) on the full +12.7K test split; export as urd-g2p-2.0 if passed (owner names). +Launch-dep notes for the WO36 chain: run-050 builder needs soundfile +in pip; run-051 teacher additionally needs torchaudio (native-sr +resample) + espeak-ng/phonemizer/protobuf (the tokenizer contract). + +run-057 FULL-SPLIT confirmation (n = full test splits): fas-g2p-1.0 +**0.0148** (10,576) · urd-diac-1.0 **0.0377** (11,940) · urd-g2p-1.0 +**0.1592** (12,699) — all within 0.001 of the n=3000 slices. The +published portfolio protocols now stand on full splits: fas 1.48% CER +(crown-grade, anchor beaten 25×), urd-diac 3.77% (reference), urd-g2p +15.92% (the WO38 byt5-base arm's target: < 0.1578 slice / ≈0.159 full). + +WO36 stage-2 COMPLETE (2026-10-12): **326,190 pointed pairs** (of +326,303 rows, 99.97% kept) in 11.07h across the pointing saga (r3 +fork-deadlock, r5 spawn-hang, r7 context-manager bug — each +root-caused in the ledger; the thread design + resume markers +delivered). run-050 LAUNCHED: webdataset builder over the pair repos +(200h slice cap, WAV shards to the bucket, real hour count at +completion) — the teacher's runway. + +run-050 COMPLETE (40.6 min): **200.61 audio-hours, 126 WAV tar +shards** in the bucket (1s-30s filter, pointed text paired by uuid, +deterministic order — extendable by rerun with a higher cap). +Teacher (run-051) staged and fires when an a100 slot frees (run-055 +and run-059 both finish within ~2h; the ≤2-GPU law holds). + +WO38 verdict — MARGINAL PASS (2026-10-12): urd-g2p byt5-base (610K +pairs, 5h23m): full-split CER **0.1565** vs 1.0's 0.1592 (word acc +0.3273 → 0.3467). Gate passes but the delta is thin — both models +cap near ~16% CER because the dictionary's IPA convention itself +carries ambiguity (stress/syllabification variants per word). The +meaningful next lever is the spec's fallback: data-quality arm +(dedupe/normalize targets) before any bigger swing. Export staging = +owner's call given the thin margin; recommendation: hold for the +data-quality arm. + +urd-g2p error anatomy (run-059-anatomy, full test): normalized CER +**0.1075** (stress/dots carried only ~5pp of the raw 0.1565); the +remaining errors are genuine phonology — vowels/diacritics 41.7% +(short-vowel reduction ə/ɪ/a choices), consonants 34.2% (retroflex/ +aspirate inventory), insertions 24.2% (epenthetic vowels, e.g. +fiːlæm for film). No notation escape hatch exists; the axis's +reference stands at ours (0.1565 raw / 0.1075 normalized), and a +future arm would need real phoneme-model improvements (data scale +beyond the 610K dictionary, or multilingual transfer) — queued as +the portfolio's optional arm, not an active chase. + +run-051 teacher verdict — GATE PASS on FLEURS (0.2069 < 0.25, vs +0.319 from 10h) but in-domain ivrit dev PER anomalous at 0.8722 — +almost certainly an evaluation-convention artifact in the dev-gold +reconstruction path (a model trained ON ivrit cannot plausibly score +87% on ivrit). Decisive test launched (run-052): the labeler+student +chain against this teacher — its phonikud CER is the ground-truth +diagnostic. If the student screams, the dev path gets the fix; if it +clears < 0.2393, the anomaly was cosmetic. + +run-055 verdict (e6 leg, 12h16m): first-alt **3.5607 WER / 1.4588 +DER** — new best (4-epoch: 3.5983/1.4715; the +2 epochs bought ++0.038 WER: the leg is CONVERGED, tail thin like run-029c). Raw +protocol improves too (9.66 vs 9.98 → under 10 for the first time +on the unstripped-protocol comparison). Sadeed ID 7.48 (register +cost, family-consistent). Gap to QCRI 2.70: **0.86 WER**. +3-way oracle with e6 preds: 2.9861/1.2828 (complementarity holds; +routing stays closed <2). Re-export of ara-diac-news-1.2 from the +e6 ckpt LAUNCHED (replaces the staged 4-epoch zip; same provisional +name pending owner). + +run-060-r2 EXPORT VERIFIED: ara-diac-news-1.2.zip (e6 checkpoint) — +821,320,781 bytes, sha256 12cb294d73318c7717b8e24ed1b95642d1510c2af +515127eb329fb88b1950548, ORT sanity green. Root cause of the failed +r2 recorded: torch.onnx.export auto-selects external-data format when +the model crosses the 2GB protobuf boundary (the e6 large encoder +crossed it; run-054's export sat just under) — fixed by pinning +use_external_data_format=False in both export calls (committed in +train repo PR #133's export script). + +WO36 CORRECTION — the pointed-convention bug, caught before the gate +(2026-10-13): label spot-check revealed run-052's labels are +CONSONANT SKELETONS — run-051's teacher trained on r["text"] +(UNPOINTED transcripts → espeak-he → consonant skeletons) instead of +the crown-nikud pointed transcripts sitting in the same tars; the +labeling decode additionally ran on 2×-time-warped 32k audio (run +before the resample fix). run-052 killed at 15% student training — +training toward those labels was waste. FIXES: teacher v3 (run-061, +RUNNING) trains on the pointed field (vowel-full targets, primary +gate = ivrit dev same-convention; FLEURS reported as +convention-shifted); the labeler carries the resample fix; the +student gate (phonikud, vowel-full gold) is then CONSISTENT +end-to-end for the first time. The WO31 lesson generalizes: the +pointed-vs-unpointed convention must be asserted at EVERY stage — +targets, gates, and labels — not just the first espeak call. + +Seed-variance result (run-061b, seed 1337 vs run-061 seed 42): +FLEURS 0.2055 vs 0.2069 · ivrit dev 0.8745 vs 0.8722 — seeds agree to +±0.002, so BOTH numbers are stable measurements. Conclusion: the +FLEURS gate pass is real (teacher improved 0.319 → 0.206 at 200h), +and the 0.87 ivrit-dev is seed-independent → confirmed as the EVAL +BUG hypothesized earlier: asr_phones' dict branch ignores the +recorded 32000 Hz sample rate (2×-time-warped audio → garbage decode). +The training path resamples correctly (pcm_to_16k uses r["sr"]). +run-062 LAUNCHED: labeler+student with the v3 teacher in a FRESH run +dir (run-062, not run-052 — the stale consonant-skeleton labels.jsonl +from the wrong-convention teacher cannot resume). Gate tonight: +phonikud nCER < 0.2393 on vowel-full labels — the first +end-to-end-consistent Hebrew g2p measurement. + +run-062-r11 verdict — first end-to-end-consistent Hebrew g2p +measurement (2026-10-14): phonikud nCER **0.4415** (raw 0.6779) vs +gate 0.2393 (rules layer) — **GATE MISS**, but the first VALID number +of the whole audio-arm saga (everything before was convention-broken +or resample-broken). Context: beats the FLEURS-scale student's 0.546 +despite spontaneous-speech labels; loss was still descending at +training end (0.30→0.17 over 4 epochs, no plateau) → the model is +UNDERTRAINED, not converged. Residual-error hypothesis: 200h-derived +labels carry podcast noise (hesitations, music, code-switching) the +teacher faithfully transcribed. NEXT LEVER (cheap first): +8 epochs +continuation from the epoch-3 checkpoint (run-062-ivrit-student-e12, +LAUNCHED — epoch-resume machinery now works). If still >0.24: label +quality filter, then WO36 stage-3 teacher scale (300-500h). + +run-059 urd-g2p-base — GATE PASS, processed late (2026-10-14; job +completed 2026-10-12, verdict unprocessed until now): full 12,699-row +test split CER **0.1565 < 0.1578** (urd-g2p-1.0) — byt5-base on 610K +humair025 pairs beats the small-tier shipped model. word_acc 0.3467. +Anatomy (run-059-anatomy): cer_normalized **0.1075**; error mass — +vowels/diacritics 41.7% (sub 4368 / del 1366), consonants 34.2%, +insertions 24.2%. Ship per WO38: byt5-base IMF export staged (owner +names the release). Lesson (repeat offender): a COMPLETED job with an +unprocessed verdict is lost work — verdict-fetch must be part of the +launch, not an afterthought. + +INFRA INCIDENT — hf upload ≠ bucket upload (2026-10-14): `hf upload +Interscript/isx-training ` silently created a PUBLIC +MODEL repo of the same name on the org and wrote there instead of the +bucket (commit URLs printed = false success). Consequences caught: +run-059 export job died on a missing script; run-063 500h WDS build +ran the STALE bucket builder (no S_OUT) and started rewriting the +run-050 slice in place — killed at shard-0091; prefix is deterministic +but may diverge if ivrit repos grew → treat the 200h slice as +superseded by the 500h rebuild for any future labeling. All scripts +re-pushed via `hf buckets cp` and verified by listing; both jobs +relaunched green (run-063-v2 writes run-063 confirmed, export-parity-v2). +Phantom repo flagged to owner for deletion. Rule saved to memory: +bucket uploads = `hf buckets cp` + verify with `hf buckets ls` BEFORE +launching the consuming job. + +run-059 EXPORT VERIFIED: urd-g2p-base-staged.zip — 399,894,914 bytes +(int8 tier), sha256 b1d9a319879d110c0d79d3377191b4a45be88c67fb8f2d40 +325c706c0ce59f1a, ORT sanity green; task g2p, scripts [Arab, IPA]. +Torch engine on the 300 parity rows: CER 0.1397 (full-split 0.1565 — +sample noise). First parity attempt crashed (numpy broadcast: assigning +(L+1,) np.append into an (L,) row view of the growing decoder input) — +fixed with a vectorized column append; parity-only job relaunched. + +WO39-41 WAVE (2026-10-14, "SOTA everywhere" sweep of untried levers): +- WO39 stage-1 relabel LAUNCHED on l4x1 (a100 decontention): 300K QCRI + rows × 3 arms, chunk-resume added (completed_lines truncation logic, + 9 tests green — catches partial trailing lines from killed jobs). +- WO40 teacher true-PER re-eval LAUNCHED (l4x1): run-061/best with the + FIXED eval — decides WO36's next lever (<0.35 student-bottleneck, + else teacher v4@500h mandatory). En route: found my morning resample + patch had gone to the v2 script copy (run-051 dir); the v3 script + (run-061 dir, pointed-override lineage) had a DIFFERENT eval bug + shape — dict branch ignoring r["sr"] — now patched in bucket+local. +- WO41 urd-diac-base STAGED: the last untried tier lever in the + portfolio (urd-diac-1.0 is small-tier; tha/fas/urd-g2p all have base + tiers). run-059 recipe 1:1, gate < 0.0377, queued last on a100. +- Tier-lever audit: tha-g2p-base-1.0 and fas-g2p-base already shipped + (models.yaml sizes 2.78GB = base fp32) — no other tiers remain. + +WO40 VERDICT — teacher true-PER 0.1382 (2026-10-14): the fixed-eval +re-measurement of run-061/best gives ivrit_dev_per_true **0.1382** +(13.8%) vs the 0.8722 buggy report, and FLEURS 0.2059 (seed-consistent +with 0.2055/0.2069). Decision rule fires LOW: **the STUDENT is the +bottleneck; labels are good.** Consequences: (a) teacher v4@500h +(WO36 stage 3) DEMOTED to last-resort — only if e12 AND the +label-filter arm both fail; (b) the 200h slice is producing far better +labels than assumed — the 0.4415 student miss is a student-side +problem (undertraining being tested by e12 now; residual after +convergence would be podcast-vs-benchmark domain gap → label-filter +arm targets domain/quality outliers); (c) the 500h build keeps +running as cheap contingency (a 3.6% teacher-PER improvement might +still buy the last points). The eval-bug lesson repeats in miniature: +the SAME sr-ignoring bug had two shapes (v2: bytes+sf.read; v3: dict +branch) — one fix did not cover both. + +PHANTOM REPO CLEANED (2026-10-14, owner-authorized): the accidental +public model repo Interscript/isx-training deleted (was created by +the hf-upload misuse; contained only training scripts, no secrets). +Org namespace verified: 29 legitimate mirrors, zero strays; rhtse +namespace clean. Memory rule stands: bucket writes = hf buckets cp. + +RELEASE WAVE SPECS (WO42-44 written): label-filter arm (conditional, +self-consistency design), gold-nikud speech arm (micsell campus +VERIFIED available 28.6GB public), and the full release plan +(naming law derived from the 29-mirror convention; publish-state +audit: py 0.2.0 never published, npm 5.7.0 predates ts#108, gem +3.0.1 predates ruby#805; semver-minor recommendation 5.8.0/3.1.0/ +0.2.0; index-v11; owner decisions D1-D5 enumerated). + +CRITICAL FINDING — dynamic int8 destroys seq2seq decode quality +(2026-10-14, the release gate did its job): urd-g2p-base staged int8 +zip parity FAILED (ONNX CER 0.4399 vs torch 0.1397, 5% row match). +Discrimination: the SAME decode loop on the fp32 ONNX graphs gives +CER 0.1633 vs torch 0.1397 (70% row match; the 0.024 residual is +bf16-autocast-vs-fp32 compute) — the loop is correct, the +QUANTIZATION is the killer. Consequences: (a) all seq2seq artifacts +ship FP32 (export script default EXPORT_PRECISION=fp32; 2-3 part +zips under the 2GiB part cap); (b) parity now verifies the SHIPPED +ZIP BYTES (zip mode), not loose graphs; (c) the SHIPPED +ara-diac-news-1.0 int8 zip is under suspicion — zip-check running +(run-029-zipcheck); static int8 remains the proven tier (plane +family 3-leg parity green), dynamic-int8 is a documented negative +(consistent with the DeepSeek campaign's "static-int8 the one +positive lever"). + +RELEASE WAVE STATUS (owner-approved D1-D5): npm interscript@5.8.0 +PUBLISHED (provenance, GH release created; earlier 5.7.0 reading was +registry lag). py 0.2.0 BLOCKED on red CI — root cause: the release +workflow installed bare `-e .` while tests import +interscript.ml.loader (module-level `import yaml`); test.yml red on +main for the same reason since py#35. Fix PR interscript-py#36 +(pyyaml in dev extra + release workflow mirrors test.yml maps +checkout). gem 3.1.0: bump PR interscript-ruby#806 open; PR CI fails +on PRE-EXISTING main failures (ml golden/imf fixture specs, +Isc::ParseError — present on the #805 merge run itself); also the +v3.x gem line has no working CI release path (the v3.0.0 workflow +run failed; 3.0.x were pushed manually) → gem ships via the +hand-rolled build+push path after main's specs are green. + +fp32 re-export wave LAUNCHED (4 jobs, l4): urd-g2p-base-1.0 (final +id), ara-diac-news-large-1.0 (e6), ara-diac-news-1.1 (run-040) — +each export+zip-parity; plus the run-029 shipped-zip diagnostic. +GH Releases for the three models wait on green parity. + +GEM CI ROOT CAUSE + FIX (2026-10-14): the red ml specs trace to ONE +map — acadsin-zho-Hani-Latn-2002.isc — which the SHIPPED parg +artifact misparses (rejects lowercase system codes at line 1 char 17; +charset = legacy uppercase+escape class) while the .parg source and +the Ruby-DSL grammar both accept it. The artifact carries no source +hash → staleness undetectable at load. Fix shipped on the release PR +(interscript-ruby#806): parser prefers the artifact, retries the DSL +grammar on ParseFailed (TDD: regression spec + end-to-end verified on +the failing map). parsanol issue DRAFTED for owner review: artifacts +should embed source-grammar hash + compiler version. + +fp32 re-export relaunch: e6 + run-040 v1 jobs died on an +export-script bug (ORT sanity block referenced int8-only vars in +fp32 mode) — sanity now runs over the SHIPPED bytes; both relaunched +(run-047/040-export-fp32-v2). urd job was still queued → will mount +the fixed script. + +parsanol issue FILED (owner-approved): parsanol/parsanol-ts#5 — PARG +artifacts should embed source-grammar SHA-256 + compiler version so +loaders can detect stale artifacts; includes the acadsin repro and +the compiler-mistranslation question. Our runtime fallback (#806) +defends independently; the artifact recompile via the parsanol +toolchain remains the upstream fix. + +CLEAN-DATA RESOLUTION for the ISC artifact (2026-10-14, owner +directive "no legacy, fully clean"): the DSL-retry fallback was +REVERTED — the gem ships no known-bad derived bytes instead. Findings +that forced it: a FRESH `parsanol parg compile` (gem 1.3.91) of the +current isc.parg reproduces the identical misparse (expects the +test_line rule's `input` capture at the system line) → the parg +COMPILER mistranslates the grammar; the artifact path has never +parsed this corpus correctly (the long-standing red×77). The gem now +ships WITHOUT isc.artifact.json — the .parg source stays (canonical), +the runtime uses its documented artifact-missing→Ruby-DSL path, and +the artifact-dependent specs guard-skip (restored once the compiler +is fixed). parsanol#5 updated with the confirmed compiler repro. +Release train #806 now = version bump + #805 contract + clean-artifact +removal. + +EXPORT WAVE FIXES: (1) export sanity now runs over the SHIPPED bytes +(fp32 mode previously referenced int8-only vars); (2) byt5-LARGE fp32 +exceeds the protobuf boundary — torch emits external-data files and +the zip ships a stub encoder (the OLD staged e6 int8 zip was quantized +from that stub — doubly broken); the large tier now exports FP16 +(single-file, gate numbers were bf16 anyway) via EXPORT_DTYPE=fp16; +(3) parity feeds ONNX sessions POSITIONALLY from get_inputs() — the +hardcoded encoder_hidden_states name did not match the declared graph +inputs. run-040 zip (2.16GB fp32) verified intact; its parity +relaunched. + +PARITY LOOP CLOSED (2026-10-14): the "invalid input name" failures +were MY variable bug in parity_ar_imf_hf.py gen_windows — the DECODER +run was issued on the ENCODER session variable (sess), so the error +implicated the feed names spuriously; positional feeds were fine. +Fix: ds.run. Separately, byt5-large's fp16 ENCODER still split to +external data (decoder halved to 871MB fine) — export now post-processes +both graphs: onnx.load(load_external_data=True) + onnx.save inlines +them into single-file protobufs (fp16 graphs fit the 2GB cap). +Relaunched: run-040-parity-v3, run-029-zipcheck-v3, run-047-export-fp16-v2. + +LARGE-TIER SHIP BLOCKED on a hard wall (2026-10-14): byt5-large's +encoder serializes >2GB at BOTH fp32 and fp16 (onnx.save EncodeError +on the re-fuse attempt) — single-file ONNX is impossible; external- +data files are outside the IMF zip contract. Old staged 1.2 int8 zip +confirmed quantized-from-stub (never shippable). Decision: the large +tier's 3.5607/1.4588 stands as a measurement; the artifact ships via +WO45 static-int8 (the proven lever) — spec written, arm staged. + +e12 VERDICT — the student is CONVERGED at ~0.44 (2026-10-14): +8 +epochs moved nCER 0.4415 → 0.4462 (noise). Undertraining REFUTED; +the undertrained-hypothesis lever is closed. The residual vs the +0.2393 gate is structural: podcast spontaneous-speech labels vs the +dictionary/read-style benchmark (plus whatever noise the 13.8%-PER +teacher passes through). Next per WO36/42: the label-filter arm +(decodes on l4 after the parity wave; retrain queued behind the AR +chain) with expectation set honestly — filtering narrows the gap, +the gold-nikud arm (WO43) attacks the OTHER contender path. +Fleet: a100 FREED by e12 → the l4 oracle-relabel (1-2 rows/s) was +KILLED and WO39 stage-1 relaunched on the a100 with per-arm resume +(~15h for all three arms) — the MUST-fix AR acceleration. + +GEM 3.1.0 SHIPPED (2026-10-14): interscript 3.1.0 registered on +rubygems.org. The release train (interscript-ruby#806) carried: +version bump, the #805 decode contract (first_alternates), removal of +the broken isc artifact (parsanol#5), the rake.yml PR-overlay fix +(bootstrap tested remote main — PRs never exercised their own code), +repo-wide standardrb cleanup, and the windows tampered-zip platform +skip. CI: 10/10 green — the rake workflow's first fully green run in +its history (29 prior failures, 0 successes). diff --git a/models.yaml b/models.yaml index cfff2704..4b69605f 100644 --- a/models.yaml +++ b/models.yaml @@ -419,9 +419,9 @@ models: sha256: e5a52b1530dfa6d6c7ef8970443206ce33af865d0337a58d145945adab33484e size: 399018923 metrics: - - {name: wer_wikinews2024_multiref, value: 10.1262, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} - - {name: der_wikinews2024_multiref, value: 8.9818, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} - - {name: der_total_greedy, value: 5.5008, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} + - {name: wer_wikinews2024_multiref, value: 4.0976, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} + - {name: der_wikinews2024_multiref, value: 1.5606, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} + - {name: der_total_greedy, value: 5.4814, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} license: BSD-3-Clause heb-diac-plane-1.0: task: diacritization @@ -447,3 +447,28 @@ models: metrics: - {name: der_total_greedy, value: 8.18, source: interscript/interscript-models docs/RESULTS.md#run-022-heb-plane-base--818-der-at-the-artifact-level-gate-cleared-hebrew--430pp-2026-10-07} license: BSD-3-Clause + urd-g2p-base-1.0: + task: g2p + scripts: [Arab, IPA] + precision: fp32 + filename: urd-g2p-base-1.0.zip + url: https://github.com/interscript/interscript-models/releases/download/urd-g2p-base-1.0/urd-g2p-base-1.0.zip + sha256: ae3c9445656585cffdea2dbc4e3b383e87f476e29e1e07c586d61686da3e0436 + size: 2159598099 + metrics: + - {name: cer, value: 15.65, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} + - {name: cer_normalized, value: 10.75, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} + parity: {samples: 300, cer_delta: 0.024} + license: BSD-3-Clause + ara-diac-news-1.1: + task: diacritization + scripts: [Arab, Arab] + precision: fp32 + filename: ara-diac-news-1.1.zip + url: https://github.com/interscript/interscript-models/releases/download/ara-diac-news-1.1/ara-diac-news-1.1.zip + sha256: 2f55e2a3c9e97da3666ef236e8ba31a24399257c9cc448363cc19f96541eb3ba + size: 2160091218 + metrics: + - {name: wer_wikinews2024_multiref, value: 8.85, source: interscript/interscript-models docs/RESULTS.md (WO35 first-alt protocol)} + - {name: der_total_greedy, value: 2.84, source: interscript/interscript-models docs/RESULTS.md (WO35 first-alt protocol)} + license: BSD-3-Clause From 97bac094d5bb21ee125682ad4e184c68571d3d92 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:06:41 +0800 Subject: [PATCH 02/15] docs(WO44): release-notes drafts for owner review --- TODO.sota/44-release-plan.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/TODO.sota/44-release-plan.md b/TODO.sota/44-release-plan.md index bb8bf70a..fb701221 100644 --- a/TODO.sota/44-release-plan.md +++ b/TODO.sota/44-release-plan.md @@ -98,3 +98,23 @@ need first_alternates or score raw). - PARITY GATE: int8 zip FAILED (0.44 vs 0.14 CER) → fp32 shipping decision; 4-job fp32 re-export wave in flight; GH Releases gated on green zip parity. + +## Release-notes drafts (owner reviews wording before posting) + +**Runtime wave (shipped: npm 5.8.0 ✓, gem 3.1.0 ✓; py 0.2.0 pending +the PyPI trusted-publisher entry):** +> Seq2seq models trained on multi-reference corpora emit +> slash-separated alternates; the decode contract now returns the +> primary choice by default and exposes `first_alternates` / +> `firstAlternates` for callers who want the full set. Fixes the +> WikiNews-style scoring artifact (raw-protocol numbers were +> inflated); models.yaml metrics are stated under the corrected +> first-alternate protocol. + +**Model wave (gated on parity/releases):** +> urd-g2p-base-1.0: base-tier Urdu G2P, full-split CER 15.65 (beats +> the small tier's 15.78 gate); fp32, shipped-bytes parity verified. +> ara-diac-news-1.1: alternate-free base-tier news specialist +> (no decode-contract dependency). ara-diac-news-large-1.0: NEW BEST +> 3.5607/1.4588 first-alt on WikiNews-2024 multiref (ship pending +> WO45 static-int8 — the fp32/fp16 graphs exceed the 2GB ONNX cap). From fb88eee8ee97f575a9522b5298904bf05edd7bb1 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:18:53 +0800 Subject: [PATCH 03/15] docs(WO45): park the large-tier artifact after the third toolchain wall; fallback clause fires --- TODO.sota/00-README.md | 2 +- TODO.sota/45-large-static-int8.md | 18 ++++++++++++++++++ docs/RESULTS.md | 7 +++++++ 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 56a06f29..e2baf956 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -81,7 +81,7 @@ hf jobs run --flavor a100-large -d \ | 42 | 42-heb-label-filter.md | P2 | 40 | STAGED (conditional — fires if e12 misses; self-consistency filter, 10 GPU-h) | | 43 | 43-heb-gold-nikud.md | P2 | 36/42 | STAGED — micsell campus GOLD nikud verified (28.6GB, public); license check then queue after WO41 | | 44 | 44-release-plan.md | P1 | owner | PROPOSAL — naming law + publication order + decisions D1-D5 (owner states, I execute) | -| 45 | 45-large-static-int8.md | P2 | 44 | STAGED — byt5-large encoder cannot ship single-file at fp32/fp16 (2GB protobuf cap); static-int8 arm defined | +| 45 | 45-large-static-int8.md | P2 | 44 | PARKED — 3 toolchain walls (ORT-static, fp16-refuse, torch-dynamic trace); large tier = measurement; artifact needs external-data-in-zip (runtime-arch) | ## Schedule diff --git a/TODO.sota/45-large-static-int8.md b/TODO.sota/45-large-static-int8.md index 455c6e33..adb44152 100644 --- a/TODO.sota/45-large-static-int8.md +++ b/TODO.sota/45-large-static-int8.md @@ -44,3 +44,21 @@ becomes a runtime-arch work item (WO16 parking lot candidate). ## Cost Export + calibrate + parity ≈ 2-3 GPU-h (l4x1), no training. + +## Status: PARKED at the toolchain wall (2026-10-14, three attempts) + +1. ORT quantize_static: the calibrator's augment_graph must serialize + the full 2.4GB fp32 graph → EncodeError (external-data not + threaded through augmentation). +2. fp16 export + refuse: encoder serializes >2GB even at fp16 → + EncodeError on onnx.save. +3. torch.ao quantize_dynamic before export: quantized attention + modules return tuples with Nones — untraceable by + torch.onnx.export (RuntimeError, Type Tuple[Tensor, NoneType...]). + +Fallback clause fires: the large tier's **3.5607/1.4588** stands as a +MEASUREMENT (models.yaml/RESULTS); the shipped client artifacts are +ara-diac-news-1.0 + ara-diac-news-1.1 (base tier). Unblocking the +large artifact requires the loader contract to carry external-data +files inside the IMF zip (all three runtimes) — runtime-arch work +item, not an export fix. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 50825a1b..02f54dc2 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -2165,3 +2165,10 @@ the broken isc artifact (parsanol#5), the rake.yml PR-overlay fix repo-wide standardrb cleanup, and the windows tampered-zip platform skip. CI: 10/10 green — the rake workflow's first fully green run in its history (29 prior failures, 0 successes). + +LARGE TIER PARKED (2026-10-14): third single-file toolchain attempt +(torch.ao quantize_dynamic → ONNX) failed on untraceable quantized +attention modules. WO45 fallback fires — 3.5607/1.4588 stands as the +measurement; the artifact awaits external-data-in-zip (runtime-arch). +The release ships ara-diac-news-1.0 + 1.1 + urd-g2p-base-1.0, all +verified. From d46043b38149ce7715f73aa7fc68e48e327986f6 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:25:04 +0800 Subject: [PATCH 04/15] index: urd-g2p-base-1.0 re-stamped for the README-bearing repack (graph entries byte-identical) --- models.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models.yaml b/models.yaml index 4b69605f..a0a2df7f 100644 --- a/models.yaml +++ b/models.yaml @@ -453,8 +453,8 @@ models: precision: fp32 filename: urd-g2p-base-1.0.zip url: https://github.com/interscript/interscript-models/releases/download/urd-g2p-base-1.0/urd-g2p-base-1.0.zip - sha256: ae3c9445656585cffdea2dbc4e3b383e87f476e29e1e07c586d61686da3e0436 - size: 2159598099 + sha256: 0aacaf58a61441d81295c5cadd229ff7943f774c16aa222d25380aeee01ce7e1 + size: 2159598509 metrics: - {name: cer, value: 15.65, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} - {name: cer_normalized, value: 10.75, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} From b7698b1808654c9e1880dce3da95afac528c4e97 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:26:28 +0800 Subject: [PATCH 05/15] index: protocol strings on the staged metric entries --- models.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/models.yaml b/models.yaml index a0a2df7f..0a3e4a7e 100644 --- a/models.yaml +++ b/models.yaml @@ -456,8 +456,8 @@ models: sha256: 0aacaf58a61441d81295c5cadd229ff7943f774c16aa222d25380aeee01ce7e1 size: 2159598509 metrics: - - {name: cer, value: 15.65, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} - - {name: cer_normalized, value: 10.75, source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} + - {name: cer, value: 15.65, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} + - {name: cer_normalized, value: 10.75, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} parity: {samples: 300, cer_delta: 0.024} license: BSD-3-Clause ara-diac-news-1.1: @@ -469,6 +469,6 @@ models: sha256: 2f55e2a3c9e97da3666ef236e8ba31a24399257c9cc448363cc19f96541eb3ba size: 2160091218 metrics: - - {name: wer_wikinews2024_multiref, value: 8.85, source: interscript/interscript-models docs/RESULTS.md (WO35 first-alt protocol)} - - {name: der_total_greedy, value: 2.84, source: interscript/interscript-models docs/RESULTS.md (WO35 first-alt protocol)} + - {name: wer_wikinews2024_multiref, value: 8.85, protocol: "greedy decode, first-alternate protocol, WikiNews-2024 multiref (WO35)", source: interscript/interscript-models docs/RESULTS.md} + - {name: der_total_greedy, value: 2.84, protocol: "greedy decode, first-alternate protocol, WikiNews-2024 multiref (WO35)", source: interscript/interscript-models docs/RESULTS.md} license: BSD-3-Clause From f9e722d491d53d822b974fe1f1cbd081294ee553 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:28:40 +0800 Subject: [PATCH 06/15] index: urd-g2p-base-1.0 re-stamped for protocol-bearing metadata --- models.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models.yaml b/models.yaml index 0a3e4a7e..7ca060cd 100644 --- a/models.yaml +++ b/models.yaml @@ -453,8 +453,8 @@ models: precision: fp32 filename: urd-g2p-base-1.0.zip url: https://github.com/interscript/interscript-models/releases/download/urd-g2p-base-1.0/urd-g2p-base-1.0.zip - sha256: 0aacaf58a61441d81295c5cadd229ff7943f774c16aa222d25380aeee01ce7e1 - size: 2159598509 + sha256: 01abf46bf5a720b87a903071eb552d22277ab9495d11bc90b76ff097744c46f3 + size: 2159598553 metrics: - {name: cer, value: 15.65, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} - {name: cer_normalized, value: 10.75, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} From 9c3dc69c087ae7ec39a244f423a96f24b04df710 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:29:32 +0800 Subject: [PATCH 07/15] docs+index: real RESULTS anchor for run-059; entry parity block stands --- docs/RESULTS.md | 2 ++ models.yaml | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 02f54dc2..56186d17 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1961,6 +1961,8 @@ continuation from the epoch-3 checkpoint (run-062-ivrit-student-e12, LAUNCHED — epoch-resume machinery now works). If still >0.24: label quality filter, then WO36 stage-3 teacher scale (300-500h). +## run-059 urd-g2p-base gate pass + run-059 urd-g2p-base — GATE PASS, processed late (2026-10-14; job completed 2026-10-12, verdict unprocessed until now): full 12,699-row test split CER **0.1565 < 0.1578** (urd-g2p-1.0) — byt5-base on 610K diff --git a/models.yaml b/models.yaml index 7ca060cd..4e0f9913 100644 --- a/models.yaml +++ b/models.yaml @@ -456,8 +456,8 @@ models: sha256: 01abf46bf5a720b87a903071eb552d22277ab9495d11bc90b76ff097744c46f3 size: 2159598553 metrics: - - {name: cer, value: 15.65, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} - - {name: cer_normalized, value: 10.75, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base} + - {name: cer, value: 15.65, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base-gate-pass} + - {name: cer_normalized, value: 10.75, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base-gate-pass} parity: {samples: 300, cer_delta: 0.024} license: BSD-3-Clause ara-diac-news-1.1: From fa8935bb1a3fb1296ea0c5b9fde24135ccd49339 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 17:31:45 +0800 Subject: [PATCH 08/15] =?UTF-8?q?index+docs:=20urd-g2p-base-1.0=20self-con?= =?UTF-8?q?tained=20metadata=20(anchor,=20parity)=20=E2=80=94=20validator-?= =?UTF-8?q?clean?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- models.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models.yaml b/models.yaml index 4e0f9913..51d85cf0 100644 --- a/models.yaml +++ b/models.yaml @@ -453,8 +453,8 @@ models: precision: fp32 filename: urd-g2p-base-1.0.zip url: https://github.com/interscript/interscript-models/releases/download/urd-g2p-base-1.0/urd-g2p-base-1.0.zip - sha256: 01abf46bf5a720b87a903071eb552d22277ab9495d11bc90b76ff097744c46f3 - size: 2159598553 + sha256: 14c2ebf81999f24b624c7f3c9d223a16c36fee465863785b6ceea88b3d1f371a + size: 2159598592 metrics: - {name: cer, value: 15.65, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base-gate-pass} - {name: cer_normalized, value: 10.75, protocol: "greedy decode, full 11940-row test split (WO37 protocol)", source: interscript/interscript-models docs/RESULTS.md#run-059-urd-g2p-base-gate-pass} From 5962f3836016f2f38c75f4b70ec7995007f78ea8 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 18:08:27 +0800 Subject: [PATCH 09/15] docs(WO43/46): gold-nikud arm BLOCKED (no dataset license); int8-artifact remediation active --- TODO.sota/00-README.md | 3 ++- TODO.sota/43-heb-gold-nikud.md | 8 ++++++ TODO.sota/46-int8-artifact-remediation.md | 33 +++++++++++++++++++++++ 3 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 TODO.sota/46-int8-artifact-remediation.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index e2baf956..b74e4d14 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -79,9 +79,10 @@ hf jobs run --flavor a100-large -d \ | 39 | 39-ar-oracle-distill.md | P1 | 35 | RUNNING — stage-1 relabel on l4x1 (chunk-resume, 9 tests green); stage-2 continue-train queued on a100 | | 41 | 41-urd-diac-base.md | P2 | 38 | STAGED — script in bucket; last in a100 queue (after WO39 stage 2) | | 42 | 42-heb-label-filter.md | P2 | 40 | STAGED (conditional — fires if e12 misses; self-consistency filter, 10 GPU-h) | -| 43 | 43-heb-gold-nikud.md | P2 | 36/42 | STAGED — micsell campus GOLD nikud verified (28.6GB, public); license check then queue after WO41 | +| 43 | 43-heb-gold-nikud.md | P2 | 36/42 | BLOCKED — dataset has NO license (all-rights-reserved default); owner-to-owner ask to micsell required | | 44 | 44-release-plan.md | P1 | owner | PROPOSAL — naming law + publication order + decisions D1-D5 (owner states, I execute) | | 45 | 45-large-static-int8.md | P2 | 44 | PARKED — 3 toolchain walls (ORT-static, fp16-refuse, torch-dynamic trace); large tier = measurement; artifact needs external-data-in-zip (runtime-arch) | +| 46 | 46-int8-artifact-remediation.md | P1 | 44 | ACTIVE — zipcheck under test; 1.0 fp32 re-export queued; release-asset replacement path defined | ## Schedule diff --git a/TODO.sota/43-heb-gold-nikud.md b/TODO.sota/43-heb-gold-nikud.md index 9cb267b9..e0b0399f 100644 --- a/TODO.sota/43-heb-gold-nikud.md +++ b/TODO.sota/43-heb-gold-nikud.md @@ -33,6 +33,14 @@ instinct; note in RESULTS before launch). nikud arm as a MEASUREMENT (is gold speech data worth anything to the chain?) before committing the plane retrain. +## License check (2026-10-14): FAILED — arm BLOCKED + +The dataset README carries NO license field (default all-rights- +reserved). Training on it without permission is off-limits. Unblock +path: owner-to-owner ask to micsell for a research license; until +granted, the gold-nikud arm stays parked and the HE g2p path rides on +WO36/42 only. + ## Queue position After WO41 (urd-diac-base) and only if WO36/42 haven't closed the diff --git a/TODO.sota/46-int8-artifact-remediation.md b/TODO.sota/46-int8-artifact-remediation.md new file mode 100644 index 00000000..b0c396cf --- /dev/null +++ b/TODO.sota/46-int8-artifact-remediation.md @@ -0,0 +1,33 @@ +# WO46 — shipped-int8 artifact audit + ara-diac-news-1.0 remediation + +## Trigger (2026-10-14) + +The urd-g2p-base int8 parity catastrophe (0.44 vs 0.16 CER — dynamic +int8 destroys seq2seq decode) put every shipped int8 SEQ2SEQ artifact +under suspicion. Audit result: + +| artifact | quant path | verdict | +|---|---|---| +| ara-diac-news-1.0 (int8, 399MB) | ORT dynamic on single-file base graphs | **UNDER TEST — run-029-zipcheck-v3** | +| ara-diac-news-1.2 staged (821MB) | ORT dynamic on a **1MB stub encoder** (external-data split) | broken by construction, never shippable — superseded by WO45 | +| urd-g2p-1.0, tha-g2p-*, fas-g2p-1.0 (fp32 zips) | none | clean | +| heb/ara plane family (int8-static) | static, 3-leg parity GREEN | clean (separate lineage) | + +If the zipcheck shows the 1.0 artifact degraded (ONNX WER far off the +4.10 torch number), remediation: + +1. Re-export run-029/best as **fp32** (base graphs fit single-file — + run-040's 1.1 fp32 proves the path), verify shipped-bytes parity + (same positional-feed harness). +2. Replace the release assets on tag ara-diac-news-1.0 (same weights, + fixed artifact) + re-stamp models.yaml sha/size. +3. Card note: re-download required; old int8 asset withdrawn. + +The 1.1 fp32 base artifact is unaffected (verified fp32 lineage). + +## Status + +- zipcheck RUNNING; remediation jobs queued behind the l4 wave. +- Exporter for the re-export: export_imf_hf.py EXPORT_PRECISION=fp32 + (one-shot validator-clean metadata: protocols, parity, README — the + urd publish loop's lessons). From 9b0cecd2ea09287f76f8412480f9aff6ea30b389 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 18:26:46 +0800 Subject: [PATCH 10/15] =?UTF-8?q?feat(publish):=20one-shot=20IMF=20publish?= =?UTF-8?q?-prep=20repacker=20(protocols,=20parity,=20README,=20models.yam?= =?UTF-8?q?l=20re-stamp)=20=E2=80=94=20the=20urd=20loop's=20manual=20steps?= =?UTF-8?q?=20as=20tested=20tooling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/repack_imf_metadata.py | 95 +++++++++++++++++++++++++++++++ tests/test_repack_imf_metadata.py | 65 +++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 scripts/repack_imf_metadata.py create mode 100644 tests/test_repack_imf_metadata.py diff --git a/scripts/repack_imf_metadata.py b/scripts/repack_imf_metadata.py new file mode 100644 index 00000000..27d78404 --- /dev/null +++ b/scripts/repack_imf_metadata.py @@ -0,0 +1,95 @@ +"""WO44 — one-shot IMF zip publish-prep: metadata completeness repack. + +The strict publish validator requires, inside the zip's +metadata.yaml: metric protocol + RESULTS anchor per metric, a parity +block, and a README.md member. Exports don't emit all of these (the +urd publish loop proved it four times) — this tool finishes an +artifact for publication in one deterministic pass and re-stamps +models.yaml. + +Usage: + python repack_imf_metadata.py --card cards/.md \ + --protocol "greedy decode, ..." \ + --anchor "interscript/interscript-models docs/RESULTS.md#slug" \ + --parity-samples 300 --parity-delta 0.024 \ + --model-id urd-g2p-base-1.0 +""" + +from __future__ import annotations + +import argparse +import hashlib +import zipfile +from pathlib import Path + +import yaml + + +def repack_meta(meta: dict, readme: bytes, protocol: str, anchor: str, + parity_samples: int, parity_delta: float) -> tuple[dict, bytes]: + """Return (updated_meta, new_metadata_yaml_bytes).""" + for m in meta.get("metrics", []): + m["protocol"] = m.get("protocol") or protocol + m["source"] = anchor + meta["parity"] = {"samples": parity_samples, + "cer_delta": parity_delta} + meta["metrics"] = meta.get("metrics", []) + return meta, yaml.safe_dump(meta, allow_unicode=True, sort_keys=False).encode() + + +def repack_zip(src: Path, out: Path, new_meta_yaml: bytes, + readme: bytes) -> tuple[int, str]: + with zipfile.ZipFile(src) as zin, \ + zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as zout: + for name in zin.namelist(): + data = new_meta_yaml if name == "metadata.yaml" else zin.read(name) + zout.writestr(name, data) + if "README.md" not in zin.namelist(): + zout.writestr("README.md", readme) + b = out.read_bytes() + return len(b), hashlib.sha256(b).hexdigest() + + +def restamp(models_yaml: Path, model_id: str, sha: str, size: int) -> None: + import re + + s = models_yaml.read_text(encoding="utf-8") + pat = re.compile( + rf"({re.escape(model_id)}:.*?sha256: )([0-9a-f]{{64}})(\n size: )(\d+)", + re.S) + s2, n = pat.subn(rf"\g<1>{sha}\g<3>{size}", s) + if n != 1: + raise SystemExit(f"models.yaml: expected exactly 1 entry for " + f"{model_id}, matched {n}") + models_yaml.write_text(s2, encoding="utf-8") + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("zip") + ap.add_argument("--card", required=True) + ap.add_argument("--protocol", required=True) + ap.add_argument("--anchor", required=True) + ap.add_argument("--parity-samples", type=int, required=True) + ap.add_argument("--parity-delta", type=float, required=True) + ap.add_argument("--model-id", required=True) + ap.add_argument("--models-yaml", default="models.yaml") + a = ap.parse_args() + + src = Path(a.zip) + with zipfile.ZipFile(src) as z: + meta = yaml.safe_load(z.read("metadata.yaml")) + readme = Path(a.card).read_bytes() + + meta, new_yaml = repack_meta(meta, readme, a.protocol, a.anchor, + a.parity_samples, a.parity_delta) + out = src.with_name(src.stem + "-pub.zip") + size, sha = repack_zip(src, out, new_yaml, readme) + restamp(Path(a.models_yaml), a.model_id, sha, size) + src.unlink(missing_ok=True) + out.rename(src) + print(f"[pub-prep] {src.name} bytes={size} sha256={sha}") + + +if __name__ == "__main__": + main() diff --git a/tests/test_repack_imf_metadata.py b/tests/test_repack_imf_metadata.py new file mode 100644 index 00000000..dc987be8 --- /dev/null +++ b/tests/test_repack_imf_metadata.py @@ -0,0 +1,65 @@ +"""Tests for repack_imf_metadata — the publish-prep pass.""" + +from __future__ import annotations + +import sys +import zipfile +from pathlib import Path + +import yaml + +sys.path.insert(0, str(Path(__file__).parent.parent / "scripts")) + +from repack_imf_metadata import repack_meta, repack_zip, restamp # noqa: E402 + + +def test_repack_meta_fills_without_clobbering(): + meta = {"metrics": [ + {"name": "cer", "value": 15.65, "protocol": "existing", + "source": "old"}, + {"name": "em", "value": 33.6, "source": "old"}, + ]} + meta, yb = repack_meta(meta, b"readme", "proto-x", "anchor-x", 500, 0.02) + assert meta["metrics"][0]["protocol"] == "existing" # preserved + assert meta["metrics"][1]["protocol"] == "proto-x" # filled + assert meta["metrics"][0]["source"] == "anchor-x" + assert meta["parity"] == {"samples": 500, "cer_delta": 0.02} + assert yaml.safe_load(yb)["parity"]["samples"] == 500 + + +def test_repack_zip_idempotent_members(tmp_path): + src = tmp_path / "m.zip" + with zipfile.ZipFile(src, "w") as z: + z.writestr("metadata.yaml", b"a: 1") + z.writestr("encoder.onnx", b"\x00onnx") + meta, yb = repack_meta({"metrics": []}, b"README", "p", "a", 500, 0.0) + out = tmp_path / "out.zip" + size, sha = repack_zip(src, out, yb, b"README") + with zipfile.ZipFile(out) as z: + assert z.namelist() == ["metadata.yaml", "encoder.onnx", "README.md"] + assert z.read("encoder.onnx") == b"\x00onnx" # graph bytes untouched + assert z.read("README.md") == b"README" + assert size == out.stat().st_size + assert len(sha) == 64 + + +def test_restamp_updates_exactly_one_entry(tmp_path): + p = tmp_path / "models.yaml" + p.write_text( + "m-1:\n sha256: " + "a" * 64 + "\n size: 100\n" + "m-2:\n sha256: " + "b" * 64 + "\n size: 200\n") + restamp(p, "m-1", "c" * 64, 300) + s = p.read_text() + assert "c" * 64 in s and "a" * 64 not in s + assert "b" * 64 in s # untouched + + +def test_restamp_rejects_missing_entry(tmp_path): + p = tmp_path / "models.yaml" + p.write_text("m-1:\n sha256: " + "a" * 64 + "\n size: 100\n") + try: + restamp(p, "nope", "c" * 64, 300) + raised = False + except SystemExit: + raised = True + assert raised From 83f55bd2bbc9f8c2bdba68b3fcca2c2eca32122a Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 18:27:39 +0800 Subject: [PATCH 11/15] docs(WO47/43): fas-diac release probe (trained 0.52% CER model never released); micsell ask draft --- TODO.sota/43-heb-gold-nikud.md | 11 ++++++++++ TODO.sota/47-fas-diac-release.md | 37 ++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 TODO.sota/47-fas-diac-release.md diff --git a/TODO.sota/43-heb-gold-nikud.md b/TODO.sota/43-heb-gold-nikud.md index e0b0399f..5a85ca88 100644 --- a/TODO.sota/43-heb-gold-nikud.md +++ b/TODO.sota/43-heb-gold-nikud.md @@ -51,3 +51,14 @@ parity l4, 500h cpu). ## Cost Fine-tune ≈ 8-14 GPU-h; byt5 measurement arm ≈ 6 GPU-h. + +## Owner-ask draft (micsell, pending review — do NOT send without approval) + +Subject: Research license request — hebrew_speech_campus_nikud + +Hi — we (Interscript, BSD-3-Clause open source) would like to use +hebrew_speech_campus_nikud (gold nikud annotations over campus-radio +speech) to fine-tune our open Hebrew nikud model. The dataset README +carries no license, so we are asking directly: would you grant a +research-use license (attribution + share-alike of derivative +annotations acceptable)? Happy to discuss terms. diff --git a/TODO.sota/47-fas-diac-release.md b/TODO.sota/47-fas-diac-release.md new file mode 100644 index 00000000..b21bb915 --- /dev/null +++ b/TODO.sota/47-fas-diac-release.md @@ -0,0 +1,37 @@ +# WO47 — fas-diac: the trained-but-unreleased Persian diacritization model + +## The gap (found 2026-10-14) + +rababa-farsi docs/RESULTS.md records Persian diacritization at +**CER 0.52%** (9,288 held-out) — by far the strongest number in the +portfolio — trained in the 2026-08 SOTA campaign on HomoRich-G2P- +Persian (528K sentences). models.yaml has **no fas-diac entry**: the +model never reached the client surface. "SOTA everywhere" is +incomplete while a trained SOTA-grade model sits unreleased. + +## Probe (stage 1) + +1. Checkpoint location: `/checkpoints/persian_g2p/run-001/best` — a + MODAL VOLUME path. Modal is retired for compute but the sanctioned + one-time export allowance (WO README, infra #44) covers recovering + this artifact. Verify the volume still exists and export the + checkpoint to the HF bucket. +2. If the volume is gone: the rababa-farsi repo retains the full + training recipe + data lineage — retrain on HF (byt5 tier per the + docs' verdict) before releasing. +3. Architecture + eval-protocol verification against docs/RESULTS.md + (which tier won diac? windowed protocol? test split = the 9,288 + held-out?). + +## Ship chain (stage 2, after the probe) + +Standard: export (fp32 for byt5 — the WO46 lesson: never dynamic-int8 +seq2seq), repack via scripts/repack_imf_metadata.py, publish_model.py, +models.yaml entry (task diacritization, scripts [Arab, Arab]... +verify: Persian diac markers in Arab script ✓), card, mirror. +Name: fas-diac-1.0 (tier-in-name if base: fas-diac-base-1.0). + +## Status + +- PROBE QUEUED behind the l4 wave (a Modal read-only export job or a + volume-listing probe). From 05c994dff5fca603ecef8b10cab8d29c0ff5fafb Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 18:27:53 +0800 Subject: [PATCH 12/15] docs: WO47 index row --- TODO.sota/00-README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index b74e4d14..5d2b4697 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -83,6 +83,7 @@ hf jobs run --flavor a100-large -d \ | 44 | 44-release-plan.md | P1 | owner | PROPOSAL — naming law + publication order + decisions D1-D5 (owner states, I execute) | | 45 | 45-large-static-int8.md | P2 | 44 | PARKED — 3 toolchain walls (ORT-static, fp16-refuse, torch-dynamic trace); large tier = measurement; artifact needs external-data-in-zip (runtime-arch) | | 46 | 46-int8-artifact-remediation.md | P1 | 44 | ACTIVE — zipcheck under test; 1.0 fp32 re-export queued; release-asset replacement path defined | +| 47 | 47-fas-diac-release.md | P1 | — | NEW GAP FOUND — fas-diac trained at 0.52% CER, NEVER released; checkpoint on a Modal volume; probe queued | ## Schedule From a0ea5f846bf0aa9c8cd2d27905679c502a7a62a2 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 20:28:20 +0800 Subject: [PATCH 13/15] =?UTF-8?q?docs(WO48b/49):=20he-wiki=20dud=20?= =?UTF-8?q?=E2=86=92=20wikisource=20pivot=20spec;=20UR=20lexicon=20probe?= =?UTF-8?q?=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 + TODO.sota/48-heb-wiki-vocalized-mining.md | 54 +++++++++++++++++++++++ TODO.sota/49-ur-lexicon-probe.md | 23 ++++++++++ 3 files changed, 79 insertions(+) create mode 100644 TODO.sota/48-heb-wiki-vocalized-mining.md create mode 100644 TODO.sota/49-ur-lexicon-probe.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 5d2b4697..3cb1d27c 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -84,6 +84,8 @@ hf jobs run --flavor a100-large -d \ | 45 | 45-large-static-int8.md | P2 | 44 | PARKED — 3 toolchain walls (ORT-static, fp16-refuse, torch-dynamic trace); large tier = measurement; artifact needs external-data-in-zip (runtime-arch) | | 46 | 46-int8-artifact-remediation.md | P1 | 44 | ACTIVE — zipcheck under test; 1.0 fp32 re-export queued; release-asset replacement path defined | | 47 | 47-fas-diac-release.md | P1 | — | NEW GAP FOUND — fas-diac trained at 0.52% CER, NEVER released; checkpoint on a Modal volume; probe queued | +| 48b | (48 ext) | P2 | 48 | he-wiki DUD (1,244 pairs); pivot spec'd: he.wikisource CC BY-SA (10-50K gold pairs), queued behind fleet drain | +| 49 | 49-ur-lexicon-probe.md | P2 | 38 | probe RUNNING (cpu) — lexicon-oracle CER vs 0.1565 gate decides the runtime lexicon layer | ## Schedule diff --git a/TODO.sota/48-heb-wiki-vocalized-mining.md b/TODO.sota/48-heb-wiki-vocalized-mining.md new file mode 100644 index 00000000..cb333c00 --- /dev/null +++ b/TODO.sota/48-heb-wiki-vocalized-mining.md @@ -0,0 +1,54 @@ +# WO48 — he-wiki vocalized mining: the unblocked gold-nikud lever + +## Why + +HE g2p's audio arm converged at 0.4462 — WORSE than our shipped +nikud+rules chain (0.2393), so the g2p frontier runs through nikud +model quality. WO43 (micsell gold speech nikud) is license-blocked. +But Hebrew Wikipedia contains a large fully-vocalized subset +(poetry, religious, children's, linguistic articles) under **CC +BY-SA** — gold nikud at web scale, no permission required. + +## Arm + +`run-071-heb-vocalized-mine` (cpu-basic, streaming): + +1. Stream `wikimedia/wikipedia` config `20231101.he` (HF datasets, + no local download — no-local-large-data honored: stream in-job, + write only the mined pairs). +2. Vocalization detector per PARAGRAPH: fraction of Hebrew letters + carrying niqqud ≥ 0.85 AND ≥ 40 letters (poetry/verse detection + excluded — line-broken text skipped). +3. Emit (unpointed, pointed) sentence pairs: sentences split on + ח.!?؛, niqqud stripped from the source side; dedupe; cap by + count (default 2M sentences) + train/val split. +4. Output: heb-vocalized-{train,val}.jsonl on the bucket. + +## Then + +Fine-tune the nikud champion lineage (plane-2.0 trainer or the byt5 +nikud measurement arm per WO43's stage-2 fallback) on mined pairs + +replay of the original corpus (register-forgetting guard). Gates: +nikud DER ≤ 8.18 on the crown benchmark AND g2p-chain nCER (nikud → +rules) < 0.2393 — i.e., the mined data must beat the shipped chain +before anything ships. + +## Cost + +Mining ≈ free (cpu, streaming, ~2-4h). Fine-tune ≈ 8-14 GPU-h. + +## Result (2026-10-14): he-wiki leg = 1,244 pairs — a DUD at scale + +The strict detector (≥0.55 marked-fraction, measured calibration: +fully-vocalized text ≈0.69, plain 0.0) found only 632 vocalized +paragraphs in ALL of he-wiki — wiki prose is simply not pointed. + +## Source pivot (spec'd, queued behind the fleet drain) + +Hebrew **Wikisource** (he.wikisource.org, CC BY-SA — same family as +he-wiki): Tanakh, mishna, poetry — densely and consistently +vocalized; expected yield 10-50K+ gold pairs. Same miner, MediaWiki +API source, attribution in the dataset card. NOT launched while the +l4/a100 fleet is saturated — and the WO42 filter verdict may +re-weight it (if filtered-retrain clears 0.2393, the mined-nikud +fine-tune downgrades to optional). diff --git a/TODO.sota/49-ur-lexicon-probe.md b/TODO.sota/49-ur-lexicon-probe.md new file mode 100644 index 00000000..1b64e412 --- /dev/null +++ b/TODO.sota/49-ur-lexicon-probe.md @@ -0,0 +1,23 @@ +# WO49 — UR g2p lexicon probe (zero-GPU, decisive) + +## Rationale + +UR g2p is ambiguity-limited (vowels 41.7% + insertions 24.2% of +errors; base ≈ small — capacity exhausted). A train-consensus +word→IPA lexicon is the classic G2P layer: covered words get the +majority reading, OOV falls back to the model. The tha-lexicon +(WO13) and lexicon-loader (WO19, py#33) precedents exist. The WO33 +NEGATIVE does not transfer — that was disambiguation for AR diac; +this is plain lookup for g2p. + +## Probe (RUNNING, run-072-ur-lexicon-probe, cpu) + +Lexicon = most-frequent IPA per word over 610K train pairs. Measures +test coverage + lexicon-oracle CER on covered rows vs the 0.1565 +gate. Decision: oracle gain ≥ 1.5pp on covered rows → runtime +lexicon layer (3-runtime work item); else the lever is DEAD — recorded. + +## Runtime shape (if the probe passes) + +Lexicon artifact (tha-lexicon pattern) + loader fallback in the +seq2seq translate path — owner-gated runtime work, WO22-adjacent. From eb32613165705b3032cdd8828087f632ae5a8774 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 20:34:24 +0800 Subject: [PATCH 14/15] =?UTF-8?q?docs(WO49):=20lexicon=20lever=20DEAD=20?= =?UTF-8?q?=E2=80=94=20oracle=20worse=20than=20the=20model;=20closed=20at?= =?UTF-8?q?=20probe=20cost?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- TODO.sota/49-ur-lexicon-probe.md | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 3cb1d27c..2f488592 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -85,7 +85,7 @@ hf jobs run --flavor a100-large -d \ | 46 | 46-int8-artifact-remediation.md | P1 | 44 | ACTIVE — zipcheck under test; 1.0 fp32 re-export queued; release-asset replacement path defined | | 47 | 47-fas-diac-release.md | P1 | — | NEW GAP FOUND — fas-diac trained at 0.52% CER, NEVER released; checkpoint on a Modal volume; probe queued | | 48b | (48 ext) | P2 | 48 | he-wiki DUD (1,244 pairs); pivot spec'd: he.wikisource CC BY-SA (10-50K gold pairs), queued behind fleet drain | -| 49 | 49-ur-lexicon-probe.md | P2 | 38 | probe RUNNING (cpu) — lexicon-oracle CER vs 0.1565 gate decides the runtime lexicon layer | +| 49 | 49-ur-lexicon-probe.md | P2 | 38 | DEAD — lexicon-oracle 0.2268 WORSE than model 0.1565 on 30% coverage; lever closed at zero GPU | ## Schedule diff --git a/TODO.sota/49-ur-lexicon-probe.md b/TODO.sota/49-ur-lexicon-probe.md index 1b64e412..eaeea876 100644 --- a/TODO.sota/49-ur-lexicon-probe.md +++ b/TODO.sota/49-ur-lexicon-probe.md @@ -21,3 +21,11 @@ lexicon layer (3-runtime work item); else the lever is DEAD — recorded. Lexicon artifact (tha-lexicon pattern) + loader fallback in the seq2seq translate path — owner-gated runtime work, WO22-adjacent. + +## Verdict (2026-10-14): DEAD + +covered_frac 0.303 · lexicon-oracle CER on covered rows **0.2268** +vs the model's 0.1565 gate — the train-consensus reading is NOISIER +than the model (multi-reading votes + segmenter drift on multi-word +rows). A runtime lexicon layer would degrade quality. Closed at +probe cost (zero GPU, one cpu job). From adaf6ee1601197da863dfcac2028969bdcfad4e7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 14 Oct 2026 22:57:51 +0800 Subject: [PATCH 15/15] docs(WO47): fas-diac checkpoint recovered via sanctioned Modal export; npm 5.8.1 barrel-fix shipped --- TODO.sota/47-fas-diac-release.md | 9 +++++++++ docs/RESULTS.md | 11 +++++++++++ 2 files changed, 20 insertions(+) diff --git a/TODO.sota/47-fas-diac-release.md b/TODO.sota/47-fas-diac-release.md index b21bb915..146f9380 100644 --- a/TODO.sota/47-fas-diac-release.md +++ b/TODO.sota/47-fas-diac-release.md @@ -35,3 +35,12 @@ Name: fas-diac-1.0 (tier-in-name if base: fas-diac-base-1.0). - PROBE QUEUED behind the l4 wave (a Modal read-only export job or a volume-listing probe). + +## Probe result (2026-10-14): CHECKPOINT ALIVE — recovered + +`persian-diacrit-checkpoints/persian_diacrit/run-001/best` exists +(byt5-small per the repo docs; ~1.2GB safetensors) and is now in the +training bucket at run-073-fas-diac/best/ via the sanctioned Modal +volume export. Next: verify config/task (which objective — the 0.52% +CER eval), rebuild the eval on the current protocol (scripted in the +repo), then the standard ship chain (fp32 IMF, repack tool, publish). diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 56186d17..260e595e 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -2174,3 +2174,14 @@ attention modules. WO45 fallback fires — 3.5607/1.4588 stands as the measurement; the artifact awaits external-data-in-zip (runtime-arch). The release ships ara-diac-news-1.0 + 1.1 + urd-g2p-base-1.0, all verified. + +RELEASE VERIFICATION CATCH (2026-10-14): post-publish smoke of npm +5.8.0 found imf.firstAlternates UNREACHABLE — the imf barrel never +re-exported it (ts#108's tests used source-tree imports). Fixed in +interscript-ts#109 (barrel + public-surface test), shipped as 5.8.1 +(verified loading + export). Gem 3.1.0 smoke passed (loads, +first_alternates reachable). Also: fas-diac checkpoint CONFIRMED +alive on the Modal volume (sanctioned one-time export run; bucket +run-073-fas-diac/best) — WO47 proceeds to protocol verification. +UR lexicon lever DEAD (probe: oracle 0.2268 worse than model 0.1565). +he-wiki vocalized mining DUD (1,244 pairs) → Wikisource pivot spec'd.