diff --git a/TODO.sota/64-b3-arabic-sota.md b/TODO.sota/64-b3-arabic-sota.md index 8a357e7..694aeed 100644 --- a/TODO.sota/64-b3-arabic-sota.md +++ b/TODO.sota/64-b3-arabic-sota.md @@ -26,3 +26,28 @@ the news specialist; decode policy unchanged (greedy, first-alternates). HF Jobs, --detach, --timeout 23h, step ckpts; ≤2 GPU budget shared with B7. Acceptance: gates green + external protocol reconciliation (B9). + + +## Oracle-distill launch scoping (2026-10-29) + +The WO39 chain is script-complete and ready to run: stage 1 +`distill_oracle_hf.py` (300K QCRI rows, three arms, per-row oracle) and +stage 2 `continue_oracle_train_hf.py` (continue-train run-047/best 2 epochs, +SadeedDiac + WikiNews eval, gate first-alt WER < 3.5607, stretch <= 2.70). + +Sole launch prerequisite: staging the three WO35 arm checkpoints +(run-047-large-strip/best — also the stage-2 init, run-040-strip-alt/best, +run-029-news-specialist/best) and the QCRI train_data mount onto HF storage. +Findings of the audit: + +- The legacy Modal volumes (secryst-checkpoints et al.) list only + thai/rababa-era artifacts — no run-047/040/029. +- rhtse/jobs-artifacts holds 78 timestamped job-sync dirs (r8 era) — + candidate location, unverified. +- Recovery options if the checkpoints are gone: (a) the news specialist arm + is SHIPPED (ara-diac-news-1.0/1.1) and the r7 canonical teacher is SHIPPED + (ara-diac-2.0 lineage) — two of three arms restageable from pinned + releases; (b) run-047-large-strip/best (the stage-2 init) may need a + short re-derivation from its recorded recipe, or the stage-2 init moves + to the closest recoverable ancestor — an owner-visible design decision + recorded before any run. diff --git a/corpora/khm-latn.yaml b/corpora/khm-latn.yaml index 43379ce..1d982e5 100644 --- a/corpora/khm-latn.yaml +++ b/corpora/khm-latn.yaml @@ -1,16 +1,21 @@ id: khm-latn task: transliteration sources: - # WO88 migration: the secryst-era khm-latn training lineage, recorded - # with honest provenance markers — the original corpus predates the - # pinning contract. Decode evidence + the khmer-dict-spice source - # (Chuon Nath + SBBIC) are documented; the exact training tsv/tsv - # generation script lives in the secryst repo - # (scripts/export_onnx_byt5.py lineage). UNVERIFIED fields are the - # retrain blockers listed in WO68/88. + # WO68 lineage recovery (2026-10-29): the secryst-era khm-latn training + # corpus identified exactly as secryst/data-khmer-translit — 17,911 word + # pairs in two parallel single-column CSVs. Staged to the training bucket + # with sha256; the legacy raw.githubusercontent fetch is retired. + # Upstream repo carries no license file (predecessor project's own); + # the ship gate stays owner-held until the license_review record lands. - kind: repo - id: secryst (~/src/interscript/secryst lineage) - license: UNVERIFIED - provenance: UNVERIFIED — recover the exact training corpus before any retrain + id: secryst/data-khmer-translit + license: UNVERIFIED — predecessor repo, no license file; owner decision pending + provenance: https://github.com/secryst/data-khmer-translit (master at recovery, 2026-10-29) splits: - train: {path: UNVERIFIED, rows: 0, sha256: UNVERIFIED} + train: {path: hf://buckets/Interscript/isx-training/khmer-translit/run-081-recovery/train.jsonl, rows: 16120, sha256: SEE-RUN} + val: {path: hf://buckets/Interscript/isx-training/khmer-translit/run-081-recovery/val.jsonl, rows: 895, sha256: SEE-RUN} + test: {path: hf://buckets/Interscript/isx-training/khmer-translit/run-081-recovery/test.jsonl, rows: 895, sha256: SEE-RUN} +raw: + data_kh.csv: {sha256: 48f9924ac50c941c1fd52a7e5a9152e4478ba6d9679e1732333ed9f76b598a81, path: hf://buckets/Interscript/isx-training/khmer-translit/data_kh.csv, rows: 17911} + data_rom.csv: {sha256: 7dff342f4ed214c05d147e71baa10482d80f4ff6d182ee15a2abbdbafb6bc71b, path: hf://buckets/Interscript/isx-training/khmer-translit/data_rom.csv, rows: 17911} +recovery_run: run-081-khm-latn-recovery (HF job rhtse/6ae32acd41e90535df78a449, seed 42, recipe = legacy run-001)