From fca5c6cc9e002e1b4bc4f6d49dbbe7039acacb03 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 25 Oct 2026 19:06:45 +0800 Subject: [PATCH] docs: khm lineage recovered (secryst); persian source dossier for owner approval --- TODO.sota/66-b5-persian-registers.md | 12 ++++++++++++ TODO.sota/68-b7-khmer-lexicon.md | 11 +++++++++++ 2 files changed, 23 insertions(+) diff --git a/TODO.sota/66-b5-persian-registers.md b/TODO.sota/66-b5-persian-registers.md index d5e16a51..f49cef77 100644 --- a/TODO.sota/66-b5-persian-registers.md +++ b/TODO.sota/66-b5-persian-registers.md @@ -7,3 +7,15 @@ Method: register-stratified eval sets (build from poetry corpora, colloquial sources); probe the scar (docs 0.52% lesson: corpus-level vs row-mean declared); targeted data if registers degrade; VRR. Acceptance: register-stratified external-protocol numbers published. + + +## Source dossier (2026-10-25, for owner approval) + +| source | register | license | note | +|---|---|---|---| +| Ganjoor (ganjoor.net) | poetry (Hafez/Saadi/…) | CC BY-SA 3.0 (site-stated) | the largest digitized divan corpus; attribution + share-alike | +| fa-Wikisource | poetry + classical prose | CC BY-SA 4.0 | attributable, steady quality | +| OSCAR-fa | colloquial/web | Common-Crawl-derived (CC0-shaded tooling) | register = modern web; quality filters needed | + +Owner picks the sources -> corpora/.yaml with sha256 pins -> +the register-hold arm (target: CER < 0.7 across registers). diff --git a/TODO.sota/68-b7-khmer-lexicon.md b/TODO.sota/68-b7-khmer-lexicon.md index 1d0e182c..f14dc9e0 100644 --- a/TODO.sota/68-b7-khmer-lexicon.md +++ b/TODO.sota/68-b7-khmer-lexicon.md @@ -26,3 +26,14 @@ prefix conditioning); fp16 zip hygiene (khm fp32 corruption lesson); conformance fixtures for the new artifact. Acceptance: gates green; hit-rate + CER on held-out; release via the strict publish path. + + +## Lineage recovery (2026-10-25) + +khm-latn-1.0 traces to the **secryst project** +(`~/src/interscript/secryst`; README documents +`scripts/export_onnx_byt5.py khm-latn-byt5.zip`). +The training corpus is re-derivable from the secryst repo's data +pipeline + khmer-dict-spice — the recovery question is now concrete: +pin the exact secryst-era corpus files, run them through +corpora/khm-latn.yaml provenance, and the retrain is unblocked.