diff --git a/TODO.sota/57-a1-authority-completeness.md b/TODO.sota/57-a1-authority-completeness.md index c51c454e..4c76b741 100644 --- a/TODO.sota/57-a1-authority-completeness.md +++ b/TODO.sota/57-a1-authority-completeness.md @@ -1,6 +1,6 @@ # WO57 — A1: authority-completeness audit -Status: PLANNED · Plane A +Status: IN_PROGRESS (phase 1 DONE 2026-10-21: docs/coverage ledger + three authority diffs — alalc 30/47, iso, bgnpcgn 17 gaps; phase 2 = encode the gaps) Goal: enumerate the UNIVERSE of authority romanization/transliteration systems and diff it against the 293 encoded in `maps/`. Every gap is either encoded or cited out-of-scope with an authority reference. diff --git a/TODO.sota/58-a2-invertibility.md b/TODO.sota/58-a2-invertibility.md index 6fcd579c..63399052 100644 --- a/TODO.sota/58-a2-invertibility.md +++ b/TODO.sota/58-a2-invertibility.md @@ -1,6 +1,6 @@ # WO58 — A2: invertibility ledger + reverse maps -Status: PLANNED · Plane A +Status: IN_PROGRESS (phase 1 DONE 2026-10-21: docs/invertibility ledger — 10 pairs probed, 1 lossless, 39/278 collisions + probe tool; phase 2 = reverse-map generation) Goal: for every system, determine whether forward∘reverse = identity; publish the loss conditions (homophone mergers, digraph ambiguity, case conventions, vowel elision). diff --git a/TODO.sota/59-a3-equivalence.md b/TODO.sota/59-a3-equivalence.md index 10456c06..d88d01e1 100644 --- a/TODO.sota/59-a3-equivalence.md +++ b/TODO.sota/59-a3-equivalence.md @@ -1,6 +1,6 @@ # WO59 — A3: system-equivalence checker + distinguishing corpora -Status: PLANNED · Plane A +Status: MOSTLY DONE (ruby shipped #816 + self-testing CI #819; ts/py port pending) Goal: mechanically decide/estimate whether two systems coincide, and produce the minimal test corpus that distinguishes them. Method: diff --git a/TODO.sota/60-a4-is1-semantics.md b/TODO.sota/60-a4-is1-semantics.md index aeccc3c8..86b14e04 100644 --- a/TODO.sota/60-a4-is1-semantics.md +++ b/TODO.sota/60-a4-is1-semantics.md @@ -1,6 +1,6 @@ # WO60 — A4: IS-1 v2 — denotational semantics -Status: PLANNED · Plane A +Status: IN_PROGRESS (9 rulings frozen #818/#820/#821; remaining: metadata/tests, item algebra, charset, funcall — see WO80 inventory) Goal: complete the ISC specification with a semantics section so every runtime behavior is derivable from the document; zero open rulings. Method: diff --git a/TODO.sota/62-b1-dispatch-doctrine.md b/TODO.sota/62-b1-dispatch-doctrine.md index 8e5ca856..21b9bad4 100644 --- a/TODO.sota/62-b1-dispatch-doctrine.md +++ b/TODO.sota/62-b1-dispatch-doctrine.md @@ -1,6 +1,6 @@ # WO62 — B1: the dispatch doctrine as registry defaults -Status: PLANNED · Plane B +Status: IN_PROGRESS (doctrine written: WO56 structural rules + E2 paper; registry-as-config pending) Goal: encode the technique-dispatch table — (script class, ambiguity type, data regime) → recipe — as registry defaults so a new language gets a recommended recipe automatically. diff --git a/TODO.sota/64-b3-arabic-sota.md b/TODO.sota/64-b3-arabic-sota.md index 243c560b..cbf6e79a 100644 --- a/TODO.sota/64-b3-arabic-sota.md +++ b/TODO.sota/64-b3-arabic-sota.md @@ -1,6 +1,6 @@ # WO64 — B3: Arabic to external SOTA (DER < 2.0, WikiNews < 8) -Status: PLANNED · launchable now +Status: IN_PROGRESS (r8 stage 2 training 2026-10-22, stage-3 SadeedDiac eval auto-chained; VRR closure = WO81) Goal: SadeedDiac DER < 2.0 (canonical 2.29), WikiNews multiref WER < 8 (news specialist 8.85). VRR reported. Method: r8 noisy-student chain — synthesize with the canonical r7 diff --git a/TODO.sota/65-b4-urdu-sota.md b/TODO.sota/65-b4-urdu-sota.md index 7d0802a5..27487421 100644 --- a/TODO.sota/65-b4-urdu-sota.md +++ b/TODO.sota/65-b4-urdu-sota.md @@ -1,6 +1,6 @@ # WO65 — B4: Urdu — diac CER < 3; G2P license path -Status: PLANNED · G2P half BLOCKED-OWNER (commercial-license data) +Status: IN_PROGRESS · G2P half BLOCKED-OWNER (commercial-license data); diac push 3.57 → < 3 pending Goal: urd-diac 3.57 → CER < 3; urd-g2p research-only → commercial- usable data lineage. Method: diac — oracle relabel arms only if teachers decorrelate (banked diff --git a/TODO.sota/67-b6-thai-lexicon.md b/TODO.sota/67-b6-thai-lexicon.md index 501fb843..acad8b10 100644 --- a/TODO.sota/67-b6-thai-lexicon.md +++ b/TODO.sota/67-b6-thai-lexicon.md @@ -1,6 +1,6 @@ # WO67 — B6: Thai — lexicon-first, three layers (PER < 2) -Status: IN_PROGRESS (lexicon phase landed 2026-10-21) +Status: IN_PROGRESS (RTGS encodings landed maps#204; lexicon corpus staged; training arm + IPA layers = WO82 pending) Lexicon phase DONE: corpora/tha-phonemic.yaml -- 28,644 train / 3,225 test (word -> phonemic-with-tones pairs, ORST vocabulary x pythainlp royin engine); 6,291 engine-corrupt rows filtered and documented; diff --git a/TODO.sota/68-b7-khmer-lexicon.md b/TODO.sota/68-b7-khmer-lexicon.md index 8d1a28f7..5b04835a 100644 --- a/TODO.sota/68-b7-khmer-lexicon.md +++ b/TODO.sota/68-b7-khmer-lexicon.md @@ -1,6 +1,6 @@ # WO68 — B7: Khmer — Chuon Nath lexicon conditioning (CER < 15) -Status: IN_PROGRESS (phase 1 landed 2026-10-21; phase 2 = training arm) +Status: IN_PROGRESS (phase 1 landed 2026-10-21; phase 2 = training arm launch) Phase 1 DONE: khm-dict corpus built + pinned (corpora/khm-dict.yaml) — 3,261 gold Chuon-Nath (word→IPA) pairs, 326-row holdout, 72,048 SBBIC vocab rows; staged at bucket Interscript/isx-training/run-080-khm-lexicon diff --git a/TODO.sota/69-b8-community-intake.md b/TODO.sota/69-b8-community-intake.md index 6752a81e..cb7444e4 100644 --- a/TODO.sota/69-b8-community-intake.md +++ b/TODO.sota/69-b8-community-intake.md @@ -1,6 +1,6 @@ # WO69 — B8: prove the contribution story (3 community languages) -Status: PLANNED +Status: PLANNED (intake doc = WO86) Goal: three external-contributor languages go end-to-end through the 2-YAML path (corpora + recipe), landing as shipped models without framework forks. diff --git a/TODO.sota/70-b9-external-benchmark-cert.md b/TODO.sota/70-b9-external-benchmark-cert.md index 1ce0c638..10012f6f 100644 --- a/TODO.sota/70-b9-external-benchmark-cert.md +++ b/TODO.sota/70-b9-external-benchmark-cert.md @@ -1,6 +1,6 @@ # WO70 — B9: external-benchmark certification -Status: PLANNED +Status: IN_PROGRESS (SadeedDiac protocol shipped as interscript-ml-tools; protocol-package consolidation pending) Goal: no self-graded SOTA claims — every headline reconciles to an external leaderboard/protocol package. Method: per language, pin the external protocol (Arabic: SadeedDiac + diff --git a/TODO.sota/71-c1-conformance-matrix.md b/TODO.sota/71-c1-conformance-matrix.md index 6d43d4a4..ca15c5c0 100644 --- a/TODO.sota/71-c1-conformance-matrix.md +++ b/TODO.sota/71-c1-conformance-matrix.md @@ -1,6 +1,6 @@ # WO71 — C1: model × runtime conformance matrix in CI -Status: PLANNED +Status: DONE (all gates green, sustained: 3-compiler CI 16,482+8,238/0, six legs; model x runtime byte-exact, CI-enforced) Goal: every shipped model byte-exact in ruby/ts/py, CI-enforced, forever. Method: fixtures already frozen in conformance/ (12 rows each; sidecar on the release). Wire each runtime's CI to download the fixture set diff --git a/TODO.sota/72-c2-metric-convention-gate.md b/TODO.sota/72-c2-metric-convention-gate.md index 43babba2..dbb34c0b 100644 --- a/TODO.sota/72-c2-metric-convention-gate.md +++ b/TODO.sota/72-c2-metric-convention-gate.md @@ -1,6 +1,6 @@ # WO72 — C2: gates reject undeclared metric conventions -Status: PLANNED +Status: DONE (anchor law enforced in src/pipeline/contracts.py; undeclared conventions rejected) Goal: quality gates refuse a metric that doesn't declare its convention (row-mean vs corpus-level) and anchor. Method: extend the gate registry (WO55 pattern): MetricDeclaration diff --git a/TODO.sota/73-c3-perf-budgets.md b/TODO.sota/73-c3-perf-budgets.md index 34c4ec5b..67299551 100644 --- a/TODO.sota/73-c3-perf-budgets.md +++ b/TODO.sota/73-c3-perf-budgets.md @@ -1,6 +1,6 @@ # WO73 — C3: performance budgets in CI -Status: PLANNED +Status: IN_PROGRESS (BudgetGate size check DONE; decode-latency budget = remainder) Goal: decode latency + artifact size budgets enforced so regressions block like test failures. Method: measure current shipped baselines (single-row greedy, p50/p95 diff --git a/TODO.sota/76-e1-paper-formalism.md b/TODO.sota/76-e1-paper-formalism.md index e33dfbe8..83d1314e 100644 --- a/TODO.sota/76-e1-paper-formalism.md +++ b/TODO.sota/76-e1-paper-formalism.md @@ -1,6 +1,6 @@ # WO76 — E1: paper — the ISC formalism, corpus, and interoperability -Status: PLANNED · Depends WO57/WO60 +Status: IN_PROGRESS (draft merged #333; camera-ready needs A4 completion) Goal: a complete draft (LREC/COLING-class) presenting ISC as an executable formalism for authority script conversion; the 293-system corpus; the differential-testing interoperability methodology; the diff --git a/TODO.sota/77-e2-paper-dispatch.md b/TODO.sota/77-e2-paper-dispatch.md index 8a09f75b..2362f584 100644 --- a/TODO.sota/77-e2-paper-dispatch.md +++ b/TODO.sota/77-e2-paper-dispatch.md @@ -1,6 +1,6 @@ # WO77 — E2: paper — technique dispatch for underdetermined mappings -Status: PLANNED · Depends WO62 + plane B results +Status: IN_PROGRESS (draft merged #334; results matrix needs plane B final numbers) Goal: a complete draft presenting the dispatch doctrine: ambiguity class → technique, with the negative-results registry (RL, CTC, LLM-teachers, correlated distillation) as a first-class contribution diff --git a/TODO.sota/78-e3-benchmark-artifacts.md b/TODO.sota/78-e3-benchmark-artifacts.md index e64cf786..e509c82f 100644 --- a/TODO.sota/78-e3-benchmark-artifacts.md +++ b/TODO.sota/78-e3-benchmark-artifacts.md @@ -1,6 +1,6 @@ # WO78 — E3: frozen benchmarks as citable artifacts -Status: PLANNED +Status: IN_PROGRESS (frontier-predictions-v1 shipped; conformance-suite bundle pending) Goal: eval protocols + conformance suites published as citable, versioned artifacts. Method: protocol packages (WO70) + conformance vectors bundled as diff --git a/TODO.sota/81-vrr-metric.md b/TODO.sota/81-vrr-metric.md new file mode 100644 index 00000000..6ab55c69 --- /dev/null +++ b/TODO.sota/81-vrr-metric.md @@ -0,0 +1,29 @@ +# WO81 — VRR: valid-reading-rate, the ambiguity-aware metric + +Status: PLANNED · Plane B (B3 closure requirement, WO56 structural rule 1) + +Single-gold DER systematically undercounts legitimate ambiguity: when a +skeleton has several dictionary-licensed readings, scoring against one +gold penalizes a correct alternative. VRR scores a prediction as valid +when it matches ANY licensed reading. + +Goal: VRR reported alongside DER for every Arabic (then Hebrew/Urdu) +headline claim; the first number lands for `ara-diac-2.0` on +SadeedDiac-25. + +Method: +- Build the licensed-reading lexicon: skeleton (undiacritized form) → + set of fully-vocalized forms, extracted from the pinned vocalized + training corpora (provenance in `corpora/*.yaml`); dedupe by exact + diacritic pattern. +- Implement `score_predictions_vrr` + CLI in interscript-ml-tools: + VRR = |{pred form ∈ licensed(skeleton)}| / |rows|, computed per row + and corpus-level, convention-declared (row-level match, exact + including diacritics). +- Run against the frozen predictions in the + `frontier-predictions-v1` release — no model calls needed. +- Record in `models.yaml` as a second metric block (protocol + anchor + per the C2 anchor law). + +Acceptance: VRR tool shipped; `ara-diac-2.0` VRR published next to +DER 2.29; the metric convention declared and gated. diff --git a/TODO.sota/82-thai-phoneme-layers.md b/TODO.sota/82-thai-phoneme-layers.md new file mode 100644 index 00000000..348bf2db --- /dev/null +++ b/TODO.sota/82-thai-phoneme-layers.md @@ -0,0 +1,25 @@ +# WO82 — Thai IPA and IPA+tone layers (the TTS layer) + +Status: PLANNED · Plane B (B6 layers 2–3; layer 1 RTGS shipped in maps#204) + +WO56 requires three separately-tracked Thai target layers: RTGS +(toneless, DONE — deterministic .isc encodings), IPA (toneless), and +IPA+tone (TTS). The phoneme layers need a tonal data supply that is not +yet identified. + +Goal: `tha-phonemic` eval reporting IPA layer v0 and IPA+tone v0 with +tone accuracy; published PER < 2 on the RTGS layer stands from B6. + +Method: +- Data supply (OWNER ASK): Royal Institute dictionary entries carry + pronunciation + tone; identify the machine-readable source and its + license (the 28,644-row staged lexicon covers orthographic→phonemic + without tone marks — audit whether tone is recoverable from it before + asking for more data). +- Learned-IPA fallback for novel tokens: ByT5 arm conditioned per the + lexicon-first doctrine (hit-rate first-class). +- Tone accuracy metric: consonant-class × syllable-type computed tone + vs lexicon tone, declared per C2. + +Acceptance: three layers tracked in models.yaml; IPA+tone v0 shipped +with tone accuracy reported; RTGS layer unchanged. diff --git a/TODO.sota/83-exporter-io-defaults.md b/TODO.sota/83-exporter-io-defaults.md new file mode 100644 index 00000000..15ae96e3 --- /dev/null +++ b/TODO.sota/83-exporter-io-defaults.md @@ -0,0 +1,24 @@ +# WO83 — Exporter defaults: fp32 graph IO always + +Status: PLANNED · Plane B/C (WO79 follow-through — the decision is +recorded, the DEFAULT is not enforced) + +WO79's owner decision: exporter-side fp16→fp32 graph-IO normalization. +The fix exists as a post-hoc tool (`tools/normalize_fp16_io.py`) and +was applied to urd/fas by hand. Future exports can still ship raw-fp16 +boundaries unless the exporter itself defaults to normalized IO. + +Goal: no export path can produce an artifact whose graph boundary is +fp16; the normalizer becomes the exporter's default, not a repair step. + +Method: +- Fold the IO-normalization into `export_imf_hf.py` (interscript-train): + weights may stay fp16; graph outputs always Cast(to=FLOAT), fp16 + decoder inputs narrowed via internal Cast, raw tensors internal. +- Publish guard: reject (or auto-normalize + record) any artifact whose + graph IO carries fp16 — enforced in `src/pipeline/`, not by memory. +- Verification: one fresh export decodes reference-identically (torch + vs artifact, ≥500 samples in shipped decode mode). + +Acceptance: exporter default normalized; publish guard blocks raw-fp16 +IO; a fresh export passes parity unchanged. diff --git a/TODO.sota/84-train-repo-hygiene.md b/TODO.sota/84-train-repo-hygiene.md new file mode 100644 index 00000000..9c9cd084 --- /dev/null +++ b/TODO.sota/84-train-repo-hygiene.md @@ -0,0 +1,25 @@ +# WO84 — interscript-train: land the local uncommitted work + +Status: PLANNED · hygiene (discovered 2026-10-22; durability lives only +on this laptop) + +The local checkout of `training/wo31-wo36-campaign-scripts` carries +uncommitted improvements that production runs already depend on +(r8_noisy_student_hf.py's self-contained decode + sync_to_bucket +durability, C-Levenshtein eval path), plus untracked research docs and +utilities. One `/tmp`-style loss and the running campaign's durability +is unexplained by any commit. + +Goal: `git status` clean on the campaign branch lineage; every behavior +a running job depends on exists in a pushed commit. + +Method: +- Branch from `training/wo31-wo36-campaign-scripts`. +- Commit the five modified scripts with per-change messages (explicit + paths only — never `git add -A`). +- Triage untracked: `TODO.research/*` docs, `conformance_runtime_check.py`, + `classes.json` → commit if source; `dist/` → never (build output). +- PR into the campaign lineage; merge after green. + +Acceptance: pushed branch + merged PR; running jobs' behavior matches +a commit; worktree clean. diff --git a/TODO.sota/85-ledger-status-sweep.md b/TODO.sota/85-ledger-status-sweep.md new file mode 100644 index 00000000..6ee05e41 --- /dev/null +++ b/TODO.sota/85-ledger-status-sweep.md @@ -0,0 +1,16 @@ +# WO85 — Ledger status sweep: headers must match verified reality + +Status: DONE (2026-10-22 sweep in the same PR that adds WO81–86) + +The `Status:` headers of WO57–78 lagged execution: C1/C2 recorded +PLANNED while all gates were green; A3/E1/E2 recorded PLANNED while +merged. A ledger that lies about state is worse than no ledger. + +Goal: every WO header reflects the verified state at sweep time; +sweeps re-run whenever a plane milestone lands. + +Method: sweep headers against evidence (merged PRs, CI, gates) — +never against intention. Dated. + +Acceptance: this file records the sweep; headers current as of +2026-10-22. diff --git a/TODO.sota/86-b8-intake-doc.md b/TODO.sota/86-b8-intake-doc.md new file mode 100644 index 00000000..dfd569b0 --- /dev/null +++ b/TODO.sota/86-b8-intake-doc.md @@ -0,0 +1,25 @@ +# WO86 — B8 intake: the community language path, documented + +Status: PLANNED · Plane B + +B8's proof is three outside languages through the 2-YAML path — but +there is no published onboarding document; the path exists only in +maintainer heads and CLAUDE.md notes. + +Goal: a contributor can onboard a new language without reading the +pipeline: what to bring (authority reference or corpus + license), the +two YAML shapes, the eval-set expectation, the gates their PR must +pass, and how a learned component slots in if the table +underdetermines. + +Method: +- `docs/contributing-language.md` in interscript-models: the 2-YAML + contract (`recipes/`, `corpora/`), provenance law, gate walkthrough + (quality/parity/license/conformance/regression), worked example + end-to-end (smallest shipped map as the template). +- Mirror a short pointer from the maps repo README. +- The path itself needs no code change — if it does, that is a + framework bug: record it, do not patch around it. + +Acceptance: doc merged; one dry-run onboard of a trivial language +through the gates using only the doc.