Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion TODO.sota/57-a1-authority-completeness.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO57 — A1: authority-completeness audit

Status: PLANNED · Plane A
Status: IN_PROGRESS (phase 1 DONE 2026-10-21: docs/coverage ledger + three authority diffs — alalc 30/47, iso, bgnpcgn 17 gaps; phase 2 = encode the gaps)
Goal: enumerate the UNIVERSE of authority romanization/transliteration
systems and diff it against the 293 encoded in `maps/`. Every gap is
either encoded or cited out-of-scope with an authority reference.
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/58-a2-invertibility.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO58 — A2: invertibility ledger + reverse maps

Status: PLANNED · Plane A
Status: IN_PROGRESS (phase 1 DONE 2026-10-21: docs/invertibility ledger — 10 pairs probed, 1 lossless, 39/278 collisions + probe tool; phase 2 = reverse-map generation)
Goal: for every system, determine whether forward∘reverse = identity;
publish the loss conditions (homophone mergers, digraph ambiguity,
case conventions, vowel elision).
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/59-a3-equivalence.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO59 — A3: system-equivalence checker + distinguishing corpora

Status: PLANNED · Plane A
Status: MOSTLY DONE (ruby shipped #816 + self-testing CI #819; ts/py port pending)
Goal: mechanically decide/estimate whether two systems coincide, and
produce the minimal test corpus that distinguishes them.
Method:
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/60-a4-is1-semantics.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO60 — A4: IS-1 v2 — denotational semantics

Status: PLANNED · Plane A
Status: IN_PROGRESS (9 rulings frozen #818/#820/#821; remaining: metadata/tests, item algebra, charset, funcall — see WO80 inventory)
Goal: complete the ISC specification with a semantics section so every
runtime behavior is derivable from the document; zero open rulings.
Method:
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/62-b1-dispatch-doctrine.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO62 — B1: the dispatch doctrine as registry defaults

Status: PLANNED · Plane B
Status: IN_PROGRESS (doctrine written: WO56 structural rules + E2 paper; registry-as-config pending)
Goal: encode the technique-dispatch table — (script class, ambiguity
type, data regime) → recipe — as registry defaults so a new language
gets a recommended recipe automatically.
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/64-b3-arabic-sota.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO64 — B3: Arabic to external SOTA (DER < 2.0, WikiNews < 8)

Status: PLANNED · launchable now
Status: IN_PROGRESS (r8 stage 2 training 2026-10-22, stage-3 SadeedDiac eval auto-chained; VRR closure = WO81)
Goal: SadeedDiac DER < 2.0 (canonical 2.29), WikiNews multiref WER < 8
(news specialist 8.85). VRR reported.
Method: r8 noisy-student chain — synthesize with the canonical r7
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/65-b4-urdu-sota.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO65 — B4: Urdu — diac CER < 3; G2P license path

Status: PLANNED · G2P half BLOCKED-OWNER (commercial-license data)
Status: IN_PROGRESS · G2P half BLOCKED-OWNER (commercial-license data); diac push 3.57 → < 3 pending
Goal: urd-diac 3.57 → CER < 3; urd-g2p research-only → commercial-
usable data lineage.
Method: diac — oracle relabel arms only if teachers decorrelate (banked
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/67-b6-thai-lexicon.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO67 — B6: Thai — lexicon-first, three layers (PER < 2)

Status: IN_PROGRESS (lexicon phase landed 2026-10-21)
Status: IN_PROGRESS (RTGS encodings landed maps#204; lexicon corpus staged; training arm + IPA layers = WO82 pending)
Lexicon phase DONE: corpora/tha-phonemic.yaml -- 28,644 train / 3,225
test (word -> phonemic-with-tones pairs, ORST vocabulary x pythainlp
royin engine); 6,291 engine-corrupt rows filtered and documented;
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/68-b7-khmer-lexicon.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO68 — B7: Khmer — Chuon Nath lexicon conditioning (CER < 15)

Status: IN_PROGRESS (phase 1 landed 2026-10-21; phase 2 = training arm)
Status: IN_PROGRESS (phase 1 landed 2026-10-21; phase 2 = training arm launch)
Phase 1 DONE: khm-dict corpus built + pinned (corpora/khm-dict.yaml) —
3,261 gold Chuon-Nath (word→IPA) pairs, 326-row holdout, 72,048 SBBIC
vocab rows; staged at bucket Interscript/isx-training/run-080-khm-lexicon
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/69-b8-community-intake.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO69 — B8: prove the contribution story (3 community languages)

Status: PLANNED
Status: PLANNED (intake doc = WO86)
Goal: three external-contributor languages go end-to-end through the
2-YAML path (corpora + recipe), landing as shipped models without
framework forks.
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/70-b9-external-benchmark-cert.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO70 — B9: external-benchmark certification

Status: PLANNED
Status: IN_PROGRESS (SadeedDiac protocol shipped as interscript-ml-tools; protocol-package consolidation pending)
Goal: no self-graded SOTA claims — every headline reconciles to an
external leaderboard/protocol package.
Method: per language, pin the external protocol (Arabic: SadeedDiac +
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/71-c1-conformance-matrix.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO71 — C1: model × runtime conformance matrix in CI

Status: PLANNED
Status: DONE (all gates green, sustained: 3-compiler CI 16,482+8,238/0, six legs; model x runtime byte-exact, CI-enforced)
Goal: every shipped model byte-exact in ruby/ts/py, CI-enforced, forever.
Method: fixtures already frozen in conformance/ (12 rows each; sidecar
on the release). Wire each runtime's CI to download the fixture set
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/72-c2-metric-convention-gate.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO72 — C2: gates reject undeclared metric conventions

Status: PLANNED
Status: DONE (anchor law enforced in src/pipeline/contracts.py; undeclared conventions rejected)
Goal: quality gates refuse a metric that doesn't declare its convention
(row-mean vs corpus-level) and anchor.
Method: extend the gate registry (WO55 pattern): MetricDeclaration
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/73-c3-perf-budgets.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO73 — C3: performance budgets in CI

Status: PLANNED
Status: IN_PROGRESS (BudgetGate size check DONE; decode-latency budget = remainder)
Goal: decode latency + artifact size budgets enforced so regressions
block like test failures.
Method: measure current shipped baselines (single-row greedy, p50/p95
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/76-e1-paper-formalism.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO76 — E1: paper — the ISC formalism, corpus, and interoperability

Status: PLANNED · Depends WO57/WO60
Status: IN_PROGRESS (draft merged #333; camera-ready needs A4 completion)
Goal: a complete draft (LREC/COLING-class) presenting ISC as an
executable formalism for authority script conversion; the 293-system
corpus; the differential-testing interoperability methodology; the
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/77-e2-paper-dispatch.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO77 — E2: paper — technique dispatch for underdetermined mappings

Status: PLANNED · Depends WO62 + plane B results
Status: IN_PROGRESS (draft merged #334; results matrix needs plane B final numbers)
Goal: a complete draft presenting the dispatch doctrine: ambiguity
class → technique, with the negative-results registry (RL, CTC,
LLM-teachers, correlated distillation) as a first-class contribution
Expand Down
2 changes: 1 addition & 1 deletion TODO.sota/78-e3-benchmark-artifacts.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WO78 — E3: frozen benchmarks as citable artifacts

Status: PLANNED
Status: IN_PROGRESS (frontier-predictions-v1 shipped; conformance-suite bundle pending)
Goal: eval protocols + conformance suites published as citable,
versioned artifacts.
Method: protocol packages (WO70) + conformance vectors bundled as
Expand Down
29 changes: 29 additions & 0 deletions TODO.sota/81-vrr-metric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# WO81 — VRR: valid-reading-rate, the ambiguity-aware metric

Status: PLANNED · Plane B (B3 closure requirement, WO56 structural rule 1)

Single-gold DER systematically undercounts legitimate ambiguity: when a
skeleton has several dictionary-licensed readings, scoring against one
gold penalizes a correct alternative. VRR scores a prediction as valid
when it matches ANY licensed reading.

Goal: VRR reported alongside DER for every Arabic (then Hebrew/Urdu)
headline claim; the first number lands for `ara-diac-2.0` on
SadeedDiac-25.

Method:
- Build the licensed-reading lexicon: skeleton (undiacritized form) →
set of fully-vocalized forms, extracted from the pinned vocalized
training corpora (provenance in `corpora/*.yaml`); dedupe by exact
diacritic pattern.
- Implement `score_predictions_vrr` + CLI in interscript-ml-tools:
VRR = |{pred form ∈ licensed(skeleton)}| / |rows|, computed per row
and corpus-level, convention-declared (row-level match, exact
including diacritics).
- Run against the frozen predictions in the
`frontier-predictions-v1` release — no model calls needed.
- Record in `models.yaml` as a second metric block (protocol + anchor
per the C2 anchor law).

Acceptance: VRR tool shipped; `ara-diac-2.0` VRR published next to
DER 2.29; the metric convention declared and gated.
25 changes: 25 additions & 0 deletions TODO.sota/82-thai-phoneme-layers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# WO82 — Thai IPA and IPA+tone layers (the TTS layer)

Status: PLANNED · Plane B (B6 layers 2–3; layer 1 RTGS shipped in maps#204)

WO56 requires three separately-tracked Thai target layers: RTGS
(toneless, DONE — deterministic .isc encodings), IPA (toneless), and
IPA+tone (TTS). The phoneme layers need a tonal data supply that is not
yet identified.

Goal: `tha-phonemic` eval reporting IPA layer v0 and IPA+tone v0 with
tone accuracy; published PER < 2 on the RTGS layer stands from B6.

Method:
- Data supply (OWNER ASK): Royal Institute dictionary entries carry
pronunciation + tone; identify the machine-readable source and its
license (the 28,644-row staged lexicon covers orthographic→phonemic
without tone marks — audit whether tone is recoverable from it before
asking for more data).
- Learned-IPA fallback for novel tokens: ByT5 arm conditioned per the
lexicon-first doctrine (hit-rate first-class).
- Tone accuracy metric: consonant-class × syllable-type computed tone
vs lexicon tone, declared per C2.

Acceptance: three layers tracked in models.yaml; IPA+tone v0 shipped
with tone accuracy reported; RTGS layer unchanged.
24 changes: 24 additions & 0 deletions TODO.sota/83-exporter-io-defaults.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# WO83 — Exporter defaults: fp32 graph IO always

Status: PLANNED · Plane B/C (WO79 follow-through — the decision is
recorded, the DEFAULT is not enforced)

WO79's owner decision: exporter-side fp16→fp32 graph-IO normalization.
The fix exists as a post-hoc tool (`tools/normalize_fp16_io.py`) and
was applied to urd/fas by hand. Future exports can still ship raw-fp16
boundaries unless the exporter itself defaults to normalized IO.

Goal: no export path can produce an artifact whose graph boundary is
fp16; the normalizer becomes the exporter's default, not a repair step.

Method:
- Fold the IO-normalization into `export_imf_hf.py` (interscript-train):
weights may stay fp16; graph outputs always Cast(to=FLOAT), fp16
decoder inputs narrowed via internal Cast, raw tensors internal.
- Publish guard: reject (or auto-normalize + record) any artifact whose
graph IO carries fp16 — enforced in `src/pipeline/`, not by memory.
- Verification: one fresh export decodes reference-identically (torch
vs artifact, ≥500 samples in shipped decode mode).

Acceptance: exporter default normalized; publish guard blocks raw-fp16
IO; a fresh export passes parity unchanged.
25 changes: 25 additions & 0 deletions TODO.sota/84-train-repo-hygiene.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# WO84 — interscript-train: land the local uncommitted work

Status: PLANNED · hygiene (discovered 2026-10-22; durability lives only
on this laptop)

The local checkout of `training/wo31-wo36-campaign-scripts` carries
uncommitted improvements that production runs already depend on
(r8_noisy_student_hf.py's self-contained decode + sync_to_bucket
durability, C-Levenshtein eval path), plus untracked research docs and
utilities. One `/tmp`-style loss and the running campaign's durability
is unexplained by any commit.

Goal: `git status` clean on the campaign branch lineage; every behavior
a running job depends on exists in a pushed commit.

Method:
- Branch from `training/wo31-wo36-campaign-scripts`.
- Commit the five modified scripts with per-change messages (explicit
paths only — never `git add -A`).
- Triage untracked: `TODO.research/*` docs, `conformance_runtime_check.py`,
`classes.json` → commit if source; `dist/` → never (build output).
- PR into the campaign lineage; merge after green.

Acceptance: pushed branch + merged PR; running jobs' behavior matches
a commit; worktree clean.
16 changes: 16 additions & 0 deletions TODO.sota/85-ledger-status-sweep.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# WO85 — Ledger status sweep: headers must match verified reality

Status: DONE (2026-10-22 sweep in the same PR that adds WO81–86)

The `Status:` headers of WO57–78 lagged execution: C1/C2 recorded
PLANNED while all gates were green; A3/E1/E2 recorded PLANNED while
merged. A ledger that lies about state is worse than no ledger.

Goal: every WO header reflects the verified state at sweep time;
sweeps re-run whenever a plane milestone lands.

Method: sweep headers against evidence (merged PRs, CI, gates) —
never against intention. Dated.

Acceptance: this file records the sweep; headers current as of
2026-10-22.
25 changes: 25 additions & 0 deletions TODO.sota/86-b8-intake-doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# WO86 — B8 intake: the community language path, documented

Status: PLANNED · Plane B

B8's proof is three outside languages through the 2-YAML path — but
there is no published onboarding document; the path exists only in
maintainer heads and CLAUDE.md notes.

Goal: a contributor can onboard a new language without reading the
pipeline: what to bring (authority reference or corpus + license), the
two YAML shapes, the eval-set expectation, the gates their PR must
pass, and how a learned component slots in if the table
underdetermines.

Method:
- `docs/contributing-language.md` in interscript-models: the 2-YAML
contract (`recipes/`, `corpora/`), provenance law, gate walkthrough
(quality/parity/license/conformance/regression), worked example
end-to-end (smallest shipped map as the template).
- Mirror a short pointer from the maps repo README.
- The path itself needs no code change — if it does, that is a
framework bug: record it, do not patch around it.

Acceptance: doc merged; one dry-run onboard of a trivial language
through the gates using only the doc.
Loading