Skip to content

Add WO89 master remaining-encodings list and WO90 Cyrillic batch record #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 7 commits into from
Oct 28, 2026
Merged

Add WO89 master remaining-encodings list and WO90 Cyrillic batch record #377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
48599e4
docs: WO89 master remaining-encodings list; WO90 Cyrillic batch shipp…
Oct 28, 2026
5dbffdb
docs: WO91 DIN/BSI/ICS batch state — DIN 32706 shipped as maps PR #21…
Oct 28, 2026
c8e5932
docs: WO89 inventory update — DIN 32706/ISO 9985 Armenian shipped (ma…
Oct 28, 2026
ca6edb4
docs: WO91 — BS 2979 and PD 6505 reclassified to blocked-owner (scan …
Oct 28, 2026
b1cdd2d
docs: WO89 - BSI pair reclassified to X (scan-only sources, owner-gated)
Oct 28, 2026
9935700
docs: DIN 31636 Hebrew shipped as maps PR #217; Yiddish column split out
Oct 28, 2026
e3f4e68
docs: BSI pair unblocked via visual reading; DIN Yiddish + BS 2979 Ru…
Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 214 additions & 0 deletions TODO.sota/89-remaining-encodings-sota.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
# WO89 — Master list: every unencoded system, model requirement, and the graph path to SOTA

Status: PLAN (2026-10-28) — supersedes ad-hoc inventory; work orders created per batch
Goal: encode every remaining reference table in the reference collection; drive all six
Track B model lines to their SOTA targets; no system left unencoded or model-blocked
without an owner decision recorded.

## Classification legend

- **D** — deterministic: the source table fully specifies the output; one `.isc` map
achieves conformance. No model involved.
- **D+M** — map is deterministic, but real-world text carries less information than
the table assumes (unwritten vowels, polyphones, unvocalized abjad). Ship the map
now; a model companion is the recorded follow-up for unvocalized input.
- **M** — output is not a character-wise function of the input (lexical readings,
unvocalized scripts as the primary input form). Requires a model.
- **X** — blocked: no Unicode-representable source encoding, no obtainable source
document, or an owner decision is the gate. Only partial sub-items exist today
(the two raster columns inside BGN/PCGN Inuktitut, documented in PR #214).

Two tracks, never conflated:
- **Track A (conformance):** transcribe the authority's table; the map is the artifact;
CI gates = strict parse + embedded tests + cross-runtime parity.
- **Track B (SOTA):** the conversion of natural, imperfectly written text; the model
is the artifact; CI gates = external-protocol metrics + quantization delta + release.

## A. Track A inventory — 72 remaining items

### A1. BGN/PCGN — 15 systems + 1 revision (all D except noted)

Reference collection: 55 PDFs; 40 systems encoded (56 maps). Inuktitut closed in PR #214.

| # | System | lang | Class | Notes / work order |
|---|--------|------|-------|--------------------|
| 1 | Adyghe | ady | D | Cyrillic table; WO90 batch 1 |
| 2 | Avar | ava | D | Cyrillic table; WO90 batch 1 |
| 3 | Chuvash | chv | D | Cyrillic table; WO90 batch 1 |
| 4 | Kabardian | kbd | D | Cyrillic table; WO90 batch 1 |
| 5 | Karachay-Balkar | krc | D | Cyrillic table; WO90 batch 1 |
| 6 | Ossetian | oss | D | Cyrillic table; WO90 batch 1 |
| 7 | Udmurt | udm | D | Cyrillic table; WO90 batch 1 |
| 8 | Yakut | sah | D | Cyrillic table; WO90 batch 1 |
| 9 | Kurdish 2012 revision | kur | D | diff against odni-kur-Arab-Latn-2007/2011; WO90 batch 1 |
| 10 | Hebrew | heb | D+M | table is deterministic on pointed text; unvocalized input → existing [[heb-vocalized-plane]] lineage is the companion |
| 11 | Modern Syriac | aii/syc | D+M | pointed table D; primary real-world input is unvocalized → M companion later; resolve syc vs syr/aii codes at encoding |
| 12 | Burmese | mya | D | table deterministic on written form (stacked consonants are explicit); real-world D+M follow-up |
| 13 | Dzongkha | dzo | D | orthographic transliteration table |
| 14 | Lao | lao | D | table deterministic; polyphone disambiguation is a D+M follow-up |
| 15 | Shan | shn | D | table deterministic |
| 16 | Khmer | khm | D+M | table deterministic on written form; subnational vowel/polyphony → existing khm model line is the companion |

### A2. ALA-LC — 47 catalog gaps (official catalog: 78 tables; 30 encoded)

Class on the table itself; D+M noted where unvocalized/unsegmented input is the norm.

| # | System | Class | # | System | Class |
|---|--------|-------|---|--------|-------|
| 1 | Armenian (hye/arm) | D | 25 | Malay (Jawi-Arabic) | D+M |
| 2 | Balinese | D | 26 | Manchu | D |
| 3 | Batak | D | 27 | Mandekan (N'ko) | D |
| 4 | Burmese | D | 28 | Manipuri (Meitei) | D |
| 5 | Cham | D | 29 | Moplah | D+M |
| 6 | Cherokee | D | 30 | Moroccan Tamazight | D |
| 7 | Chinese (Wade-Giles) | D | 31 | Non-Slavic Cyrillic | D |
| 8 | Church Slavic | D | 32 | Ottoman Turkish | D+M |
| 9 | Coptic | D | 33 | Pushto | D+M |
| 10 | Deseret | D | 34 | Romanian (Cyrillic) | D |
| 11 | Hebrew and Yiddish | D+M | 35 | Rusyn | D |
| 12 | Inuktitut | D | 36 | Santali (Ol) | D |
| 13 | Japanese (McCune-Reischauer) | D (kana) / M (kanji readings) | 37 | Shan | D |
| 14 | Javanese/Sundanese/Madurese | D | 38 | Sindhi | D |
| 15 | Jawi-Pegon | D | 39 | Syriac | D+M |
| 16 | Judeo-Arabic | D+M | 40 | Tamashek | D |
| 17 | Kashmiri | D | 41 | Thai | D |
| 18 | Kazakh (Arabic script) | D+M | 42 | Tibetan | D |
| 19 | Khmer | D | 43 | Tod-Oirat-Old Kalmyk | D |
| 20 | Kurdish | D | 44 | Uighur | D+M |
| 21 | Ladino | D | 45 | Urdu | D+M |
| 22 | Lao | D | 46 | Uzbek | D |
| 23 | Lepcha | D | 47 | Vai | D |
| 24 | Limbu | D | | | |

### A3. DIN — 5 documents (all D)

Reference collection: 8 DIN PDFs; 4 encoded (32706, 32707, 33903, 33904).

| Doc | Class | Notes |
|-----|-------|-------|
| DIN 1460-2 | D | subject confirmed against the document at encoding |
| DIN 31635 | D | Arabic |
| DIN 31636 | D | Hebrew column SHIPPED — maps PR #217; Yiddish column still open |
| DIN 32706 | D | SHIPPED — maps PR #216 (Armenian; also closes the ISO 9985 scope gap in A6) |
| DIN 32708 | D | subject confirmed at encoding |

### A4. BSI — 2 documents (reclassified D -> X, 2026-10-28)

| Doc | Class | Notes |
|-----|-------|-------|
| BS 2979:1958 | D (via visual reading) | unblocked by owner directive; Russian column SHIPPED — maps PR #219; remaining: ukr/bel/bul/srp columns + Tables D, E, G (WO91) |
| PD 6505:1982 | D (via visual reading) | unblocked by owner directive; scan readable visually, not yet transcribed (WO91) |

### A5. ICS 630-01 — Annex D Chinese (D)

29 documents in the reference collection; Annexes encoded through WO-series
(Annex G = PR #213). Annex D (Chinese personal names, pinyin) remains.

### A6. ISO scope gaps (all D tables; scope, not new authorities)

| Gap | Class | Notes |
|-----|-------|-------|
| ISO 9 non-Russian Cyrillic coverage | D | verify per-letter coverage of the 1995 table vs corpus |
| ISO 259 Hebrew | D | letter mapping; unvocalized input is Track B |
| ISO 7098 Chinese (pinyin) | D | large per-character table |
| ISO 9985 Armenian | D | closed by DIN 32706 map (maps PR #216), which is aligned with ISO 9985:1996 |
| ISO 15919 Indic remainder | D | enumerate the script set vs corpus; encode the missing scripts |

### A7. Source-acquisition diffs

| Item | Class |
|------|-------|
| ICAO systems: obtain current source docs, diff vs corpus | D |
| UNGEGN systems: obtain source docs, diff vs corpus | D |

**Track A totals: 72 items — 63 D, 8 D+M, 1 M-part (Japanese kanji), 0 X (the two
scan-only BSI documents were unblocked by the owner directive to read the scans
visually). Every remaining table is encodable as a deterministic map.**

## B. Track B matrix — model lines vs SOTA targets

| Line | Model / metric | Current | Target | WO | Levers (graph edges) |
|------|----------------|---------|--------|----|----------------------|
| Hebrew vocalization | heb-vocalized-plane-2.0 DER | 8.18 | **DER < 5** (int8 < 6) | 63 | data: additional labeled corpora (Dicta machine labels precedent); plane-conditioning scale; decode policy |
| Arabic vocalization | ara-vocalized-2.0 DER / WikiNews | 2.2864 / 11.83 | **DER < 2.0 / WER < 8** | 64 | r6-morph knowledge injection on v2 lineage; news-domain data; oracle-target quality (r8 noisy-student NEGATIVE — do not retry) |
| Persian registers | fas-phonemic-1.0 CER | 1.6 | **CER < 0.7 all registers** | 66 | register-balanced corpora; G2P lineage (tha-g2p template) |
| Thai phonemic | tha v6a PER (clean split) | 19.29 | **PER < 2** | 67 | lexicon (v6a shipped-candidate line; curriculum NEGATIVE), schedule-matched controls, tha-small 2.85 lineage as student template |
| Khmer Latin | khm-latn-1.0 CER / dict arm | 27.42 / 15.02 (dict-internal) | **CER < 15 + hit-rate** | 68 | lineage recovery (secryst corpus re-derivation, now concrete); owner-gated SEAlang / full Chuon Nath IPA |
| Urdu diac | urd-diac CER | 3.57 | **CER < 3** | 65 | decorrelated teachers only; VRR |
| Urdu G2P | urd-ipa-1.0 CER | 14.77 | license gate green | 65 | BLOCKED-OWNER: commercial-license data selection (option drafts in WO50) |

Standing negatives — never retry as written: RL distillation, LLM-teacher labels,
noisy-student relabel on correlated teachers, curriculum on full mixes, beam decode
on flat byte students, direct-division Sinkhorn.

## C. Graph path — reference to SOTA

```mermaid
flowchart TD
subgraph TrackA[Track A: conformance — 72 items]
REF[reference document<br/>PDF/scan/doc] -->|full transcription| ISC[.isc map<br/>ISO 24229 name]
ISC --> T1[embedded tests<br/>every table row + examples]
ISC --> P1[strict parse<br/>Ruby + TypeScript]
T1 --> CI1[CI: tests + parity green]
P1 --> CI1
CI1 --> PR[PR to interscript/maps]
PR --> REL[release artifact<br/>maps gem + index]
end

subgraph TrackB[Track B: SOTA — 6 active lines]
MAPD[conformant map] -->|apply to raw corpus| SILVER[silver pairs<br/>unvocalized → vocalized]
LEX[lexicon / gazetteer / Wikidata] --> GOLD[gold pairs + holdout]
SILVER --> TRAIN[ByT5 train<br/>HF Jobs --detach ≥4h, step ckpts]
GOLD --> EVAL[external protocol eval<br/>DER/CER/PER + VRR]
TRAIN --> EVAL
EVAL -->|meets target| QUANT[int8/fp16 export<br/>delta gate]
QUANT --> PUB[publish_model → GH Releases<br/>+ HF + models.yaml]
PUB --> RUN[runtime: npm gem worker<br/>parity fixtures]
EVAL -->|misses| FB[feedback levers:<br/>data · supervision · features · decode]
FB --> SILVER
FB --> TRAIN
end

REL -.->|map silver| TrackB
RUN -.->|error mining| FB
```

Dependency edges that matter:

1. Track A maps are the silver-data generators for every Track B line with a
deterministic surface (Hebrew map ↔ vocalization pairs; Urdu ALA-LC ↔ diac pairs;
Khmer BGN/PCGN ↔ latn pairs). Encoding a map feeds its model line.
2. Lexicons gate three targets: Thai (PER < 2 runs through lexicon + schedule controls),
Khmer (open phoneme supply exhausted at ~3.3k gold pairs), Persian (registers).
3. Owner decisions gate two: Urdu G2P license; Khmer SEAlang/Chuon-Nath licensing.
4. The Thai PER < 2 target additionally needs the length-prior mechanism solved
(v6b curriculum NEGATIVE) — filtered fine-tune or compound downweighting.

## D. Execution order

- **P0 — deterministic batches, unblocked now (Track A):**
1. WO90: 8 BGN/PCGN Cyrillic systems + Kurdish 2012 revision (9 maps).
2. WO91: DIN 5 + BSI 2 + ICS Annex D (8 maps).
3. WO92: ALA-LC batches — alphabets/syllabaries first (Cherokee, Deseret, Vai,
N'ko, Ol, Meitei, Lepcha, Limbu, …), Arabic-script tables next, Chinese/Japanese last.
4. WO93: ISO scope gaps + ICAO/UNGEGN acquisition diffs.
5. Housekeeping: merge PRs #211–#214 (owner authorization); migrate #166/#167
`.imp` → `.isc`; TS/Python strict-parse port; PyPI trusted publisher.
- **P1 — D+M companions:** after each map ships, open the model-companion work
order only for lines with an existing Track B lineage (heb, aii, khm, urd, ottoman/jawi
as future lines).
- **P2 — Track B pushes (parallel with P0):**
- B1 Hebrew → data lever first (63).
- B3 Arabic → r6-morph injection on v2 + news data (64).
- B4 Urdu diac → decorrelated teacher arms (65).
- B5 Persian → register-balanced corpora (66).
- B6 Thai → schedule-matched controls + lexicon filtering (67).
- B7 Khmer → lineage recovery then owner licensing decision (68).
- **P3 — owner-gated:** Urdu G2P license; Khmer licensing; any new minor version.

## Acceptance

- Every row of section A has a merged PR or a recorded blocker with owner decision.
- Every row of section B is at or past its target with external-protocol numbers,
quantization delta green, and runtime parity fixtures shipped.
- No negative lever retried as written; every miss recorded in its WO.
74 changes: 74 additions & 0 deletions TODO.sota/90-wo90-bgnpcgn-cyrillic-batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# WO90 — BGN/PCGN Cyrillic batch: 8 systems + Kurdish 2012 verdict

Status: SHIPPED (2026-10-28) — PR #215
Goal: encode the eight remaining BGN/PCGN Cyrillic systems from the reference
collection; resolve the "Kurdish 2012 revision" item.
Method: transcribe each table in full (every row and note), embed per-row tests plus
the source examples, verify under the Ruby interpreter, strict-parse the corpus,
ship as one PR.
Reference collection: `interscript-references/bgn-pcgn/` (text extracted to
/tmp/wo90-{0..8}.txt; glyph anomalies verified against the PDF text layer).

## Systems

| File | Source doc | System date | Checked | Entries | Context rules |
|------|-----------|-------------|---------|---------|---------------|
| bgnpcgn-ady-Cyrl-Latn-2012 | ROMANIZATION OF ADYGHE.pdf | 2012 | 2017-10 | 68 | У word-initial w |
| bgnpcgn-ava-Cyrl-Latn-2011 | ROMANIZATION OF AVAR.pdf | 2011 | 2017-10 | 47 | Е word-initial + post-vowel ye |
| bgnpcgn-chv-Cyrl-Latn-2011 | ROMANIZATION OF CHUVASH.pdf | 2011 | 2017-10 | 37 | Е/Ё initial + post-vowel/post-й-ъ-ь ye/yë |
| bgnpcgn-kbd-Cyrl-Latn-2011 | ROMANIZATION OF KABARDIAN.pdf | 2011 | 2017-10 | 59 | У word-initial w |
| bgnpcgn-krc-Cyrl-Latn-2008 | TABLE OF CORRESPONDENCES FOR KARACHAY-BALKAR.pdf | 2008 | 2017-10 | 38 | У post-vowel w; ъ/ь omitted outside digraphs |
| bgnpcgn-oss-Cyrl-Latn-2009 | ROMANIZATION OF OSSETIAN.pdf | 2009 | 2017-11 | 43 | У three-way (adjacent vowel / post-labializing / u) |
| bgnpcgn-udm-Cyrl-Latn-2011 | ROMANIZATION OF UDMURT.pdf | 2011 | 2017-11 | 38 | Е ye initial, post-vowel, post-йъь, post-дзлнст |
| bgnpcgn-sah-Cyrl-Latn-2012 | ROMANIZATION OF YAKUT.pdf | 2012 | 2017-11 | 40 | Е/Ё initial + post-vowel ye/yë |

## Source anomalies (documented in the maps, not silently corrected)

- ADYGHE + KABARDIAN note 4 prints the H̱ inventory codepoint as "0048+0031";
the glyph shown is h + combining macron below (U+0331) and the tables render the
romanization as the precomposed ẖ (U+1E96). The maps output ẖ (U+1E96) and quote
the note as printed.
- CHUVASH presents Ҫ/ҫ with the Latin Ç codepoints U+00C7/U+00E7 in the document's
text layer. The map accepts both those codepoints and the standard Chuvash
Cyrillic U+04AA/U+04AB, all to ś.
- YAKUT pairs capital Ѳ (text-layer U+0472, FITA) with lowercase ө (U+04E9). The
standard Yakut pair is Ө/ө (U+04E8/U+04E9). The map accepts both capitals to Ö
and records the source pairing.
- OSSETIAN note 2's example output prints "Respublikӕ" with the CYRILLIC ӕ where
the Roman æ (U+00E6) belongs. The embedded test uses Respublikæ and the map
records the source typo.
- YAKUT prints the lowercase form of НЬ as the ligature њ (U+045A). The map maps
нь (U+043D+044C), НЬ, and U+045A to ny and records the presentation.

## Kurdish "2012 revision" — VERDICT: reissue, no new map

`Kurdish_system_30April2012.pdf` is headed "ROMANIZATION SYSTEM FOR KURDISH
BGN/PCGN 2007" — the 30 April 2012 file restates the 2007 system already encoded as
`bgnpcgn-kur-Arab-Latn-2007.isc` (all 32 consonant rows, 10 vowel rows, notes 1-7,
special rules 1-7, inventory). No separate system exists; the A1 inventory item
closes as "reissue of 2007".

Observation recorded for a separate owner decision: in the existing 2007 map the
vowel row 9 (و with macron above, romanized ö, "rare") appears as
`sub "و" "ö"` after `sub "و" "u"` in the same parallel block — dead code
under first-match; the rare row is not reachable. Fix candidate for a dedicated PR;
not bundled here.

## Acceptance

- 8 new maps, each with the full table + all notes transcribed and embedded tests
covering every row (upper and lower) plus the source examples.
- All tests pass under the Ruby interpreter; corpus strict-parses (293 files).
- One PR to interscript/maps; coverage docs updated after merge.

## Result (2026-10-28)

- 8 maps encoded, 777 embedded tests, all passing under the Ruby interpreter:
ady 137/137, ava 97/97, chv 84/84, kbd 118/118, krc 80/80, oss 90/90,
udm 82/82, sah 89/89.
- Corpus strict parse: 300/300 .isc files.
- Branch `feat/bgnpcgn-cyrillic-batch`, commit 9ef9448, PR
https://github.com/interscript/maps/pull/215
- Kurdish verdict recorded above (reissue of 2007; no new map).
- Latent-gap observation on bgnpcgn-kur-Arab-Latn-2007 (dead و→ö row)
recorded for a separate owner decision; not bundled here.
Loading
Loading