|
| 1 | +# Learning the Underdetermined: A Dispatch Doctrine for Neural Completion of Rule-Definable Script Mappings |
| 2 | + |
| 3 | +**Draft 1 — WO77/E2 (resources & evaluation paper). Owner review before any submission.** |
| 4 | + |
| 5 | +## Abstract |
| 6 | + |
| 7 | +Authority romanization systems are rules; the languages they romanize |
| 8 | +are not always rule-complete. Abjads omit vowels, abugidas imply them, |
| 9 | +and the residual ambiguity must be resolved by a model — a component |
| 10 | +that authorities do not govern and that the field has no shared |
| 11 | +methodology for building. This paper contributes a dispatch doctrine: |
| 12 | +an empirically derived mapping from (script class, ambiguity type, |
| 13 | +data regime) to training technique, grounded in a negative-results |
| 14 | +registry from a systematic campaign, two structural rules that govern |
| 15 | +the composition of learned components with rule systems, and an |
| 16 | +ambiguity-aware evaluation metric (valid-reading rate) that corrects |
| 17 | +the systematic undercounting of single-reference diacritization |
| 18 | +error rates. We report per-language results under external-protocol |
| 19 | +evaluation for Arabic, Urdu, Persian, Thai, Khmer, and Hebrew, and we |
| 20 | +release every verdict — positive and negative — in a machine-checkable |
| 21 | +registry. |
| 22 | + |
| 23 | +## 1. The underdetermination spectrum |
| 24 | + |
| 25 | +Script conversion mappings form a spectrum. At one pole, the mapping |
| 26 | +is a function the authority fully decrees: Cyrillic → Latin with a |
| 27 | +published correspondence table. At the other, the source orthography |
| 28 | +genuinely underdetermines the target: an Arabic skeleton (كتب) is a |
| 29 | +*relation* over readings (kataba, kutiba, kutub, ...), a Thai |
| 30 | +orthographic syllable underdetermines its tone, a Khmer cluster |
| 31 | +underdetermines its vowel. The intermediate zone — where the authority |
| 32 | +decrees conventions *conditioned on an inferred layer* — is where |
| 33 | +learned components belong. |
| 34 | + |
| 35 | +The design error this paper argues against is treating the whole |
| 36 | +spectrum as one problem class: applying one neural recipe to every |
| 37 | +language, or conversely, hand-rolling a new pipeline per language. The |
| 38 | +dispatch doctrine makes the technique choice a function of measurable |
| 39 | +properties of the language pair. |
| 40 | + |
| 41 | +## 2. The doctrine |
| 42 | + |
| 43 | +**Rule 1 — Vocalization is composition, not a sibling task.** An |
| 44 | +unvocalized skeleton is a relation over readings; vocalization upgrades |
| 45 | +it to a function, which the rule system then applies. The two must not |
| 46 | +be conflated: romanizing a bare skeleton by convention (what |
| 47 | +bibliographic systems do) is a different product from predicting the |
| 48 | +hidden reading. Composition, not pipeline ad-hocery: the rule language |
| 49 | +declares the call, the model fills the relation, the decode policy |
| 50 | +selects among licensed readings. |
| 51 | + |
| 52 | +**Rule 2 — Abugidas presuppose a reading.** Thai and Khmer |
| 53 | +orthographies are positional syllabaries: vowel signs interleave with |
| 54 | +consonant clusters, inherent vowels are implied, tone is computed from |
| 55 | +orthographic class — and the major romanization convention (RTGS) |
| 56 | +discards the tone that the orthography computes. The rule system is |
| 57 | +deterministic *over an inferred layer*. The doctrine: lexicon-first |
| 58 | +(dictionary hit-rate is the first-class metric), trained fallback for |
| 59 | +novel tokens, and a rule composite for the convention; the three |
| 60 | +target layers (toneless romanization, phonemic, phonemic + tone) are |
| 61 | +tracked separately. |
| 62 | + |
| 63 | +**The dispatch table** (script class × ambiguity × data regime → |
| 64 | +technique), with the registry-defaulted form: lexical ambiguity |
| 65 | +dominant → lexicon + morphology-conditioned SFT; phonotactic |
| 66 | +reordering → byte-level seq2seq (ByT5-class); register variance → |
| 67 | +register-stratified data; teacher availability → oracle relabel only |
| 68 | +when teacher arms disagree; distillation only across decorrelated |
| 69 | +teachers. |
| 70 | + |
| 71 | +## 3. The negative-results registry |
| 72 | + |
| 73 | +Four refuted techniques are recorded with the same rigor as positives, |
| 74 | +each with its motivating failure: |
| 75 | + |
| 76 | +- **Reinforcement learning for diacritization** is knowledge-limited at |
| 77 | + SFT convergence: the reward signal cannot exceed what the supervised |
| 78 | + model already knows. |
| 79 | +- **CTC** cannot model Thai phonotactics: the conditional independence |
| 80 | + of CTC paths escapes the mode collapse of seq2seq training but |
| 81 | + plateaus far above the byte-level seq2seq baseline. |
| 82 | +- **LLMs as label teachers** hallucinate diacritics at rates |
| 83 | + incompatible with training; banned by registry law, enforced by the |
| 84 | + pipeline validators. |
| 85 | +- **Distillation on correlated teacher arms** transfers nothing: arms |
| 86 | + trained on the same data family agree on the same errors, leaving no |
| 87 | + oracle signal. Complementarity, not accuracy, predicts transfer. |
| 88 | + |
| 89 | +## 4. Per-language programs |
| 90 | + |
| 91 | +| language | task | technique (dispatch) | result | external protocol | |
| 92 | +|---|---|---|---|---| |
| 93 | +| Arabic | diacritization | ByT5 SFT + oracle relabel + noisy-student chain | DER 2.29 (canonical), 4.57 (fast tier) | SadeedDiac-25 | |
| 94 | +| Arabic | news register | register-closure specialist | WER 8.85 | WikiNews-2024 multiref | |
| 95 | +| Urdu | diacritization | base-model SFT | CER 3.57 | held-out docs split | |
| 96 | +| Persian | diacritization | SFT + register work | CER 0.603 row-mean | 9,288-row docs split | |
| 97 | +| Thai | G2P | lexicon-first + ByT5 fallback (CTC refuted) | PER 2.85 | published G2P suite | |
| 98 | +| Khmer | transliteration | lexicon-conditioned ByT5 (in progress) | CER 27.4 → target < 15 | held-out corpus | |
| 99 | +| Hebrew | nikud | plane architecture (data-gated) | DER 8.18 int8 | ReNikud protocol | |
| 100 | + |
| 101 | +## 5. Evaluation: valid-reading rate |
| 102 | + |
| 103 | +Single-reference DER counts every divergence from one gold reading as |
| 104 | +an error, systematically undercounting models that produce *other |
| 105 | +licensed readings*. We propose reporting valid-reading rate (VRR) — |
| 106 | +the proportion of predictions that are dictionary-licensed readings — |
| 107 | +alongside DER/WER, and evaluating multireference sets with |
| 108 | +oracle-validity rather than best-match. The multireference |
| 109 | +reconciliation failures that motivated this are documented with |
| 110 | +examples. |
| 111 | + |
| 112 | +## 6. Governance |
| 113 | + |
| 114 | +Every claim in Section 4 links to a frozen artifact: a pinned corpus |
| 115 | +(SHA-256, license-reviewed), a conformance-vector set reproduced |
| 116 | +byte-exactly in three independent runtimes, a prediction file, and an |
| 117 | +external protocol. Verdicts live in a machine-checkable registry that |
| 118 | +the training pipeline consults; refuted techniques require a new |
| 119 | +register entry to retry. |
| 120 | + |
| 121 | +## 7. Conclusion |
| 122 | + |
| 123 | +The dispatch doctrine converts per-language modeling from craft to |
| 124 | +policy: measurable properties select the technique, the registry |
| 125 | +prevents re-treading refuted ground, and the evaluation admits |
| 126 | +legitimate ambiguity. The doctrine is released as registry defaults so |
| 127 | +that the next language is a configuration, not a research project. |
| 128 | + |
| 129 | +## References (to complete) |
| 130 | + |
| 131 | +- ByT5; sequitur G2P; SadeedDiac-25; ReNikud; PyThaiNLP. |
| 132 | +- The negative-results registry (interscript-models TODO.sota). |
| 133 | +- RTGS; UNGEGN romanization; the ISC formalism (companion paper). |
0 commit comments