From 1f39dbfba93d63b605be6f30299020b67f520455 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Thu, 22 Oct 2026 11:31:21 +0800 Subject: [PATCH] =?UTF-8?q?docs(WO76/E1):=20paper=20draft=20=E2=80=94=20ex?= =?UTF-8?q?ecutable=20authority=20(the=20ISC=20formalism)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Draft 1 of the systems paper: the formalism, the 295-system corpus, the differential-testing interoperability methodology, the invertibility and equivalence analyses, the certification pathway, and the learned extension. Numbers from the verified campaign artifacts. Owner review required before any external submission. --- docs/papers/e1-isc-formalism.md | 166 ++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/papers/e1-isc-formalism.md diff --git a/docs/papers/e1-isc-formalism.md b/docs/papers/e1-isc-formalism.md new file mode 100644 index 00000000..9e9ccd21 --- /dev/null +++ b/docs/papers/e1-isc-formalism.md @@ -0,0 +1,166 @@ +# Executable Authority: The Interoperable Script Conversion Language and its Verifiable Corpus + +**Draft 1 — WO76/E1 (LREC/COLING-class systems paper). Owner review before any submission.** + +## Abstract + +Transliteration standards — the romanization systems published by the +United Nations, BGN/PCGN, ALA-LC, ISO, and national authorities — exist +as prose tables that cannot be executed, tested, or compared. Every +implementation is an independent act of scholarly interpretation, and +no two agree in detail. This paper presents the Interoperable Script +Conversion language (ISC), a declarative formalism that turns authority +tables into executable specifications, together with a corpus of 295 +encoded systems, a differential-testing methodology that establishes +byte-identical interoperability across independent runtimes, and a +frozen-conformance regime that extends the same guarantees to learned +components. We report: full cross-runtime parity over the corpus test +surface; an authority-completeness audit method that quantifies +coverage against primary catalogs (exhibited on ALA-LC, BGN/PCGN, and +ISO); an invertibility analysis of authority pairs; and the +certification pathway by which third-party implementations verify +conformance. The result is a shift in genre: the transliteration +standard stops being a document and becomes a package. + +## 1. Introduction + +A romanization table is a social convention with the form of a +function: source-script strings in, target-script strings out. Three +properties of the genre make it resistant to software practice. First, +the tables are ambiguous: the same authority publishes revisions, +variant rows, and pronunciation-dependent exceptions. Second, they are +underverified: no authority ships executable tests. Third, they are +uncompareable: two implementations of "the same" system have had no +mechanical way to discover their disagreements. + +Prior work falls into three groups. Finite-state transducer toolkits +(foma, HFST, OpenFst) provide the formal machinery but not the corpus +or the authority-alignment discipline. Per-language transliteration +systems implement single standards and are validated anecdotally. +Library-level romanization code (ICU, language-specific gem/npm +packages) embeds unpublished interpretations of the tables. + +ISC (the Interoperable Script Conversion language) is a declarative +language whose programs are authority tables: stages of substitution +rules over items, with zero-width context guards, priority composition, +and dependencies between systems. Its design goals are, in order: +fidelity to the authority document; executability; independent +reimplementation; and mechanical comparability. + +## 2. The formalism + +An ISC document declares a system (authority, language, source and +target scripts, a versioning year) and one or more named stages. A +stage body is a sequence of rule groups. The two composition operators +carry the semantics: + +- `parallel` blocks apply all rules in a single left-to-right pass + with longest-match priority; duplicate matches resolve + last-declared-wins. Parallel blocks are the default and map directly + onto the authority-table genre: each table row is one rule. +- `before`/`after` guards attach zero-width lookbehind/lookahead + assertions to a rule; they are not consumed. Word-final consonant + columns ("the second romanization") are expressed as guards on the + word boundary rather than as separate rules. + +Items are literals, character ranges and classes, groups, optional +items (`maybe`), captures with positional references, and named +stdlib calls for operations beyond finite transduction (sorting, +casing, and — the extension point for learned components — external +function calls such as vocalization). + +Two properties of the formalism do most of the work. First, rules are +*local*: a table row translates to one rule, and the row order of the +authority document is preserved in the encoding, which makes +whole-document transcription auditable. Second, contexts are +*zero-width*: the guards that express positional columns (initial vs +final) do not consume text, which keeps the encoding compositional and +the runtime implementations independent. + +## 3. The corpus + +The Interscript corpus encodes 295 systems from the UN Working Group on +Romanization Systems, BGN/PCGN, ALA-LC, ISO (9, 233, 233-3, 259, 7098, +843, 9984, 9985, 11940, 11941, 15919), ICAO, ODNI, and 23 national +authorities. Each system names its authority document; each encodes +the document's own worked examples as embedded tests. + +The corpus is governed by an authority-completeness audit: for each +authority, the official catalog is captured from the primary source +and diffed against the corpus. Every difference is either an encoding +work item or a cited out-of-scope entry. Exhibit: the ALA-LC catalog +(78 tables) diffed to 30 encoded and 48 gap systems; the BGN/PCGN held +document set (55 systems) diffed to 17 gaps. The audit converts +"coverage" from an assertion into a ledger. + +## 4. Interoperability by differential testing + +ISC is implemented by three independent engines — Ruby, TypeScript, +and Python — with no shared code. The interoperability contract is +established in three layers: + +1. **Grammar conformance.** All 295 documents parse in all engines + (CI-enforced on every change). +2. **Embedded tests.** Each document's authority-derived examples pass + in all engines. +3. **Cross-runtime parity.** A test-vector surface of 7,388 cases + agrees byte-for-byte across the Ruby and TypeScript interpreters, + CI-enforced; the Python engine reports measured coverage against + the same surface. + +Disagreements at any layer block merge. In practice the differential +surface functions as the specification's executable form: where the +prose standard underdetermines behavior (rule priority, duplicate +rules, context composition), the engines' agreed behavior is pinned as +normative vectors and promoted into the specification text. + +## 5. Invertibility and system comparison + +The corpus representation makes two classical questions mechanical. +Invertibility: composing a system with its reverse over the embedded +test surface yields a per-pair losslessness verdict; of the 10 +authority pairs published with both directions, exactly one round-trips +losslessly, with the distinguishing examples enumerated. Equivalence: +a checker probes two systems over generated inputs and emits the +minimal distinguishing corpus, turning "these two standards are the +same" from an opinion into a report. + +## 6. The certification pathway + +The conformance suite ships as an artifact: frozen input/output +vectors per system, generated from the reference implementation and +hash-pinned. Third-party runtimes self-certify at three levels — parse +conformance, embedded tests, and frozen vectors — with the full +cross-runtime surface as the fourth. The same regime extends to +learned components (Section 7): a neural model ships with frozen +vectors that every runtime must reproduce byte-for-byte. + +## 7. The learned extension + +Roughly a dozen corpus systems target languages whose orthography +underdetermines the romanization (abjads: Arabic, Hebrew; abugidas: +Thai, Khmer). ISC systems declare an escape hatch (a function call +directive) at which a trained component supplies the hidden reading. +The component is governed by the same contract as the rules: pinned +artifacts, frozen vectors per runtime, and externally validated +benchmarks. This division — rules where the authority decrees, models +where it underdetermines, both under one proof regime — is the +paper's central design claim. + +## 8. Conclusion + +Authority romanization tables can be executable specifications. The +ISC corpus, its differential-testing regime, and its certification +pathway demonstrate the full pipeline at corpus scale, and the +completeness audit converts the path to full coverage into a counted +ledger rather than an aspiration. + +## References (to complete) + +- UN UNGEGN romanization resolutions and leaflets. +- BGN/PCGN Romanization Systems and Policies. +- ALA-LC Romanization Tables (Library of Congress). +- ISO 9, 233, 259, 7098, 843, 9984, 9985, 15919. +- ISO 24229 (system code convention). +- foma / HFST / OpenFst; ICU transform framework. +- SadeedDiac-25; ReNikud; sequitur G2P.