diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..692c07bb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,54 @@ +# CLAUDE.md — interscript-models + +The contract, zoo, and training spine for Interscript's neural layer. +Project purpose and direction live in README.md. + +## Layout + +- `models.yaml` + published `index-vN` tags — the resolution point for + every runtime (sha256-pinned, split-part support). +- `SPEC.md` — IMF v1 artifact format (normative). `runtime/` — the + conformance-proven reference loader; source of the frozen fixtures in + `conformance/`. +- `recipes/-.yaml` + `corpora/.yaml` — a language is + two YAML files plus a held-out eval set. Training config is data. +- `src/pipeline/` — the spine: contracts, gates (registry pattern), + planner, runner (HF Jobs), publish guard, housekeeper. +- `TODO.sota/` — the campaign ledger: work orders, verdicts, negatives. + Write the work order before doing the work; record negative results + with the same care as positives. + +## Ship gates (enforced by `src/pipeline/publish.py`, not by memory) + +- Explicit owner-stated semver; green gates (quality / parity / license + / conformance / regression); conformance fixtures present. +- Parity >= 500 samples in the SHIPPED decode mode (single-row greedy by + default; batched mode only for artifacts exported with a masked + encoder, `EXPORT_ATTENTION_MASK=1`). +- Metric conventions declared per metric (row-mean vs corpus-level CER). +- `UNVERIFIED` / `LicenseRef-*` licenses block publishing. Owner waivers + are `license_review:` records — the license string itself never + loosens. + +## Hard rules + +- PRs only, never commit to main. Versions are owner decisions — never + pick a number. +- NEVER use an LLM as a label teacher for diacritization — they + hallucinate haraqat. Enforced as a registry law, not a convention. +- Training on HF Jobs, never Modal: `--detach`, `--timeout >= 4h`, step + checkpoints, bucket I/O via HfFileSystem mount; <= 2 concurrent + big-GPU apps. +- Data channel: GitHub Releases + sha256 sidecar only — never + raw.githubusercontent.com. +- License: BSD-3-Clause for all ML code and weights. +- The package is immutable at runtime: artifacts ship to Releases/HF; + nothing writes inside the installed package. + +## Runtime expectations + +- ByT5 byte table: ids are byte + 3 with trailing EOS (id 1). Raw + `text.bytes` is silent garbage; parity alone cannot catch this bug. +- Exports take only `input_ids` unless built with a masked encoder; + batched padded decode on unmasked encoders corrupts encoder states + (the WO51 finding). diff --git a/README.md b/README.md index 428db64a..8fa5a3bd 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,17 @@ The **contract** for Interscript's phonological layer — the normative definition of what a "hidden reading" model is, and the zoo that publishes models conforming to it. +Interscript turns the world's transliteration standards from paper +tables into executable, verifiable software. Authority schemes +(BGN/PCGN, ISO, UN/UNGEGN, ALA-LC, ICAO, ODNI, ...) are encoded once in +the ISC language; every runtime produces provably identical output; and +languages that rules cannot fully carry — Arabic, Urdu, Persian, Hebrew +diacritization; grapheme-to-phoneme — are served by neural models +published under the same proof regime: pinned artifacts, sha256 +verification, frozen conformance vectors, byte-exact runtimes. Users +are libraries (MARC), media (subtitles), gazetteers (GeoNames), and +name pipelines that must romanize reproducibly under a named authority. + This repo owns three things and nothing else: 1. **The `models.yaml` index** — the stable URL every runtime resolves @@ -163,6 +174,27 @@ versions (never moving refs), held-out test splits mandatory, requires an explicit owner version, green gates, and frozen conformance fixtures (`conformance/CONFORMANCE.md`). + +## Direction (audit 2026-10) + +1. **SOTA everywhere, per-language technique dispatch.** The isxml + spine makes a language two YAML files and a technique a registered + stage; per-language rows (Hebrew plane/IPA, Arabic noisy-student + chains, Thai hybrid tiers, Urdu/Persian base models) land as config + PRs gated by machine-checkable thresholds. Refuted techniques (RL + for diacritization, LLM teachers, CTC for Thai phonotactics, + stage-2 distillation on correlated arms) stay in the registry as + negatives so nobody re-treads them. +2. **Contract-first distribution.** Every model ships only with + provenance-encoded corpora, resolved licenses, >=500-sample runtime + parity, and frozen conformance fixtures in every client runtime. +3. **The spec as the standard.** IMF v1 + the index + conformance + vectors are the implementable contract; third-party runtimes adopt + the contract, not our code. +4. **Ecosystem surfaces.** api.interscript.org (edge), + interscript.org (front door), MARC / subtitles / GeoNames as demand + channels. + ## Environment (as implemented by every crystal) `SECRYST_INDEX` (index URL or path; default: `models.yaml` on this