docs: project purpose + direction; add CLAUDE.md

CLAUDE.md

@@ -0,0 +1,54 @@
+# CLAUDE.md — interscript-models
+
+The contract, zoo, and training spine for Interscript's neural layer.
+Project purpose and direction live in README.md.
+
+## Layout
+
+- `models.yaml` + published `index-vN` tags — the resolution point for
+  every runtime (sha256-pinned, split-part support).
+- `SPEC.md` — IMF v1 artifact format (normative). `runtime/` — the
+  conformance-proven reference loader; source of the frozen fixtures in
+  `conformance/`.
+- `recipes/<lang>-<task>.yaml` + `corpora/<source>.yaml` — a language is
+  two YAML files plus a held-out eval set. Training config is data.
+- `src/pipeline/` — the spine: contracts, gates (registry pattern),
+  planner, runner (HF Jobs), publish guard, housekeeper.
+- `TODO.sota/` — the campaign ledger: work orders, verdicts, negatives.
+  Write the work order before doing the work; record negative results
+  with the same care as positives.
+
+## Ship gates (enforced by `src/pipeline/publish.py`, not by memory)
+
+- Explicit owner-stated semver; green gates (quality / parity / license
+  / conformance / regression); conformance fixtures present.
+- Parity >= 500 samples in the SHIPPED decode mode (single-row greedy by
+  default; batched mode only for artifacts exported with a masked
+  encoder, `EXPORT_ATTENTION_MASK=1`).
+- Metric conventions declared per metric (row-mean vs corpus-level CER).
+- `UNVERIFIED` / `LicenseRef-*` licenses block publishing. Owner waivers
+  are `license_review:` records — the license string itself never
+  loosens.
+
+## Hard rules
+
+- PRs only, never commit to main. Versions are owner decisions — never
+  pick a number.
+- NEVER use an LLM as a label teacher for diacritization — they
+  hallucinate haraqat. Enforced as a registry law, not a convention.
+- Training on HF Jobs, never Modal: `--detach`, `--timeout >= 4h`, step
+  checkpoints, bucket I/O via HfFileSystem mount; <= 2 concurrent
+  big-GPU apps.
+- Data channel: GitHub Releases + sha256 sidecar only — never
+  raw.githubusercontent.com.
+- License: BSD-3-Clause for all ML code and weights.
+- The package is immutable at runtime: artifacts ship to Releases/HF;
+  nothing writes inside the installed package.
+
+## Runtime expectations
+
+- ByT5 byte table: ids are byte + 3 with trailing EOS (id 1). Raw
+  `text.bytes` is silent garbage; parity alone cannot catch this bug.
+- Exports take only `input_ids` unless built with a masked encoder;
+  batched padded decode on unmasked encoders corrupts encoder states
+  (the WO51 finding).

README.md

@@ -4,6 +4,17 @@ The **contract** for Interscript's phonological layer — the normative
 definition of what a "hidden reading" model is, and the zoo that
 publishes models conforming to it.
 
+Interscript turns the world's transliteration standards from paper
+tables into executable, verifiable software. Authority schemes
+(BGN/PCGN, ISO, UN/UNGEGN, ALA-LC, ICAO, ODNI, ...) are encoded once in
+the ISC language; every runtime produces provably identical output; and
+languages that rules cannot fully carry — Arabic, Urdu, Persian, Hebrew
+diacritization; grapheme-to-phoneme — are served by neural models
+published under the same proof regime: pinned artifacts, sha256
+verification, frozen conformance vectors, byte-exact runtimes. Users
+are libraries (MARC), media (subtitles), gazetteers (GeoNames), and
+name pipelines that must romanize reproducibly under a named authority.
+
 This repo owns three things and nothing else:
 
 1. **The `models.yaml` index** — the stable URL every runtime resolves
@@ -163,6 +174,27 @@ versions (never moving refs), held-out test splits mandatory,
 requires an explicit owner version, green gates, and frozen conformance
 fixtures (`conformance/CONFORMANCE.md`).
 
+
+## Direction (audit 2026-10)
+
+1. **SOTA everywhere, per-language technique dispatch.** The isxml
+   spine makes a language two YAML files and a technique a registered
+   stage; per-language rows (Hebrew plane/IPA, Arabic noisy-student
+   chains, Thai hybrid tiers, Urdu/Persian base models) land as config
+   PRs gated by machine-checkable thresholds. Refuted techniques (RL
+   for diacritization, LLM teachers, CTC for Thai phonotactics,
+   stage-2 distillation on correlated arms) stay in the registry as
+   negatives so nobody re-treads them.
+2. **Contract-first distribution.** Every model ships only with
+   provenance-encoded corpora, resolved licenses, >=500-sample runtime
+   parity, and frozen conformance fixtures in every client runtime.
+3. **The spec as the standard.** IMF v1 + the index + conformance
+   vectors are the implementable contract; third-party runtimes adopt
+   the contract, not our code.
+4. **Ecosystem surfaces.** api.interscript.org (edge),
+   interscript.org (front door), MARC / subtitles / GeoNames as demand
+   channels.
+
 ## Environment (as implemented by every crystal)
 
 `SECRYST_INDEX` (index URL or path; default: `models.yaml` on this