WO52 M0: corpus/recipe contracts + isxml validate

TODO.sota/51-artifact-parity-gates.md

@@ -0,0 +1,72 @@
+# WO51 — Artifact parity gates: the bf16/fp32 "export defect" case
+
+Status: **CLOSED 2026-10-17** — artifact exonerated; the harness was the defect
+Priority: unblocks urd-diac-base / urd-g2p-base / ara publishes (ship flavor = fp16)
+
+## The anomaly
+
+urd-diac-base: torch bf16 gate CER **0.0357** vs ONNX-fp32 artifact
+**0.1391–0.2783** (~11% of rows cascade to garbage in ANY
+ONNX/fp16/quantized decode). Same class seen on urd-g2p-base.
+
+## Probes run (all in bucket `run-066-urd-diac-base/export/`)
+
+| probe | verdict |
+|---|---|
+| probe3way (torch-bf16 vs torch-fp32 vs ORT-fp32, batched) | B≈A, C≠B → "export guilty" (misleading — see below) |
+| probe_logit_diff v4 (single-row, manual byte ids) | enc gap **0.0**, per-step argmax **identical** all rows — torch≡ORT when fed identical single-row ids |
+| fp16 re-export | 0.1938 — worse; precision is not the variable |
+| dynamic int8 | 0.44 — dynamic quantization separately broken for byt5 decode (static-int8 fine) |
+
+## Working hypothesis (probe_single_vs_batch.py, running)
+
+The exported encoder takes **only `input_ids` — no attention_mask**.
+Any batched decode that right-pads with id 0 corrupts encoder states;
+single-row decode cannot. Explains every observation: probe3way
+diverged only in batch-of-8; logit-diff agreed only single-row;
+zipcheck-v8 (runtime = single-window) verified GOOD; "single-window
+parity untouched".
+
+**CONFIRMED + CLOSED (run-066-single-parity, 2026-10-17):**
+`PAD_GAPS [0.048, 0.0769, 0.063, 0.1078, 0.0, 0.0541, 0.1338, 0.0607]`
+— 7/8 rows' encoder states differ single-vs-padded-batch (0.0 rows =
+longest-in-batch, nothing padding them); `ort_single_cer = 0.0352` ==
+torch-batched-mask == gate; `single_vs_batched_agree = 1.0` on 300/300
+rows. **The artifact was always correct; the maskless batched parity
+harness was the entire defect.** No re-export required.
+
+**fp16 recheck (run-066-fp16-single): `ort_single_cer = 0.0352`,
+`single_vs_batched_agree = 1.0` — fp16 ≡ fp32 ≡ gate.** The 0.1938
+"refutation" was entirely the poisoned harness. **Ship flavor: fp16**
+(1.35GB) over fp32 (2.16GB), identical verified quality.
+
+**Remedy adopted: (a) + (b) as capability.** Current artifacts ship
+under the single-row decode contract (runtime already conforms;
+contract now schema-enforced in recipes via `parity.mode`); the
+framework exporter (WO52 M5) grows an attention-mask option for future
+batched serving, gated by a batched-parity conformance fixture.
+
+## Permanent gate redesign (feeds WO52)
+
+1. **Parity must evaluate in the runtime's actual decode mode.** A
+   gate that batches when the runtime decodes single-row tests
+   nothing.
+2. Three-way probe is a release gate, not a debugging tool: record
+   bf16/fp32/ORT CERs + row agreement in the artifact metadata.
+3. Dynamic int8 is banned for byt5 seq2seq artifacts (recorded
+   negative); static int8 (plane family) unaffected.
+4. probe_logit_diff must use `tok()` encoding (manual `b+3` without
+   trailing EOS is the known byt5 byte-table trap — v4 ran on garbage
+   inputs; its engine-agreement result still valid, its CERs void).
+5. parity-v7 verdict INVALID (torch half defaulted to the g2p
+   checkpoint while zip was diac) — cross-model comparison; harness
+   envs must pin both halves to the same run id.
+
+## Release posture
+
+urd-diac-base: fp16 zip CLEARED (gate 0.0357 torch / artifact 0.0352
+single-row). urd-g2p-base and ara int8 cleared by the same contract
+(their zipchecks ran the real single-window runtime path). Publishing
+waits for the complete ship set (WO52 M2 proof lane + fas run-077
+verdict + relabel stage-2) per the owner's only-ship-when-complete
+order — nothing ships half-fixed.

TODO.sota/52-unified-framework.md

@@ -0,0 +1,145 @@
+# WO52 — The unified Interscript ML framework (recipes + spine + registry)
+
+Status: PROPOSAL (owner review before implementation)
+Priority: strategic — makes every future language/technique a config PR
+
+## Diagnosis (honest)
+
+We have three bodies of work that do not connect:
+
+1. **`interscript-models/src/framework/`** — a real abstraction skeleton
+   (registry pattern: register_data_module/evaluator/model_module,
+   TrainingPipeline, trainers, exporters; "new task = one dir under
+   src/tasks/"). Built for the custom-arch era (sinkhorn, muon, ride).
+2. **`interscript-train/`** — where actual winning practice lives:
+   standalone per-language scripts (train_urd_diac_base_hf.py,
+   distill_oracle_hf.py, export_imf_hf.py, parity_*, probe3way.py,
+   publish_model.py) hand-launched on HF Jobs for 70+ runs.
+3. **The release chain** — models.yaml, repack_imf_metadata.py,
+   publish_model.py, IMF spec, npm/gem/py/worker runtimes.
+
+Every SOTA result of the last quarter came from (2)+(3) while (1) sat
+unused — because the techniques that won (byt5 SFT + oracle relabel +
+decode policy) are recipes over shared ops, not new model modules, and
+the framework had no stage for corpus→gate→publish.
+
+**Target:** one spine where the proven scripts become stages, a
+language is a pair of YAML files, a technique is a registered stage,
+and nothing merges without machine-checkable gates. Adopted from KLF
+(kotoshu): `route: legacy_*` honest routing — the framework declares
+the contract while proven scripts remain the engines until each is
+absorbed. No big-bang rewrite.
+
+## Architecture: five layers
+
+```
+L0 CONTRACTS      corpora/<src>.yaml          (sources, license, pins, splits)
+                  recipes/<lang>-<task>.yaml  (stage DAG, gates, decode policy)
+L1 SPINE          src/framework/ + pipeline/  (isxml CLI: corpus|train|gate|publish)
+                  runner: HF Jobs launcher w/ laws encoded
+L2 GATES          parity (runtime-mode), probe3way, zipcheck,
+                  quality thresholds (from recipe), license, claims-from-JSON
+L3 PUBLISH        GH Releases (parts + sha256) ← primary channel
+                  HF model repos (canonical) + models.yaml registry stamp
+L4 CLIENTS        IMF spec + conformance fixtures
+                  npm / ruby gem / python / CF worker all resolve
+                  model-id@major from models.yaml → download → verify → run
+```
+
+### L0 — Contracts (the contribution surface)
+
+`recipes/urd-diac.yaml` (illustrative):
+
+```yaml
+id: urd-diac
+task: diacritization        # diacritization | g2p | transliteration
+corpora:
+  train: urd-dia-data@2026-06   # corpus registry ref (pinned, sha256)
+  test:  urd-dia-data@2026-06#test
+model:
+  family: byt5-base           # registered model module
+stages:                       # ordered DAG; each stage = registered plugin
+  - stage: sft                # → legacy route: train_urd_diac_base_hf.py
+  - stage: decode_policy      # repetition penalty 1.2, no-repeat-3gram
+gates:
+  quality: {cer_max: 0.045}   # from recipe; registry keeps best-known
+  parity:  {mode: single-row, agree_min: 0.98, three_way: true}
+  license: [data, weights]
+publish:
+  channel: gh-releases        # law: never raw.githubusercontent
+negative_results:             # recorded so nobody retries these
+  - rl-rollouts (knowledge-limited at SFT convergence)
+  - llm-teacher labels (haraqat hallucination)
+  - dynamic-int8 (destroys byt5 decode)
+  - ctc for thai phonotactics (plateaus ~70% PER)
+```
+
+### L1 — Spine
+
+`isxml <stage> <recipe>` — four verbs. The runner encodes the
+operational laws that today live only in memory/agents:
+`--detach` always (attached runs die with the client), `--timeout 4h+`,
+step checkpoints, explicit `HfApi` bucket sync (`repo_type=None`) on
+every append, ≤2 concurrent big-GPU apps, literal job commands (no
+IFS-split assembly), smoke run before any batch.
+
+### L2 — Gates (hardened by WO51)
+
+| gate | check | blocking |
+|---|---|---|
+| parity | torch vs artifact **in the runtime's decode mode** (single-row today) | yes |
+| three-way | bf16/fp32/ORT CER + row agreement recorded in metadata | yes |
+| quality | recipe thresholds vs test split | yes |
+| license | corpus licenses resolved before training (micsell = blocked) | yes |
+| claims | model card derived from eval JSON only (never hand-written) | yes |
+| regression | new metrics vs models.yaml best-known | flags, owner waives |
+
+### L3/L4 — Publish & mount
+
+Unchanged from the shipped system (parts ≤2GiB, sha256 sidecars,
+models.yaml stamp); new: runtimes gain one shared resolver
+(`load("urd-diac@1")`) + the conformance fixture set becomes the
+client-side contract (KLF's frozen-fixture rule: byte-identical or
+doesn't merge).
+
+## Contribution flows
+
+| I want to… | I write | CI runs |
+|---|---|---|
+| add a language | corpora/*.yaml + recipes/*.yaml + eval gold | schema+license check, 20-min smoke train, gate dry-run |
+| add a technique | stages/<name>/ plugin + unit tests | tests; then a recipe opts in; A/B vs gold |
+| add data | corpus source entry | purity + license gate |
+| add a client | IMF runtime impl | shared conformance fixtures |
+| fix the spine | framework code | everything re-runs gate dry-runs |
+
+## Migration (route:legacy — each step a PR, spine never breaks)
+
+- **M0** recipes/ + corpora/ schemas + validators + `isxml` CLI skeleton (TDD)
+- **M1** runner.py: HF Jobs launcher w/ laws; wraps existing scripts verbatim
+- **M2** urd-diac recipe end-to-end (corpus→sft→gate→publish) — proves the spine on the best-understood lane
+- **M3** gates module absorbs parity_*/probe3way/zipcheck + WO51 contract
+- **M4** ara + fas recipes; corpus registry with licenses + pins; absorb oracle_relabel as a stage (WO39)
+- **M5** models.yaml resolver in npm/gem/py + conformance fixtures v1; deprecate src/gpu/modal_* (Modal retired)
+
+## SOTA matrix (2026-10-17)
+
+| lane | best verified | vs external SOTA | gate status |
+|---|---|---|---|
+| AR diac | r6 WER 2.58 / CER 1.53 (canonical teacher); int8 1.0 zipcheck GOOD | competitive; r7 conventional-density drift (negative) | release-ready, wording owner-gated |
+| UR diac | base CER 0.0357 (gate) | artifact number pending WO51 single-parity | HELD on WO51 |
+| UR g2p | 0.16 CER (torch) | held since int8/fp32 divergence | HELD on WO51 |
+| FA diac | retrain 0.65% vs docs 0.52% MISS | epoch arm undecided | owner decision |
+| HE g2p/nikud | CER poor; label-filter + license-walled data | blocked on data supply | WO42/43 parked (owner asks drafted) |
+| TH g2p | ByT5 direction; CTC refuted; small tier shipped | plane family latency-only niche | small-tier contract met |
+| Oracle relabel (WO39) | p040/p029 complete, p047 resuming (fixed durable sync) | stage-2 fires on completion | running |
+
+## Relations
+
+- Consumes verdicts of [[51-artifact-parity-gates]] (gate redesign).
+- Encodes laws from memory: no-LLM-teachers (recipe validator rejects),
+  no-local-large-data, data channel = GH Releases, versions
+  owner-decided, HF-not-Modal, library packages side-effect-free.
+- KLF cross-pollination: adopted route:legacy + laws-in-code +
+  fixture-frozen conformance; our advantages kept: pinned/sha256
+  corpora, numeric parity gates, models.yaml registry, negative-result
+  registry. Gaps found in KLF filed as kotoshu/kotoshu#251.

corpora/fas-dia.yaml

@@ -0,0 +1,10 @@
+id: fas-dia
+task: diacritization
+sources:
+  - kind: bucket
+    id: Interscript/isx-training/run-076-fas-diac-retrain
+    revision: run-076
+    license: UNVERIFIED  # recovered-corpus provenance review open — gate blocks publish until resolved
+splits:
+  train: {path: train.jsonl}  # pin rows+sha256 from bucket manifest (M2)
+  test: {path: test.jsonl}

corpora/urd-dia.yaml

@@ -0,0 +1,11 @@
+id: urd-dia
+task: diacritization
+sources:
+  - kind: bucket
+    id: rhtse/jobs-artifacts/rababa-urdu-cc174b96
+    revision: data-diacrit
+    license: UNVERIFIED  # provenance review open — gate blocks publish until resolved
+    mount: /data-dia
+splits:
+  train: {path: data-diacrit/train.jsonl}  # pin rows+sha256 from bucket manifest (M2)
+  test: {path: data-diacrit/test.jsonl}

pyproject.toml

@@ -54,10 +54,11 @@ dev = [
 [project.scripts]
 interscript-ml = "imf.cli:main"
 interscript-sadeed-eval = "sadeedbench.cli:main"
+isxml = "pipeline.cli:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
-include = ["framework*", "tasks*", "imf*", "sadeedbench*"]
+include = ["framework*", "tasks*", "imf*", "sadeedbench*", "pipeline*"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

recipes/fas-diac.yaml

@@ -0,0 +1,15 @@
+id: fas-diac
+task: diacritization
+corpora:
+  train: fas-dia@v1
+  test: fas-dia@v1#test
+model: byt5-small
+stages:
+  - stage: sft
+    route: legacy:interscript-train/train_fas_diac_hf.py
+    params: {epochs: 6, bs: 16, accum: 2, gate: 0.0052}
+gates:
+  quality: {cer_max: 0.0052}
+  parity: {mode: single-row, agree_min: 0.98, three_way: true}
+negative_results:
+  - 3-epoch arm reached 0.65% vs docs 0.52% — epoch arm running (run-077)

recipes/urd-diac.yaml

@@ -0,0 +1,19 @@
+id: urd-diac
+task: diacritization
+corpora:
+  train: urd-dia@v1
+  test: urd-dia@v1#test
+model: byt5-base
+stages:
+  - stage: sft
+    route: legacy:interscript-train/train_urd_diac_base_hf.py
+    params: {epochs: 2, bs: 16, accum: 2}
+  - stage: decode_policy
+    params: {rep_penalty: 1.2, no_repeat_ngram: 3}
+gates:
+  quality: {cer_max: 0.045}
+  parity: {mode: single-row, agree_min: 0.98, three_way: true}
+negative_results:
+  - dynamic-int8 quantization destroys byt5 seq2seq decode (0.44 vs 0.16 CER)
+  - RL rollouts are knowledge-limited at SFT convergence — never retry
+  - LLM teachers hallucinate haraqat — banned for diacritization labels

src/pipeline/__init__.py

@@ -0,0 +1,17 @@
+"""WO52 pipeline spine (TODO.sota/52) — contracts, runner, gates."""
+
+from pipeline.contracts import (
+    CorpusSpec,
+    RecipeSpec,
+    load_corpus,
+    load_recipe,
+    validate_recipe_dir,
+)
+
+__all__ = [
+    "CorpusSpec",
+    "RecipeSpec",
+    "load_corpus",
+    "load_recipe",
+    "validate_recipe_dir",
+]