WO52 M4: ara/urd-g2p recipes + corpus license provenance

TODO.sota/47-fas-diac-release.md

@@ -59,3 +59,28 @@ VERIFIED number in hand. Next-session options: (a) one more epoch
 arm (the original recipe likely trained longer — check configs/),
 (b) ship 0.65% as the verified artifact with the docs discrepancy
 noted. Both are cheap; (a) first.
+
+## 2026-10-18: the gap is the backbone, not epochs
+
+- run-076 (small, 3ep): 0.65% · run-077 (small, 6ep): **0.60%** — both
+  MISS vs docs 0.52%; epochs saturating.
+- Gate split verified byte-identical to the docs split (data/test.jsonl
+  9,288 rows; sha256 67513aa16ec142ab matches the staged bucket copy) —
+  the comparison was always apples-to-apples.
+- rababa-farsi/docs/RESULTS.md records the docs recipe explicitly:
+  **ByT5-base, 3 epochs**, 464K derived pairs. Our arms ran byt5-small
+  — wrong backbone. Corpus count (445,869 after in-training filters of
+  the 464K usable) is not the variable.
+- **run-078-fas-diac-base launched** (byt5-base, 3ep, same corpus and
+  gate). recipes/fas-diac.yaml corrected to byt5-base. Expect ~27h;
+  epoch-resume relaunch if the 23h timeout lands first.
+
+### run-078 (byt5-base, our trainer): 0.63% MISS — and the LR was found
+
+Base beat small on neither surface (0.63 vs 0.60). Root cause found in
+modal_app_diacrit.py: run-001 trained at **lr 5e-4, warmup 500, HF
+linear decay, wd 0**; our trainer used 3e-4/cosine/warmup-200/wd-0.01.
+Epochs/bs/backbone all matched. **run-079-fas-diac-docslr launched**
+with the full docs optimizer recipe (trainer now has U_LR/U_WARMUP/
+U_SCHED/U_WD knobs). Matrix so far, same split/corpus:
+small-3ep 0.65 · small-6ep 0.60 · base-3ep@3e-4 0.63 · docs@5e-4 0.52?

TODO.sota/52-unified-framework.md

@@ -1,6 +1,12 @@
 # WO52 — The unified Interscript ML framework (recipes + spine + registry)
 
-Status: PROPOSAL (owner review before implementation)
+Status: **IN IMPLEMENTATION** — owner-approved 2026-10-17
+Progress: M0 contracts + M1 runner + M2 gates/planner merged (#294);
+M4 recipes landed (ara-diac, urd-g2p + corpora registry + license
+provenance — `corpora/PROVENANCE.md`). Remaining: M3 (pins: rows +
+sha256 manifests; probe/zipcheck absorbed as gate inputs), M5
+(models.yaml resolver in clients, conformance fixtures, exporter
+attention-mask option, deprecate src/gpu/modal_*).
 Priority: strategic — makes every future language/technique a config PR
 
 ## Diagnosis (honest)

corpora/PROVENANCE.md

@@ -0,0 +1,29 @@
+# Corpus license provenance — the license gate's evidence file
+
+Verified 2026-10-17. The gate blocks on `UNVERIFIED` (terms not read)
+and on `LicenseRef-*` (read but restricted). Only verified-open SPDX
+licenses clear it.
+
+| corpus | upstream | license | verdict |
+|---|---|---|---|
+| fas-dia | [MahtaFetrat/HomoRich-G2P-Persian](https://huggingface.co/datasets/MahtaFetrat/HomoRich-G2P-Persian) | **CC0-1.0** (HF cardData, verified via API) | CLEAR |
+| urd-g2p-dict | [humair025/urdu-g2p-dictionary](https://huggingface.co/datasets/humair025/urdu-g2p-dictionary) | **non-commercial** (HF card `license_name`, terms at github.com/humair-m/urdu-g2p) | BLOCKED — owner: may weights trained on it ship under the BSD release lane? |
+| qcri-wiki | QCRI `Wikipedia_20240420.diac.jsonl`; system repo [kdarwish/Qatara](https://github.com/kdarwish/Qatara) carries **no license file** | default copyright | BLOCKED — owner: contact QCRI for terms |
+| urd-dia | mixture of the `mahwizzzz` + `humairmunirawn` corpora (rababa-urdu/docs/paper-urdu/main.pdf) | none found on either source (HF/GitHub search) | BLOCKED — owner: contact authors or replace the corpus |
+| tashkeela (classical, r8-mined) | tashkeela-processed via SourceForge/Kaggle (rababa-tashkeela/README.adoc) | **GPLv2** "as per the original datasets" | CLEAR for training; copyleft note for weight redistribution |
+
+## Owner decision queue (what the license gate is holding)
+
+1. **urd-g2p** — non-commercial data: ship weights research-only, or
+   retrain on an open replacement before release?
+2. **qcri-wiki (ara-diac)** — QCRI has no license on Qatara; their own
+   models are published. Ask QCRI for explicit terms, or retrain the
+   ara lane on Tashkeela-derived data only (GPLv2 implications then
+   apply to weights).
+3. **urd-dia** — no licenses found on the mixed sources. Contact the
+   paper authors, or rebuild the corpus from licensed sources.
+
+Notes:
+- Derived corpora (staged jsonl splits) inherit their upstream license.
+- micsell/hebrew_speech_kan_nikud: NO license (WO43) — excluded from
+  the registry entirely; owner ask drafted in TODO.sota/43.

corpora/fas-dia.yaml

@@ -1,10 +1,19 @@
 id: fas-dia
 task: diacritization
 sources:
+  # Upstream: verified CC0-1.0 via HF cardData (2026-10-17).
+  # haraqat targets derived deterministically from HomoRich's Mapped
+  # Phoneme field per rababa-farsi/docs/RESULTS.md — derived work of a
+  # CC0 dataset stays CC0.
+  - kind: hf-dataset
+    id: MahtaFetrat/HomoRich-G2P-Persian
+    revision: main
+    license: CC0-1.0
+  # Staged jsonl splits consumed by training (bucket copy of the above).
   - kind: bucket
     id: Interscript/isx-training/run-076-fas-diac-retrain
     revision: run-076
-    license: UNVERIFIED  # recovered-corpus provenance review open — gate blocks publish until resolved
+    license: CC0-1.0
 splits:
-  train: {path: train.jsonl}  # pin rows+sha256 from bucket manifest (M2)
+  train: {path: train.jsonl}  # pin rows+sha256 from bucket manifest (M3)
   test: {path: test.jsonl}

corpora/qcri-wiki.yaml

@@ -0,0 +1,16 @@
+id: qcri-wiki
+task: diacritization
+sources:
+  # QCRI's published Wikipedia_20240420.diac.jsonl (~5M words, BiLSTM
+  # silver ~3% WER), staged on our dataset repo for training mounts.
+  # The QCRI system repo (kdarwish/Qatara) carries NO license file —
+  # default copyright applies. Owner decision: contact QCRI for terms.
+  - kind: hf-dataset
+    id: Interscript/arabic-r8
+    revision: main
+    license: LicenseRef-no-license-found  # Qatara repo unlicensed — owner: contact QCRI
+    mount: /train_data
+splits:
+  train: {path: qcri-wiki.jsonl}  # pin rows+sha256 from dataset manifest (M3)
+  test: {path: wikinews/}  # WikiNews-2024 multiref (OOD register gate)
+  sadeed: {path: sadeeddiac-25.parquet}  # SadeedDiac-25 (identity gate)

corpora/urd-dia.yaml

@@ -1,11 +1,15 @@
 id: urd-dia
 task: diacritization
 sources:
+  # Upstream: mixture of the mahwizzzz and humairmunirawn Urdu corpora
+  # (per rababa-urdu/docs/paper-urdu/main.pdf; uploaded from
+  # /opt/urdu/data in rababa-urdu/modal_app_diacrit.py). No license
+  # found on either source (HF/GitHub search 2026-10-17).
   - kind: bucket
     id: rhtse/jobs-artifacts/rababa-urdu-cc174b96
     revision: data-diacrit
-    license: UNVERIFIED  # provenance review open — gate blocks publish until resolved
+    license: LicenseRef-unresolved  # no license found on sources — owner: contact authors or replace corpus
     mount: /data-dia
 splits:
-  train: {path: data-diacrit/train.jsonl}  # pin rows+sha256 from bucket manifest (M2)
+  train: {path: data-diacrit/train.jsonl}  # pin rows+sha256 from bucket manifest (M3)
   test: {path: data-diacrit/test.jsonl}

corpora/urd-g2p-dict.yaml

@@ -0,0 +1,20 @@
+id: urd-g2p-dict
+task: g2p
+sources:
+  # Verified via HF card (2026-10-17): license_name "non-commercial",
+  # terms at github.com/humair-m/urdu-g2p (LICENSE). Owner decision:
+  # whether weights trained on it may ship under our BSD release lane.
+  - kind: hf-dataset
+    id: humair025/urdu-g2p-dictionary
+    revision: main
+    license: LicenseRef-non-commercial  # verified restricted — owner: weights-distribution decision
+  # Staged copy consumed by training (610K pairs via rababa-urdu/data);
+  # inherits upstream non-commercial terms.
+  - kind: bucket
+    id: rhtse/jobs-artifacts/rababa-urdu-cc174b96
+    revision: data
+    license: LicenseRef-non-commercial
+    mount: /data
+splits:
+  train: {path: data/train.jsonl}  # pin rows+sha256 from bucket manifest (M3)
+  test: {path: data/test.jsonl}

recipes/ara-diac.yaml

@@ -0,0 +1,22 @@
+id: ara-diac
+task: diacritization
+corpora:
+  train: qcri-wiki@2024-04-20
+  test: qcri-wiki@2024-04-20#test
+model: byt5-base
+stages:
+  - stage: oracle_relabel
+    env_prefix: D
+    route: legacy:interscript-train/distill_oracle_hf.py
+    params: {subset: 300000}
+  - stage: sft
+    env_prefix: O
+    route: legacy:interscript-train/continue_oracle_train_hf.py
+    params: {epochs: 2}
+gates:
+  quality: {wer_max: 3.5607}  # run-047 first-alt line; stretch 2.70 (QCRI frontier parity)
+  parity: {mode: single-row, agree_min: 0.98, three_way: true}
+negative_results:
+  - r9-yallamorph convention-density drift — r7 stays canonical
+  - RIDE residual-displacement arm negative
+  - LLM teachers hallucinate haraqat — banned for diacritization labels

recipes/fas-diac.yaml

@@ -3,13 +3,16 @@ task: diacritization
 corpora:
   train: fas-dia@v1
   test: fas-dia@v1#test
-model: byt5-small
+model: byt5-base
 stages:
   - stage: sft
     route: legacy:interscript-train/train_fas_diac_hf.py
-    params: {epochs: 6, bs: 16, accum: 2, gate: 0.0052}
+    params: {epochs: 3, bs: 16, accum: 2, gate: 0.0052}
 gates:
   quality: {cer_max: 0.0052}
   parity: {mode: single-row, agree_min: 0.98, three_way: true}
 negative_results:
-  - 3-epoch arm reached 0.65% vs docs 0.52% — epoch arm running (run-077)
+  - byt5-small undershoots the docs recipe on the same split and data
+    (0.65% at 3ep, 0.60% at 6ep vs docs 0.52%) — the docs 0.52% line is
+    ByT5-base 3ep on the 464K derived pairs (rababa-farsi RESULTS.md);
+    backbone, not epochs, is the variable (run-077/run-078)

recipes/urd-g2p.yaml

@@ -0,0 +1,15 @@
+id: urd-g2p
+task: g2p
+corpora:
+  train: urd-g2p-dict@v1
+  test: urd-g2p-dict@v1#test
+model: byt5-base
+stages:
+  - stage: sft
+    route: legacy:interscript-train/train_urd_g2p_hf.py
+gates:
+  quality: {cer_max: 0.17}  # torch greedy 0.16 (artifact ≡ torch under the WO51 single-row contract)
+  parity: {mode: single-row, agree_min: 0.98, three_way: true}
+negative_results:
+  - dynamic-int8 quantization destroys byt5 decode (0.44 vs 0.16 CER)
+  - batched-without-attention-mask parity harness measures nothing (WO51)

src/pipeline/contracts.py

@@ -23,7 +23,8 @@
 KNOWN_STAGES = frozenset(
     {"sft", "oracle_relabel", "distill", "decode_policy", "plane", "lexicon_fusion"}
 )
-LANG_RE = re.compile(r"^[a-z]{2,3}-[a-z0-9-]+$")
+LANG_RE = re.compile(r"^[a-z]{2,3}-[a-z0-9-]+$")  # recipes: <lang>-<task>
+CORPUS_ID_RE = re.compile(r"^[a-z][a-z0-9-]*$")  # corpora: source names
 CORPUS_REF_RE = re.compile(
     r"^(?P<name>[a-z][a-z0-9-]*)@"
     r"(?P<pin>\d{4}-\d{2}-\d{2}|v\d+(?:\.\d+)*)"
@@ -52,7 +53,7 @@ class SplitSpec(_Strict):
 
 
 class CorpusSpec(_Strict):
-    id: str = Field(pattern=LANG_RE.pattern)
+    id: str = Field(pattern=CORPUS_ID_RE.pattern)
     task: TaskKind
     sources: list[CorpusSource] = Field(min_length=1)
     splits: dict[str, SplitSpec]
@@ -95,6 +96,7 @@ class StageSpec(_Strict):
     stage: str
     route: str | None = None
     params: dict[str, Any] = Field(default_factory=dict)
+    env_prefix: str = Field(default="U", pattern=r"^[A-Z]{1,3}$")
 
     @model_validator(mode="after")
     def _implementable(self) -> StageSpec:
@@ -153,6 +155,12 @@ def load_corpus(path: Path) -> CorpusSpec:
     return CorpusSpec.model_validate(body)
 
 
+def _license_blocked(lic: str) -> bool:
+    # UNVERIFIED = terms not read; LicenseRef-* = read but restricted
+    # (non-commercial, unlicensed, custom) — both need owner review.
+    return lic == "UNVERIFIED" or lic.startswith("LicenseRef-")
+
+
 def validate_recipe_dir(recipes_dir: Path, corpora_dir: Path) -> list[str]:
     problems: list[str] = []
     corpora: dict[str, CorpusSpec] = {}
@@ -186,10 +194,15 @@ def validate_recipe_dir(recipes_dir: Path, corpora_dir: Path) -> list[str]:
                     f"recipe task '{r.task}'"
                 )
             if r.gates.license and any(
-                s.license == "UNVERIFIED" for s in c.sources
+                _license_blocked(s.license) for s in c.sources
             ):
+                blocked = ", ".join(
+                    f"{s.id}={s.license}"
+                    for s in c.sources
+                    if _license_blocked(s.license)
+                )
                 problems.append(
-                    f"{rpath.name}: corpus '{c.id}' has UNVERIFIED license — "
-                    "resolve before publishing (gates.license)"
+                    f"{rpath.name}: corpus '{c.id}' restricted or unread "
+                    f"license ({blocked}) — owner review required before publishing"
                 )
     return problems

src/pipeline/gates.py

@@ -73,17 +73,19 @@ def check_parity(recipe: RecipeSpec, parity: dict | None) -> GateResult:
 
 
 def check_license(recipe: RecipeSpec, corpora_dir: Path) -> GateResult:
+    from pipeline.contracts import _license_blocked
+
     names = {ref.split("@", 1)[0] for ref in recipe.corpora.values()}
     for name in sorted(names):
         corpus = load_corpus(Path(corpora_dir) / f"{name}.yaml")
-        bad = [s for s in corpus.sources if s.license == "UNVERIFIED"]
+        bad = [s for s in corpus.sources if _license_blocked(s.license)]
         if bad:
-            ids = ", ".join(s.id for s in bad)
+            ids = ", ".join(f"{s.id}={s.license}" for s in bad)
             return GateResult(
                 "license",
                 False,
-                f"corpus '{name}': UNVERIFIED license ({ids}) — "
-                "resolve provenance before publishing",
+                f"corpus '{name}': restricted or unread license ({ids}) — "
+                "owner review required before publishing",
             )
     return GateResult("license", True, f"corpora {sorted(names)} licensed")
 

src/pipeline/planner.py

@@ -33,8 +33,9 @@ def plan_stage(
         return None
     corpora_dir = corpora_dir or DEFAULT_CORPORA_DIR
     script = stage.route.split(":", 1)[-1].rsplit("/", 1)[-1]
-    env = {f"U_{k.upper()}": str(v) for k, v in stage.params.items()}
-    env["U_RUN"] = run
+    prefix = stage.env_prefix
+    env = {f"{prefix}_{k.upper()}": str(v) for k, v in stage.params.items()}
+    env[f"{prefix}_RUN"] = run
     volumes = [Volume(CKPT_BUCKET, CKPT_MOUNT, "rw")]
     seen: set[str] = set()
     idx = 0
@@ -45,10 +46,15 @@ def plan_stage(
         seen.add(name)
         corpus = load_corpus(corpora_dir / f"{name}.yaml")
         for src in corpus.sources:
-            if src.kind != "bucket":
-                continue
+            if src.kind == "gh-release":
+                continue  # release assets are fetched at runtime, not mounted
+            url = (
+                f"hf://buckets/{src.id}"
+                if src.kind == "bucket"
+                else f"hf://datasets/{src.id}"
+            )
             mount = src.mount or f"/data/{idx}"
-            volumes.append(Volume(f"hf://buckets/{src.id}", mount, "ro"))
+            volumes.append(Volume(url, mount, "ro"))
             idx += 1
     return JobSpec(
         name=run,

tests/test_pipeline_contracts.py

@@ -148,11 +148,52 @@ def test_unverified_license_is_blocking(tmp_path: Path) -> None:
 
 def test_shipped_contracts_validate() -> None:
     problems = validate_recipe_dir(RECIPES, CORPORA)
-    unverified = [p for p in problems if "UNVERIFIED" in p]
-    unexpected = [p for p in problems if "UNVERIFIED" not in p]
+    blocked = [p for p in problems if "owner review" in p]
+    unexpected = [p for p in problems if "owner review" not in p]
     assert unexpected == [], f"contract drift: {unexpected}"
-    for p in unverified:
-        assert "urd-dia" in p or "fas-dia" in p, p
+    open_corpora = {"urd-dia", "qcri-wiki", "urd-g2p-dict"}
+    for p in blocked:
+        assert any(c in p for c in open_corpora), p
+    assert not any("fas-dia" in p for p in blocked), (
+        "fas CC0 verification must keep the fas lane clear"
+    )
+
+
+def test_restricted_but_read_license_blocks(tmp_path: Path) -> None:
+    from pipeline.contracts import _license_blocked  # noqa: PLC0415
+
+    assert _license_blocked("LicenseRef-non-commercial")
+    assert _license_blocked("LicenseRef-no-license-found")
+    assert not _license_blocked("CC0-1.0")
+    assert not _license_blocked("GPLv2")
+
+    import yaml  # noqa: PLC0415
+
+    corpus = {
+        "id": "xx-src",
+        "task": "transliteration",
+        "sources": [
+            {"kind": "hf-dataset", "id": "org/ds", "license": "LicenseRef-non-commercial"}
+        ],
+        "splits": {
+            "train": {"path": "train.jsonl", "rows": 10},
+            "test": {"path": "test.jsonl", "rows": 5},
+        },
+    }
+    recipe = {
+        "id": "xx-test",
+        "task": "transliteration",
+        "corpora": {"train": "xx-src@v1", "test": "xx-src@v1#test"},
+        "model": "byt5-small",
+        "stages": [{"stage": "sft", "params": {"epochs": 1}}],
+        "gates": {"quality": {"cer_max": 0.1}, "parity": {"agree_min": 0.98}},
+    }
+    (tmp_path / "corpora").mkdir()
+    (tmp_path / "recipes").mkdir()
+    (tmp_path / "corpora" / "xx-src.yaml").write_text(yaml.safe_dump(corpus))
+    (tmp_path / "recipes" / "xx-test.yaml").write_text(yaml.safe_dump(recipe))
+    problems = validate_recipe_dir(tmp_path / "recipes", tmp_path / "corpora")
+    assert any("LicenseRef-non-commercial" in p and "owner review" in p for p in problems)
 
 
 def test_load_corpus_roundtrip() -> None: