From 54d04efb6aa2806fcdeb2b6041c807e30023e12b Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 17 Oct 2026 22:20:08 +0800 Subject: [PATCH 1/4] =?UTF-8?q?feat(pipeline):=20WO52=20M4=20=E2=80=94=20a?= =?UTF-8?q?ra/urd-g2p=20recipes=20+=20corpus=20license=20provenance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the recipe registry across all four shippable lanes: - ara-diac: oracle_relabel (D_*) + sft (O_*) stages routing to the WO39 chain; gate = run-047's 3.5607 first-alt WER line - urd-g2p: sft via train_urd_g2p_hf.py; CER gate 0.17 - StageSpec gains env_prefix (U/D/O legacy conventions); planner mounts hf-dataset sources (hf://datasets/) as well as buckets - corpora registry: qcri-wiki + urd-g2p-dict added; fas-dia verified CC0-1.0 (HF cardData via API) — its license gate now opens; urd-dia/qcri-wiki/urd-g2p-dict stay UNVERIFIED with precise pointers. corpora/PROVENANCE.md records the evidence per corpus (incl. tashkeela GPLv2 note and micsell no-license exclusion) - corpus id grammar split from recipe id grammar (source names are not lang-task ids) The shipped-contracts test pins the exact open-license set, so the gate state cannot drift silently. Suite 229 green. --- TODO.sota/52-unified-framework.md | 8 +++++++- corpora/PROVENANCE.md | 20 ++++++++++++++++++++ corpora/fas-dia.yaml | 13 +++++++++++-- corpora/qcri-wiki.yaml | 16 ++++++++++++++++ corpora/urd-dia.yaml | 7 +++++-- corpora/urd-g2p-dict.yaml | 18 ++++++++++++++++++ recipes/ara-diac.yaml | 22 ++++++++++++++++++++++ recipes/urd-g2p.yaml | 15 +++++++++++++++ src/pipeline/contracts.py | 6 ++++-- src/pipeline/planner.py | 16 +++++++++++----- tests/test_pipeline_contracts.py | 6 +++++- tests/test_pipeline_plan_gate.py | 14 ++++++++++++++ 12 files changed, 148 insertions(+), 13 deletions(-) create mode 100644 corpora/PROVENANCE.md create mode 100644 corpora/qcri-wiki.yaml create mode 100644 corpora/urd-g2p-dict.yaml create mode 100644 recipes/ara-diac.yaml create mode 100644 recipes/urd-g2p.yaml diff --git a/TODO.sota/52-unified-framework.md b/TODO.sota/52-unified-framework.md index 4deaeb24..e986ecca 100644 --- a/TODO.sota/52-unified-framework.md +++ b/TODO.sota/52-unified-framework.md @@ -1,6 +1,12 @@ # WO52 — The unified Interscript ML framework (recipes + spine + registry) -Status: PROPOSAL (owner review before implementation) +Status: **IN IMPLEMENTATION** — owner-approved 2026-10-17 +Progress: M0 contracts + M1 runner + M2 gates/planner merged (#294); +M4 recipes landed (ara-diac, urd-g2p + corpora registry + license +provenance — `corpora/PROVENANCE.md`). Remaining: M3 (pins: rows + +sha256 manifests; probe/zipcheck absorbed as gate inputs), M5 +(models.yaml resolver in clients, conformance fixtures, exporter +attention-mask option, deprecate src/gpu/modal_*). Priority: strategic — makes every future language/technique a config PR ## Diagnosis (honest) diff --git a/corpora/PROVENANCE.md b/corpora/PROVENANCE.md new file mode 100644 index 00000000..c06d283e --- /dev/null +++ b/corpora/PROVENANCE.md @@ -0,0 +1,20 @@ +# Corpus license provenance — the license gate's evidence file + +Verified with source links on 2026-10-17. `UNVERIFIED` below is a +blocking state: `isxml gate` refuses to publish any model trained on +that corpus until the terms are read and recorded here. + +| corpus | upstream | license | status | +|---|---|---|---| +| fas-dia | [MahtaFetrat/HomoRich-G2P-Persian](https://huggingface.co/datasets/MahtaFetrat/HomoRich-G2P-Persian) | **CC0-1.0** (HF cardData, verified via API) | CLEAR | +| urd-dia | paper-derived dataset (rababa-urdu/docs/paper-urdu/main.pdf; uploaded from /opt/urdu/data per rababa-urdu/modal_app_diacrit.py) | unknown | UNVERIFIED — read the paper's dataset terms | +| urd-g2p-dict | [humair025/urdu-g2p-dictionary](https://huggingface.co/datasets/humair025/urdu-g2p-dictionary) | HF card says `other` (custom terms) | UNVERIFIED — read card text | +| qcri-wiki | QCRI published Wikipedia_20240420.diac.jsonl (WO17: "provenance + license recorded in the dataset repo") | not yet read | UNVERIFIED — locate QCRI's dataset repo and read terms | +| tashkeela (classical, r8-mined) | tashkeela-processed via SourceForge/Kaggle (rababa-tashkeela/README.adoc) | **GPLv2** "as per the original datasets" | CLEAR (copyleft — note for weights distribution review) | + +Notes: +- Derived corpora (staged jsonl splits) inherit their upstream license. +- GPLv2 tashkeela: training-usage vs weight-redistribution implications + are an owner decision — recorded here so the question is visible. +- micsell/hebrew_speech_kan_nikud: NO license (WO43) — excluded from + the registry entirely; owner ask drafted in TODO.sota/43. diff --git a/corpora/fas-dia.yaml b/corpora/fas-dia.yaml index d6c3a78b..4f661143 100644 --- a/corpora/fas-dia.yaml +++ b/corpora/fas-dia.yaml @@ -1,10 +1,19 @@ id: fas-dia task: diacritization sources: + # Upstream: verified CC0-1.0 via HF cardData (2026-10-17). + # haraqat targets derived deterministically from HomoRich's Mapped + # Phoneme field per rababa-farsi/docs/RESULTS.md — derived work of a + # CC0 dataset stays CC0. + - kind: hf-dataset + id: MahtaFetrat/HomoRich-G2P-Persian + revision: main + license: CC0-1.0 + # Staged jsonl splits consumed by training (bucket copy of the above). - kind: bucket id: Interscript/isx-training/run-076-fas-diac-retrain revision: run-076 - license: UNVERIFIED # recovered-corpus provenance review open — gate blocks publish until resolved + license: CC0-1.0 splits: - train: {path: train.jsonl} # pin rows+sha256 from bucket manifest (M2) + train: {path: train.jsonl} # pin rows+sha256 from bucket manifest (M3) test: {path: test.jsonl} diff --git a/corpora/qcri-wiki.yaml b/corpora/qcri-wiki.yaml new file mode 100644 index 00000000..d5850b3a --- /dev/null +++ b/corpora/qcri-wiki.yaml @@ -0,0 +1,16 @@ +id: qcri-wiki +task: diacritization +sources: + # QCRI's published Wikipedia_20240420.diac.jsonl (~5M words, BiLSTM + # silver ~3% WER), staged on our dataset repo for training mounts. + # WO17: "provenance + license recorded in the dataset repo" — the + # QCRI terms still need to be read; gate blocks publish until then. + - kind: hf-dataset + id: Interscript/arabic-r8 + revision: main + license: UNVERIFIED # read QCRI dataset terms — gate blocks publish until resolved + mount: /train_data +splits: + train: {path: qcri-wiki.jsonl} # pin rows+sha256 from dataset manifest (M3) + test: {path: wikinews/} # WikiNews-2024 multiref (OOD register gate) + sadeed: {path: sadeeddiac-25.parquet} # SadeedDiac-25 (identity gate) diff --git a/corpora/urd-dia.yaml b/corpora/urd-dia.yaml index ffba1d34..3edf7bae 100644 --- a/corpora/urd-dia.yaml +++ b/corpora/urd-dia.yaml @@ -1,11 +1,14 @@ id: urd-dia task: diacritization sources: + # Upstream: paper-derived Urdu diacritization dataset — terms not yet + # read. See rababa-urdu/docs/paper-urdu/main.pdf (upload path was + # /opt/urdu/data in rababa-urdu/modal_app_diacrit.py). - kind: bucket id: rhtse/jobs-artifacts/rababa-urdu-cc174b96 revision: data-diacrit - license: UNVERIFIED # provenance review open — gate blocks publish until resolved + license: UNVERIFIED # read paper-urdu dataset terms — gate blocks publish until resolved mount: /data-dia splits: - train: {path: data-diacrit/train.jsonl} # pin rows+sha256 from bucket manifest (M2) + train: {path: data-diacrit/train.jsonl} # pin rows+sha256 from bucket manifest (M3) test: {path: data-diacrit/test.jsonl} diff --git a/corpora/urd-g2p-dict.yaml b/corpora/urd-g2p-dict.yaml new file mode 100644 index 00000000..14825933 --- /dev/null +++ b/corpora/urd-g2p-dict.yaml @@ -0,0 +1,18 @@ +id: urd-g2p-dict +task: g2p +sources: + # HF card declares "license: other" (custom terms) — read the card + # text before publishing anything trained on it. + - kind: hf-dataset + id: humair025/urdu-g2p-dictionary + revision: main + license: UNVERIFIED # custom terms ("other") — read card text — gate blocks publish + # Staged copy consumed by training (610K pairs via rababa-urdu/data). + - kind: bucket + id: rhtse/jobs-artifacts/rababa-urdu-cc174b96 + revision: data + license: UNVERIFIED # inherits upstream custom terms + mount: /data +splits: + train: {path: data/train.jsonl} # pin rows+sha256 from bucket manifest (M3) + test: {path: data/test.jsonl} diff --git a/recipes/ara-diac.yaml b/recipes/ara-diac.yaml new file mode 100644 index 00000000..9a0b8e34 --- /dev/null +++ b/recipes/ara-diac.yaml @@ -0,0 +1,22 @@ +id: ara-diac +task: diacritization +corpora: + train: qcri-wiki@2024-04-20 + test: qcri-wiki@2024-04-20#test +model: byt5-base +stages: + - stage: oracle_relabel + env_prefix: D + route: legacy:interscript-train/distill_oracle_hf.py + params: {subset: 300000} + - stage: sft + env_prefix: O + route: legacy:interscript-train/continue_oracle_train_hf.py + params: {epochs: 2} +gates: + quality: {wer_max: 3.5607} # run-047 first-alt line; stretch 2.70 (QCRI frontier parity) + parity: {mode: single-row, agree_min: 0.98, three_way: true} +negative_results: + - r9-yallamorph convention-density drift — r7 stays canonical + - RIDE residual-displacement arm negative + - LLM teachers hallucinate haraqat — banned for diacritization labels diff --git a/recipes/urd-g2p.yaml b/recipes/urd-g2p.yaml new file mode 100644 index 00000000..8248db9e --- /dev/null +++ b/recipes/urd-g2p.yaml @@ -0,0 +1,15 @@ +id: urd-g2p +task: g2p +corpora: + train: urd-g2p-dict@v1 + test: urd-g2p-dict@v1#test +model: byt5-base +stages: + - stage: sft + route: legacy:interscript-train/train_urd_g2p_hf.py +gates: + quality: {cer_max: 0.17} # torch greedy 0.16 (artifact ≡ torch under the WO51 single-row contract) + parity: {mode: single-row, agree_min: 0.98, three_way: true} +negative_results: + - dynamic-int8 quantization destroys byt5 decode (0.44 vs 0.16 CER) + - batched-without-attention-mask parity harness measures nothing (WO51) diff --git a/src/pipeline/contracts.py b/src/pipeline/contracts.py index 350af8bb..e3726d4e 100644 --- a/src/pipeline/contracts.py +++ b/src/pipeline/contracts.py @@ -23,7 +23,8 @@ KNOWN_STAGES = frozenset( {"sft", "oracle_relabel", "distill", "decode_policy", "plane", "lexicon_fusion"} ) -LANG_RE = re.compile(r"^[a-z]{2,3}-[a-z0-9-]+$") +LANG_RE = re.compile(r"^[a-z]{2,3}-[a-z0-9-]+$") # recipes: - +CORPUS_ID_RE = re.compile(r"^[a-z][a-z0-9-]*$") # corpora: source names CORPUS_REF_RE = re.compile( r"^(?P[a-z][a-z0-9-]*)@" r"(?P\d{4}-\d{2}-\d{2}|v\d+(?:\.\d+)*)" @@ -52,7 +53,7 @@ class SplitSpec(_Strict): class CorpusSpec(_Strict): - id: str = Field(pattern=LANG_RE.pattern) + id: str = Field(pattern=CORPUS_ID_RE.pattern) task: TaskKind sources: list[CorpusSource] = Field(min_length=1) splits: dict[str, SplitSpec] @@ -95,6 +96,7 @@ class StageSpec(_Strict): stage: str route: str | None = None params: dict[str, Any] = Field(default_factory=dict) + env_prefix: str = Field(default="U", pattern=r"^[A-Z]{1,3}$") @model_validator(mode="after") def _implementable(self) -> StageSpec: diff --git a/src/pipeline/planner.py b/src/pipeline/planner.py index 5c3ce7c9..79efc8c3 100644 --- a/src/pipeline/planner.py +++ b/src/pipeline/planner.py @@ -33,8 +33,9 @@ def plan_stage( return None corpora_dir = corpora_dir or DEFAULT_CORPORA_DIR script = stage.route.split(":", 1)[-1].rsplit("/", 1)[-1] - env = {f"U_{k.upper()}": str(v) for k, v in stage.params.items()} - env["U_RUN"] = run + prefix = stage.env_prefix + env = {f"{prefix}_{k.upper()}": str(v) for k, v in stage.params.items()} + env[f"{prefix}_RUN"] = run volumes = [Volume(CKPT_BUCKET, CKPT_MOUNT, "rw")] seen: set[str] = set() idx = 0 @@ -45,10 +46,15 @@ def plan_stage( seen.add(name) corpus = load_corpus(corpora_dir / f"{name}.yaml") for src in corpus.sources: - if src.kind != "bucket": - continue + if src.kind == "gh-release": + continue # release assets are fetched at runtime, not mounted + url = ( + f"hf://buckets/{src.id}" + if src.kind == "bucket" + else f"hf://datasets/{src.id}" + ) mount = src.mount or f"/data/{idx}" - volumes.append(Volume(f"hf://buckets/{src.id}", mount, "ro")) + volumes.append(Volume(url, mount, "ro")) idx += 1 return JobSpec( name=run, diff --git a/tests/test_pipeline_contracts.py b/tests/test_pipeline_contracts.py index 42694648..6d97951f 100644 --- a/tests/test_pipeline_contracts.py +++ b/tests/test_pipeline_contracts.py @@ -151,8 +151,12 @@ def test_shipped_contracts_validate() -> None: unverified = [p for p in problems if "UNVERIFIED" in p] unexpected = [p for p in problems if "UNVERIFIED" not in p] assert unexpected == [], f"contract drift: {unexpected}" + open_corpora = {"urd-dia", "qcri-wiki", "urd-g2p-dict"} for p in unverified: - assert "urd-dia" in p or "fas-dia" in p, p + assert any(c in p for c in open_corpora), p + assert not any("fas-dia" in p for p in unverified), ( + "fas CC0 verification must keep the fas lane clear" + ) def test_load_corpus_roundtrip() -> None: diff --git a/tests/test_pipeline_plan_gate.py b/tests/test_pipeline_plan_gate.py index 09b883f5..cc8c9745 100644 --- a/tests/test_pipeline_plan_gate.py +++ b/tests/test_pipeline_plan_gate.py @@ -47,6 +47,20 @@ def test_metadata_stage_yields_no_job() -> None: assert plan_stage(recipe, recipe.stages[1], run="r1") is None +def test_env_prefix_and_dataset_mounts() -> None: + recipe = load_recipe(RECIPES / "ara-diac.yaml") + relabel = plan_stage(recipe, recipe.stages[0], run="run-x") + assert relabel is not None + assert relabel.env["D_SUBSET"] == "300000" + assert relabel.env["D_RUN"] == "run-x" + urls = {v.url for v in relabel.volumes} + assert "hf://datasets/Interscript/arabic-r8" in urls + mounts = {v.mount: v.mode for v in relabel.volumes} + assert mounts["/train_data"] == "ro" and mounts["/ckpt"] == "rw" + sft = plan_stage(recipe, recipe.stages[1], run="run-x") + assert sft.env["O_EPOCHS"] == "2" and sft.env["O_RUN"] == "run-x" + + def test_quality_gate() -> None: recipe = load_recipe(RECIPES / "urd-diac.yaml") ok = evaluate(recipe, CORPORA, metrics={"cer": 0.0352}, parity=None) From 95f322cf7a9d031d90cb0da40b933657bf8874db Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 17 Oct 2026 22:25:38 +0800 Subject: [PATCH 2/4] =?UTF-8?q?fix(pipeline):=20license=20verdicts=20verif?= =?UTF-8?q?ied=20=E2=80=94=20LicenseRef=20blocking=20state?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified all four corpora upstream licenses from primary sources: fas CC0-1.0 (gate opens), urd-g2p non-commercial, qcri-wiki unlicensed (Qatara repo has no license file), urd-dia sources unlocated/no license. Schema now distinguishes UNVERIFIED (unread) from LicenseRef-* (read but restricted) — both block; only SPDX-open licenses clear. PROVENANCE.md carries the evidence + the three-item owner decision queue the gate is holding. --- corpora/PROVENANCE.md | 29 +++++++++++++------- corpora/qcri-wiki.yaml | 6 ++--- corpora/urd-dia.yaml | 9 ++++--- corpora/urd-g2p-dict.yaml | 12 +++++---- src/pipeline/contracts.py | 17 +++++++++--- src/pipeline/gates.py | 10 ++++--- tests/test_pipeline_contracts.py | 45 +++++++++++++++++++++++++++++--- tests/test_pipeline_plan_gate.py | 5 ++-- 8 files changed, 98 insertions(+), 35 deletions(-) diff --git a/corpora/PROVENANCE.md b/corpora/PROVENANCE.md index c06d283e..79958d7d 100644 --- a/corpora/PROVENANCE.md +++ b/corpora/PROVENANCE.md @@ -1,20 +1,29 @@ # Corpus license provenance — the license gate's evidence file -Verified with source links on 2026-10-17. `UNVERIFIED` below is a -blocking state: `isxml gate` refuses to publish any model trained on -that corpus until the terms are read and recorded here. +Verified 2026-10-17. The gate blocks on `UNVERIFIED` (terms not read) +and on `LicenseRef-*` (read but restricted). Only verified-open SPDX +licenses clear it. -| corpus | upstream | license | status | +| corpus | upstream | license | verdict | |---|---|---|---| | fas-dia | [MahtaFetrat/HomoRich-G2P-Persian](https://huggingface.co/datasets/MahtaFetrat/HomoRich-G2P-Persian) | **CC0-1.0** (HF cardData, verified via API) | CLEAR | -| urd-dia | paper-derived dataset (rababa-urdu/docs/paper-urdu/main.pdf; uploaded from /opt/urdu/data per rababa-urdu/modal_app_diacrit.py) | unknown | UNVERIFIED — read the paper's dataset terms | -| urd-g2p-dict | [humair025/urdu-g2p-dictionary](https://huggingface.co/datasets/humair025/urdu-g2p-dictionary) | HF card says `other` (custom terms) | UNVERIFIED — read card text | -| qcri-wiki | QCRI published Wikipedia_20240420.diac.jsonl (WO17: "provenance + license recorded in the dataset repo") | not yet read | UNVERIFIED — locate QCRI's dataset repo and read terms | -| tashkeela (classical, r8-mined) | tashkeela-processed via SourceForge/Kaggle (rababa-tashkeela/README.adoc) | **GPLv2** "as per the original datasets" | CLEAR (copyleft — note for weights distribution review) | +| urd-g2p-dict | [humair025/urdu-g2p-dictionary](https://huggingface.co/datasets/humair025/urdu-g2p-dictionary) | **non-commercial** (HF card `license_name`, terms at github.com/humair-m/urdu-g2p) | BLOCKED — owner: may weights trained on it ship under the BSD release lane? | +| qcri-wiki | QCRI `Wikipedia_20240420.diac.jsonl`; system repo [kdarwish/Qatara](https://github.com/kdarwish/Qatara) carries **no license file** | default copyright | BLOCKED — owner: contact QCRI for terms | +| urd-dia | mixture of the `mahwizzzz` + `humairmunirawn` corpora (rababa-urdu/docs/paper-urdu/main.pdf) | none found on either source (HF/GitHub search) | BLOCKED — owner: contact authors or replace the corpus | +| tashkeela (classical, r8-mined) | tashkeela-processed via SourceForge/Kaggle (rababa-tashkeela/README.adoc) | **GPLv2** "as per the original datasets" | CLEAR for training; copyleft note for weight redistribution | + +## Owner decision queue (what the license gate is holding) + +1. **urd-g2p** — non-commercial data: ship weights research-only, or + retrain on an open replacement before release? +2. **qcri-wiki (ara-diac)** — QCRI has no license on Qatara; their own + models are published. Ask QCRI for explicit terms, or retrain the + ara lane on Tashkeela-derived data only (GPLv2 implications then + apply to weights). +3. **urd-dia** — no licenses found on the mixed sources. Contact the + paper authors, or rebuild the corpus from licensed sources. Notes: - Derived corpora (staged jsonl splits) inherit their upstream license. -- GPLv2 tashkeela: training-usage vs weight-redistribution implications - are an owner decision — recorded here so the question is visible. - micsell/hebrew_speech_kan_nikud: NO license (WO43) — excluded from the registry entirely; owner ask drafted in TODO.sota/43. diff --git a/corpora/qcri-wiki.yaml b/corpora/qcri-wiki.yaml index d5850b3a..1accdbd2 100644 --- a/corpora/qcri-wiki.yaml +++ b/corpora/qcri-wiki.yaml @@ -3,12 +3,12 @@ task: diacritization sources: # QCRI's published Wikipedia_20240420.diac.jsonl (~5M words, BiLSTM # silver ~3% WER), staged on our dataset repo for training mounts. - # WO17: "provenance + license recorded in the dataset repo" — the - # QCRI terms still need to be read; gate blocks publish until then. + # The QCRI system repo (kdarwish/Qatara) carries NO license file — + # default copyright applies. Owner decision: contact QCRI for terms. - kind: hf-dataset id: Interscript/arabic-r8 revision: main - license: UNVERIFIED # read QCRI dataset terms — gate blocks publish until resolved + license: LicenseRef-no-license-found # Qatara repo unlicensed — owner: contact QCRI mount: /train_data splits: train: {path: qcri-wiki.jsonl} # pin rows+sha256 from dataset manifest (M3) diff --git a/corpora/urd-dia.yaml b/corpora/urd-dia.yaml index 3edf7bae..bdec4144 100644 --- a/corpora/urd-dia.yaml +++ b/corpora/urd-dia.yaml @@ -1,13 +1,14 @@ id: urd-dia task: diacritization sources: - # Upstream: paper-derived Urdu diacritization dataset — terms not yet - # read. See rababa-urdu/docs/paper-urdu/main.pdf (upload path was - # /opt/urdu/data in rababa-urdu/modal_app_diacrit.py). + # Upstream: mixture of the mahwizzzz and humairmunirawn Urdu corpora + # (per rababa-urdu/docs/paper-urdu/main.pdf; uploaded from + # /opt/urdu/data in rababa-urdu/modal_app_diacrit.py). No license + # found on either source (HF/GitHub search 2026-10-17). - kind: bucket id: rhtse/jobs-artifacts/rababa-urdu-cc174b96 revision: data-diacrit - license: UNVERIFIED # read paper-urdu dataset terms — gate blocks publish until resolved + license: LicenseRef-unresolved # no license found on sources — owner: contact authors or replace corpus mount: /data-dia splits: train: {path: data-diacrit/train.jsonl} # pin rows+sha256 from bucket manifest (M3) diff --git a/corpora/urd-g2p-dict.yaml b/corpora/urd-g2p-dict.yaml index 14825933..b933ac25 100644 --- a/corpora/urd-g2p-dict.yaml +++ b/corpora/urd-g2p-dict.yaml @@ -1,17 +1,19 @@ id: urd-g2p-dict task: g2p sources: - # HF card declares "license: other" (custom terms) — read the card - # text before publishing anything trained on it. + # Verified via HF card (2026-10-17): license_name "non-commercial", + # terms at github.com/humair-m/urdu-g2p (LICENSE). Owner decision: + # whether weights trained on it may ship under our BSD release lane. - kind: hf-dataset id: humair025/urdu-g2p-dictionary revision: main - license: UNVERIFIED # custom terms ("other") — read card text — gate blocks publish - # Staged copy consumed by training (610K pairs via rababa-urdu/data). + license: LicenseRef-non-commercial # verified restricted — owner: weights-distribution decision + # Staged copy consumed by training (610K pairs via rababa-urdu/data); + # inherits upstream non-commercial terms. - kind: bucket id: rhtse/jobs-artifacts/rababa-urdu-cc174b96 revision: data - license: UNVERIFIED # inherits upstream custom terms + license: LicenseRef-non-commercial mount: /data splits: train: {path: data/train.jsonl} # pin rows+sha256 from bucket manifest (M3) diff --git a/src/pipeline/contracts.py b/src/pipeline/contracts.py index e3726d4e..b64a62aa 100644 --- a/src/pipeline/contracts.py +++ b/src/pipeline/contracts.py @@ -155,6 +155,12 @@ def load_corpus(path: Path) -> CorpusSpec: return CorpusSpec.model_validate(body) +def _license_blocked(lic: str) -> bool: + # UNVERIFIED = terms not read; LicenseRef-* = read but restricted + # (non-commercial, unlicensed, custom) — both need owner review. + return lic == "UNVERIFIED" or lic.startswith("LicenseRef-") + + def validate_recipe_dir(recipes_dir: Path, corpora_dir: Path) -> list[str]: problems: list[str] = [] corpora: dict[str, CorpusSpec] = {} @@ -188,10 +194,15 @@ def validate_recipe_dir(recipes_dir: Path, corpora_dir: Path) -> list[str]: f"recipe task '{r.task}'" ) if r.gates.license and any( - s.license == "UNVERIFIED" for s in c.sources + _license_blocked(s.license) for s in c.sources ): + blocked = ", ".join( + f"{s.id}={s.license}" + for s in c.sources + if _license_blocked(s.license) + ) problems.append( - f"{rpath.name}: corpus '{c.id}' has UNVERIFIED license — " - "resolve before publishing (gates.license)" + f"{rpath.name}: corpus '{c.id}' restricted or unread " + f"license ({blocked}) — owner review required before publishing" ) return problems diff --git a/src/pipeline/gates.py b/src/pipeline/gates.py index 1c8ea917..4184e89f 100644 --- a/src/pipeline/gates.py +++ b/src/pipeline/gates.py @@ -73,17 +73,19 @@ def check_parity(recipe: RecipeSpec, parity: dict | None) -> GateResult: def check_license(recipe: RecipeSpec, corpora_dir: Path) -> GateResult: + from pipeline.contracts import _license_blocked + names = {ref.split("@", 1)[0] for ref in recipe.corpora.values()} for name in sorted(names): corpus = load_corpus(Path(corpora_dir) / f"{name}.yaml") - bad = [s for s in corpus.sources if s.license == "UNVERIFIED"] + bad = [s for s in corpus.sources if _license_blocked(s.license)] if bad: - ids = ", ".join(s.id for s in bad) + ids = ", ".join(f"{s.id}={s.license}" for s in bad) return GateResult( "license", False, - f"corpus '{name}': UNVERIFIED license ({ids}) — " - "resolve provenance before publishing", + f"corpus '{name}': restricted or unread license ({ids}) — " + "owner review required before publishing", ) return GateResult("license", True, f"corpora {sorted(names)} licensed") diff --git a/tests/test_pipeline_contracts.py b/tests/test_pipeline_contracts.py index 6d97951f..4c8393d4 100644 --- a/tests/test_pipeline_contracts.py +++ b/tests/test_pipeline_contracts.py @@ -148,17 +148,54 @@ def test_unverified_license_is_blocking(tmp_path: Path) -> None: def test_shipped_contracts_validate() -> None: problems = validate_recipe_dir(RECIPES, CORPORA) - unverified = [p for p in problems if "UNVERIFIED" in p] - unexpected = [p for p in problems if "UNVERIFIED" not in p] + blocked = [p for p in problems if "owner review" in p] + unexpected = [p for p in problems if "owner review" not in p] assert unexpected == [], f"contract drift: {unexpected}" open_corpora = {"urd-dia", "qcri-wiki", "urd-g2p-dict"} - for p in unverified: + for p in blocked: assert any(c in p for c in open_corpora), p - assert not any("fas-dia" in p for p in unverified), ( + assert not any("fas-dia" in p for p in blocked), ( "fas CC0 verification must keep the fas lane clear" ) +def test_restricted_but_read_license_blocks(tmp_path: Path) -> None: + from pipeline.contracts import _license_blocked # noqa: PLC0415 + + assert _license_blocked("LicenseRef-non-commercial") + assert _license_blocked("LicenseRef-no-license-found") + assert not _license_blocked("CC0-1.0") + assert not _license_blocked("GPLv2") + + import yaml # noqa: PLC0415 + + corpus = { + "id": "xx-src", + "task": "transliteration", + "sources": [ + {"kind": "hf-dataset", "id": "org/ds", "license": "LicenseRef-non-commercial"} + ], + "splits": { + "train": {"path": "train.jsonl", "rows": 10}, + "test": {"path": "test.jsonl", "rows": 5}, + }, + } + recipe = { + "id": "xx-test", + "task": "transliteration", + "corpora": {"train": "xx-src@v1", "test": "xx-src@v1#test"}, + "model": "byt5-small", + "stages": [{"stage": "sft", "params": {"epochs": 1}}], + "gates": {"quality": {"cer_max": 0.1}, "parity": {"agree_min": 0.98}}, + } + (tmp_path / "corpora").mkdir() + (tmp_path / "recipes").mkdir() + (tmp_path / "corpora" / "xx-src.yaml").write_text(yaml.safe_dump(corpus)) + (tmp_path / "recipes" / "xx-test.yaml").write_text(yaml.safe_dump(recipe)) + problems = validate_recipe_dir(tmp_path / "recipes", tmp_path / "corpora") + assert any("LicenseRef-non-commercial" in p and "owner review" in p for p in problems) + + def test_load_corpus_roundtrip() -> None: c = load_corpus(FIXTURES / "corpora" / "urd-dia.yaml") assert c.id == "urd-dia" diff --git a/tests/test_pipeline_plan_gate.py b/tests/test_pipeline_plan_gate.py index cc8c9745..aee7cd33 100644 --- a/tests/test_pipeline_plan_gate.py +++ b/tests/test_pipeline_plan_gate.py @@ -90,11 +90,12 @@ def test_parity_gate_runtime_mode() -> None: def test_license_gate_blocks_publish(tmp_path: Path) -> None: - recipe = load_recipe(RECIPES / "urd-diac.yaml") # refs shipped urd-dia: UNVERIFIED + recipe = load_recipe(RECIPES / "urd-diac.yaml") # refs shipped urd-dia: restricted results = evaluate(recipe, CORPORA, metrics={"cer": 0.0352}, parity=None) lic = [g for g in results if g.name == "license"][0] assert not lic.passed - assert "UNVERIFIED" in lic.detail + assert "owner review" in lic.detail + assert "LicenseRef" in lic.detail def test_gate_results_files_end_to_end(tmp_path: Path) -> None: From f0e72b5e7a0f79f8f08125c7b625edd4f3f3b683 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 18 Oct 2026 00:35:28 +0800 Subject: [PATCH 3/4] =?UTF-8?q?docs(wO47):=20backbone=20diagnosed=20?= =?UTF-8?q?=E2=80=94=20docs=200.52%=20is=20byt5-base;=20run-078=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/47-fas-diac-release.md | 15 +++++++++++++++ recipes/fas-diac.yaml | 9 ++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/TODO.sota/47-fas-diac-release.md b/TODO.sota/47-fas-diac-release.md index 87686b2a..781e40c2 100644 --- a/TODO.sota/47-fas-diac-release.md +++ b/TODO.sota/47-fas-diac-release.md @@ -59,3 +59,18 @@ VERIFIED number in hand. Next-session options: (a) one more epoch arm (the original recipe likely trained longer — check configs/), (b) ship 0.65% as the verified artifact with the docs discrepancy noted. Both are cheap; (a) first. + +## 2026-10-18: the gap is the backbone, not epochs + +- run-076 (small, 3ep): 0.65% · run-077 (small, 6ep): **0.60%** — both + MISS vs docs 0.52%; epochs saturating. +- Gate split verified byte-identical to the docs split (data/test.jsonl + 9,288 rows; sha256 67513aa16ec142ab matches the staged bucket copy) — + the comparison was always apples-to-apples. +- rababa-farsi/docs/RESULTS.md records the docs recipe explicitly: + **ByT5-base, 3 epochs**, 464K derived pairs. Our arms ran byt5-small + — wrong backbone. Corpus count (445,869 after in-training filters of + the 464K usable) is not the variable. +- **run-078-fas-diac-base launched** (byt5-base, 3ep, same corpus and + gate). recipes/fas-diac.yaml corrected to byt5-base. Expect ~27h; + epoch-resume relaunch if the 23h timeout lands first. diff --git a/recipes/fas-diac.yaml b/recipes/fas-diac.yaml index ff549c7c..7cfe4430 100644 --- a/recipes/fas-diac.yaml +++ b/recipes/fas-diac.yaml @@ -3,13 +3,16 @@ task: diacritization corpora: train: fas-dia@v1 test: fas-dia@v1#test -model: byt5-small +model: byt5-base stages: - stage: sft route: legacy:interscript-train/train_fas_diac_hf.py - params: {epochs: 6, bs: 16, accum: 2, gate: 0.0052} + params: {epochs: 3, bs: 16, accum: 2, gate: 0.0052} gates: quality: {cer_max: 0.0052} parity: {mode: single-row, agree_min: 0.98, three_way: true} negative_results: - - 3-epoch arm reached 0.65% vs docs 0.52% — epoch arm running (run-077) + - byt5-small undershoots the docs recipe on the same split and data + (0.65% at 3ep, 0.60% at 6ep vs docs 0.52%) — the docs 0.52% line is + ByT5-base 3ep on the 464K derived pairs (rababa-farsi RESULTS.md); + backbone, not epochs, is the variable (run-077/run-078) From 66890326279c8d13b1ebcf88f73abca9dada70c7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 18 Oct 2026 06:20:08 +0800 Subject: [PATCH 4/4] docs(wO47): run-001 optimizer recipe recovered (5e-4/linear/w500); run-079 launched --- TODO.sota/47-fas-diac-release.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/TODO.sota/47-fas-diac-release.md b/TODO.sota/47-fas-diac-release.md index 781e40c2..0f861e5f 100644 --- a/TODO.sota/47-fas-diac-release.md +++ b/TODO.sota/47-fas-diac-release.md @@ -74,3 +74,13 @@ noted. Both are cheap; (a) first. - **run-078-fas-diac-base launched** (byt5-base, 3ep, same corpus and gate). recipes/fas-diac.yaml corrected to byt5-base. Expect ~27h; epoch-resume relaunch if the 23h timeout lands first. + +### run-078 (byt5-base, our trainer): 0.63% MISS — and the LR was found + +Base beat small on neither surface (0.63 vs 0.60). Root cause found in +modal_app_diacrit.py: run-001 trained at **lr 5e-4, warmup 500, HF +linear decay, wd 0**; our trainer used 3e-4/cosine/warmup-200/wd-0.01. +Epochs/bs/backbone all matched. **run-079-fas-diac-docslr launched** +with the full docs optimizer recipe (trainer now has U_LR/U_WARMUP/ +U_SCHED/U_WD knobs). Matrix so far, same split/corpus: +small-3ep 0.65 · small-6ep 0.60 · base-3ep@3e-4 0.63 · docs@5e-4 0.52?