Skip to content

WO51: batched-mask per-model findings #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 19, 2026
Merged

WO51: batched-mask per-model findings #318

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions TODO.sota/51-artifact-parity-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,21 @@ numerically faithful, `parity.mode: batched-mask` in the schema since
M0). The exporter grows the option behind an env; batched artifacts
must pass a batched parity probe (`probe_batched_mask.py`) before
release. The WO51 saga is fully closed: both remedies proven.

## Batched-mask per-model findings (2026-10-19, polish round)

Masked exports produced for all three shipped models; batched parity
probes (64 rows, padded+mask) against torch batched-with-mask:

| model | PAD_GAPS | torch CER | ORT-batched CER | agreement | disposition |
|---|---|---|---|---|---|
| urd-diac-base (fp16) | ~1e-4 | 2.94% | 2.36% | 0.953 | batched OK |
| fas-diac-base (fp16) | ~1e-4 | 0.86% | 1.64% | 0.922 | batched degrades ~2x — usable, gate before serving |
| urd-g2p-base (fp16) | ~1e-4 | 15.88% | 22.19% | **0.688** | batched NOT shippable — IPA outputs amplify fp16 divergence |

The attention-mask mechanism is correct everywhere (gaps ~1e-4);
batched decode QUALITY is model-dependent. Consequence: batched-mask
artifacts ship per model only behind a passing `parity.mode:
batched-mask` probe at the recipe's agree_min — urd-g2p stays
single-row-only. Masked zips remain on the bucket as verified
experiments, not release assets.
Loading