Skip to content

feat(WO67/B6): tha-phonemic lexicon corpus (ORST x royin) #332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 21, 2026
Merged

feat(WO67/B6): tha-phonemic lexicon corpus (ORST x royin) #332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion TODO.sota/67-b6-thai-lexicon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# WO67 — B6: Thai — lexicon-first, three layers (PER < 2)

Status: PLANNED · launchable now
Status: IN_PROGRESS (lexicon phase landed 2026-10-21)
Lexicon phase DONE: corpora/tha-phonemic.yaml -- 28,644 train / 3,225
test (word -> phonemic-with-tones pairs, ORST vocabulary x pythainlp
royin engine); 6,291 engine-corrupt rows filtered and documented;
bucket run-080-thai-lexicon, size-verified. NOTE the layer honesty:
royin emits PHONEMIC+tones, NOT RTGS -- the RTGS layer requires
encoding bgnpcgn-tha (a BGN/PCGN gap-system row: ROMANIZATION OF
THAI.pdf, whole-document transcription) then the rule composite.
NEXT: bgnpcgn-tha encoding; hit-rate metric over the eval set; the
three-layer (RTGS/IPA/IPA+tone) eval harness.
Goal: published PER < 2; lexicon hit-rate reported; tone accuracy for
the TTS layer (RTGS / IPA / IPA+tone tracked separately).
Method: Royal Institute dictionary (or equivalent licensed lexicon) as
Expand Down
19 changes: 19 additions & 0 deletions corpora/tha-phonemic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
id: tha-phonemic
task: g2p
sources:
# ORST (Royal Institute Dictionary) wordlist via pythainlp's corpus
# packaging (Apache-2.0); phonemic readings generated by pythainlp's
# "royin" engine. Owner note: the underlying wordlist is Royal
# Institute of Thailand material redistributed by pythainlp -- terms
# recorded here for awareness, SPDX-open packaging clears the gate.
# 6,291 rows (17%) dropped: the royin engine repeats syllables on
# certain clusters (e.g. rust-class รักษา words) -- generator defect,
# filtered, documented in WO67.
- kind: bucket
id: Interscript/isx-training/run-080-thai-lexicon
revision: run-080-thai-lexicon
license: Apache-2.0
mount: /data-tha
splits:
train: {path: tha-phonemic-train.jsonl, rows: 28644, sha256: 68a799711aa0d4ca8e3d1796cdcaf1b8599c82f0b7390d7d5ea3f53442c39c7d}
test: {path: tha-phonemic-test.jsonl, rows: 3225, sha256: a5abe80b9c9f119fa8bdeb51ca22a931b2c5b2d2e946d9ea190161f937b53966}
Loading