diff --git a/TODO.sota/81-vrr-metric.md b/TODO.sota/81-vrr-metric.md index 6ab55c69..8c4f01cc 100644 --- a/TODO.sota/81-vrr-metric.md +++ b/TODO.sota/81-vrr-metric.md @@ -1,6 +1,21 @@ # WO81 โ€” VRR: valid-reading-rate, the ambiguity-aware metric -Status: PLANNED ยท Plane B (B3 closure requirement, WO56 structural rule 1) +Status: IN_PROGRESS โ€” metric SHIPPED (models#340: build_lexicon / +score_predictions_vrr / score_text_vrr, word-level); first verdict +computed 2026-10-22 (run-081-vrr, cpu job; tools/vrr_lexicon_verdict.py): + +| run | student VRR | teacher VRR | words | +|---|---|---|---| +| run-006 (rung 2.0) | 0.7974 | 0.8101 | 51,808 | +| run-007 (rung 2.1) | 0.8004 | 0.8101 | 51,808 | + +Lexicon: tashkeela-full train (declared source โ€” classical register), +309,508 skeletons. Verdict artifact: +bucket Interscript/isx-training/run-081-vrr/vrr-verdict.json (release +asset rides E3). NEXT: gold-column ceiling (oracle VRR) to separate +model error from lexicon coverage; publish VRR into the models.yaml +metric blocks once the rung-2.0 DER provenance question (WO64/B9) is +resolved. Single-gold DER systematically undercounts legitimate ambiguity: when a skeleton has several dictionary-licensed readings, scoring against one