[Triage] [extends: 'Benchmark gate metadata does not record the lm-eval version'] lm-eval's BOS default flips inside the declared range (0.4.4: never add BOS for causal LMs; 0.4.13: tokeni… (CR-A16-115) #826

Needs triage. Reviewers disagreed on this finding, so it is not counted in the backlog. Decide whether it is a real defect: if yes, set severity and wave; if not, close it with the reason.

CR-A16-115 — [extends: 'Benchmark gate metadata does not record the lm-eval version'] lm-eval's BOS default flips inside the declared range (0.4.4: never add BOS for causal LMs; 0.4.13: tokenizer default), so bas…

severity Low · confidence disputed · single reviewer · not reproduced (code trace)

What happens. The docs tell operators to derive min_score from a --benchmark-only baseline (docs/usermanuals/en/evaluation/benchmarks.md:68-96). At lm-eval 0.4.4 HFLM defaults add_bos_token=False (huggingface.py:78), and tok_encode passes add_special_tokens=False for causal models (huggingface.py:750). Gemma is the only exception. At 0.4.13 add_bos_token defaults to None (huggingface.py:94), and _add_special_kwargs returns {} (models/utils.py:982), so the tokenizer's own default applies: Llama, Mistral and other BOS-adding tokenizers get a BOS at 0.4.13 but not at 0.4.4. ForgeLM pins only lm-eval>=0.4.4,<1.0.0 and records no lm-eval version, so a baseline and a later gate can silently score different token sequences. The 2-shot prompt text itself was identical across versions on disjoint dev/test splits (request-list sha256 prefix d002635f140fafe0 at both versions). 0.4.4 additionally drops few-shot examples equal to the test doc and 0.4.13 does not, which only matters when splits overlap. fewshot_as_multiturn changed from False to True, but it has no effect while apply_chat_template=False, which ForgeLM always leaves at the default (0.4.13 api/task.py:1032-1040).

Failure scenario. A baseline is taken with lm-eval 0.4.4 and the CI image later resolves 0.4.13. For a Llama-family model every prompt gains a BOS token, scores shift, and the gate passes or fails on a measurement change rather than a model change, with no version recorded to explain it.

Where.

• pyproject.toml:84
• forgelm/benchmark.py:188
• forgelm/benchmark.py:100-137
• docs/usermanuals/en/evaluation/benchmarks.md:68-96

Evidence.

• Source in the unzipped wheels at the lines cited. harness3.py run gives identical loglikelihood request lists at 0.4.4 and 0.4.13 (cmp IDENTICAL). The BOS effect is not demonstrated dynamically because tiny-qwen has no BOS token.

Recommended fix. Record lm_eval.version (and the resolved add_bos_token) in benchmark_results.json and the audit event. Consider passing add_bos_token explicitly, or narrowing the lm-eval range.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}