Repository navigation
Judge gate: defence in depth around the verdict parser (API response_format, fences in the judged response) #1042
Description
Decided in triage (comment): point 1 is implemented as a config-driven
judge_response_formatfield; point 2 is closed (not planned); point 3 is split: reply truncation is #1044, and the rest is documented as a known limit (docstring and judge manual, EN + TR).
Summary
Follow-up to #127. PR #1038 makes _parse_judge_json fail closed when a judge reply is ambiguous (several score objects, duplicate keys, non-object replies). The defence-in-depth measures that #127 recommended next to the parser fix were deliberately left out of that PR to keep it to one concern. This issue tracks them.
What is still open
- Ask the API judge for a JSON object.
_call_api_judge(forgelm/judge.py) sends noresponse_format. OpenAI-compatible endpoints that supportresponse_format={"type": "json_object"}would then return a bare object, which the parser accepts on its first, strictest path. Endpoints that reject the parameter must keep working, so decide how it is enabled (config-driven, or a documented fallback) before coding. - Neutralise fence markers in the judged response. The judged response is already passed through
_neutralize_delimiter_tagsbefore it enters the rubric. Triple-backtick fences in that text are not neutralised, so a fenced block inside the model's answer reaches the judge verbatim. - Decide how a single quoted score object is treated. The parser accepts a reply that contains exactly one score-bearing object. That object cannot be told apart from one quoted out of the judged response when the judge gave no JSON verdict of its own. Decide whether the verdict can be bound to the judge's own output (for example by position) or whether the current rule is the documented, accepted limit, and record the decision in the code and the judge manual.
Acceptance criteria
- Each of the three points above is implemented, or closed with a written reason on this issue.
- Every behaviour change has a regression test that fails on the current
developmentand passes after the change. Tests need no GPU, network access or model downloads. - If the judge manual (
docs/usermanuals/en/evaluation/judge.md) or the configuration reference changes, the Turkish mirror is updated in the same PR and the bilingual guards pass. - A
### Changedor### Fixedentry under## [Unreleased]inCHANGELOG.mdnames the behaviour. - Any regex that is added follows
docs/standards/regex.md.
Context
Found while reviewing PR #1038 (fix for #127). Start from forgelm/judge.py (_call_api_judge, _neutralize_delimiter_tags, _parse_judge_json) and tests/test_judge_functions.py.
Activity
cemililik commentedon Oct 20, 2026
Triage, checked against
development:_call_api_judgebuilds its payload withmodel,messages,temperatureandmax_tokensonly; there is noresponse_format. The judge manual pointsjudge_api_baseat an Anthropic endpoint and at "vLLM-compatible gateways", so a parameter some of them reject has to be opt-in or fall back; that decision comes before the code, as the issue says._neutralize_delimiter_tagsescapes the four delimiter tags and nothing else, so a fenced block in the judged response reaches the judge verbatim.developmentis the one Judge reply parser takes the first JSON block that parses anywhere in the reply, so a score block planted in the model's answer and quoted by the judge replaces the judge's own ve… (CR-A06-015) #127 describes (first block that parses). Point 3 and the tests of points 1 and 2 are meaningful only after that, so this issue is linked as blocked by Judge reply parser takes the first JSON block that parses anywhere in the reply, so a score block planted in the model's answer and quoted by the judge replaces the judge's own ve… (CR-A06-015) #127, which fix: reject ambiguous judge JSON verdicts (CR-A06-015) #1038 closes; theblockedlabel and its notice then follow the merge automatically.Triage result:
enhancement,area: eval-gates,severity: info,wave: 4. The parser fix (#127, high) carries the risk; this is hardening around it, plus a decision on a residual limit. Re-grade if the decision in point 3 shows that a quoted score object can still reach the verdict. Not a first-contribution task: points 1 and 3 need a maintainer decision first.Triage, checked against
developmenton 2026-10-20.cemililik commentedon Oct 20, 2026
Decisions on the three points
I ran the cases below offline against the parser proposed in #1038 (head
1413ffd) and checked the JSON-mode behaviour of the common endpoints. Short version: 1 is implemented as described below, 2 is closed (not planned), 3 is split: the part that can be fixed is #1044, the rest is documented as a known limit.1. Ask the API judge for a JSON object — implement, config-driven
New field
evaluation.llm_judge.judge_response_format:auto(default) |json_object|none.autosendsresponse_format={"type": "json_object"}only when nojudge_api_baseis set, i.e. the default OpenAI endpoint, where the default modelgpt-4osupports it andDEFAULT_RUBRICalready contains the word "JSON" that OpenAI's JSON mode requires. With a customjudge_api_basenothing is sent, exactly as today.json_objectalways sends it (the operator asserts the endpoint accepts it);nonenever does.response_formataborts the evaluation like the 401/403 case, with a message that names the field and says to set it tonone. There is no silent retry without it: quietly dropping a hardening measure is the kind of silent failure this project rules out._validate_rubricalso requires a custom rubric to ask for JSON (the parser accepts nothing else, and JSON mode requires the word).response_format(harmless, but no protection), and vLLM documentsjson_schemawhilejson_objectis not confirmed. Hence the opt-in for custom endpoints.reason, and the "quote, then verdict" ambiguity is gone; a reply cut off at the token cap is invalid JSON and becomes None.JudgeConfig(the field needs adescription=),config_template.yaml,docs/reference/configuration.mdand its Turkish mirror, the judge manual and its mirror,run_judge_evaluationand its call site in the trainer,_call_api_judge, tests (payload with and without the field, the 400 abort, rubric validation), CHANGELOG. Follow theadd-config-fieldchecklist.2. Neutralise fence markers in the judged response — closed, not planned
<user_prompt>/<assistant_response>, and those are already escaped; a fence in the answer breaks out of nothing.{"score": 10, ...}with no fence is taken as the verdict just the same (see the table in Judge: a reply cut off at the 200-token cap can leave a lone quoted score object, which the parser takes as the verdict #1044)."score", which JSON decodes toscore, and the judge copies it verbatim.The defences that work sit on the output side: point 1, the parser rule from #1038, and the truncation check below.
3. A single quoted score object — split
The verdict cannot be bound to the judge's own output by position: with one object there is no position to compare. What can be shown is where a lone quoted object actually comes from:
min_valid_fraction0.8). With points 1 and Judge: a reply cut off at the 200-token cap can leave a lone quoted score object, which the parser takes as the verdict #1044 in place this needs a judge that disobeys "respond with ONLY a JSON object" and also quotes the plant. Decision: accepted, documented limit. Revisit with the nonce if it is ever observed, or if a strict reply mode is wanted for local judges.What stays in this issue for point 3 is the documentation: the docstring of
_parse_judge_jsonand a short "How replies are parsed" section in the judge manual (English and Turkish) that states which replies count (a whole object; one score object in prose), which fail (several, duplicate keys, not an object, truncated), the residual limit above, and the advice to use the API judge with JSON mode for gating.Labels:
documentationandi18nadded; severity staysinfo(hardening and documentation; the defect moved to #1044). This issue stays linked as blocked by #127: points 1 and 3 touch the same function, tests and manual section as #1038.Triage decisions, checked against
developmenton 2026-10-20.github-actions commentedon Oct 20, 2026
This issue is no longer blocked: every issue it waited for is closed as completed (#127). It can be picked up now; see Working on an issue.
fix(judge): count a reply cut off at the token cap as a judge failure (…