[devangpratap]

Description

GCG renders the user turn as f"{goal} {control}", and every accepted candidate comes back as a string decoded from the control tokens. On Gemma and on byte-level BPE tokenizers the first control token carries the separator space (▁!, Ġ!), so the decoded candidate starts with a space. Writing it back through control_str then renders goal control with a double space.

LengthPreservingFilter and get_filtered_cands cannot catch this because they re-tokenize the candidate on its own, where the leading space is absorbed into the first token. In the rendered prompt it becomes an extra token.

I ran 6 accepted candidates through LengthPreservingFilter and control_str on the real tokenizers of 10 models. On main:

• Gemma 2 and Gemma 3: the 4-token control grows to 5 after the first accepted candidate and to 6 or 7 later, and neighbouring candidate words merge (suncat).
• Qwen2.5, SmolLM2, DeepSeek-R1-Distill-Qwen, Llama 3.2 and Phi-4: the control keeps 4 tokens, but a stray space token sits between _goal_slice and _control_slice, so the model is optimized on a prompt with an extra token.
• TinyLlama, Phi-3 and Mistral v0.3 are unaffected, since their decode already drops the leading space.

_update_ids now drops that one leading space before rendering. Every write to the control goes through it (__init__, control_str, control_toks). With this change all 10 tokenizers keep 4 control tokens and render no double space, and the control tokens in the prompt are the ones the candidate was scored with. On main they differed for Gemma on every accepted candidate.

The fix is in the rendering rather than in the filters because the decoded form is the right candidate string: logits() and the length check tokenize it on its own, and " z sun" tokenizes to Ġz Ġsun, matching the prompt, while "z sun" gives z Ġsun.

PR 5 of the #2489 series. The adapter-contract PR is next; I already have it in progress.

Tests and Documentation

• New test_decoded_candidate_with_the_separator_space_renders_like_the_original with the byte-level offline tokenizer: setting control_str to the decoded form " ! !" gives the same control, input_ids and control slice as the original prompt. It fails on main.
• uv run pytest tests/unit/executor/promptgen/gcg/: 445 passed, 0 skipped.
• pre-commit run on the changed files passes, including ruff and ty.
• No documentation changes.