Repository navigation
FIX Count only rendered goals toward ComicJailbreak max_examples - #3590
Merged
Richard Lundeen (richlundeen) merged 2 commits intoOct 29, 2026
Conversation
A goal with no text for any selected template rendered nothing but still counted toward max_examples. Most goals have no Code text, so templates=["code"] with max_examples=10 rendered no goal at all and failed with "SeedDataset cannot be empty"; templates=["instruction"] rendered 2. The cap was also checked only after a goal was rendered, so max_examples=0 still rendered one goal. Count only goals that rendered something, and reject max_examples below 1. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Richard Lundeen (richlundeen)
approved these changes
Oct 28, 2026
Richard Lundeen (richlundeen)
left a comment
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
Only goals that produce seeds now count toward max_examples. The regression case covers empty template text before valid goals, and invalid limits are rejected.
No blocking issues found by code inspection. Tests were not run for this review. AI-assisted review (GitHub Copilot).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3589.
_ComicJailbreakDatasetcounted every goal it looked at towardmax_examples, including goals that have no text for any selected template and render nothing. Most goals have noCodetext (213 of 300), sotemplates=["code"], max_examples=10rendered no goal and failed withSeedDataset cannot be empty, andtemplates=["instruction"], max_examples=10rendered 2 goals. The cap was also checked only after a goal was rendered, somax_examples=0still rendered one goal.Changes:
max_examplesonly when it rendered at least one template.__init__raisesValueErrorwhenmax_examplesis below 1.0cannot work, because an emptySeedDatasetis invalid. The docstring now states both rules.Tests and Documentation
tests/unit/datasets/test_comic_jailbreak_dataset.py: withtemplates=["code"], max_examples=2, the goals withoutCodetext are skipped and the next two goals withCodetext are rendered;max_examplesof0and-1are rejected. These tests fail onmain; the existingmax_examplestest is unchanged.codeand forinstruction, andmax_examples=0raises.pytest tests/unit/datasets/test_comic_jailbreak_dataset.py: 38 passed.ruff check,ruff formatandty checkpass on the changed files.🤖 Generated with Claude Code