Repository navigation
FEAT: Add tool role-play converter and technique support - #3584
Merged
Richard Lundeen (richlundeen) merged 7 commits intoOct 27, 2026
Merged
Richard Lundeen (richlundeen) merged 7 commits into
Richard Lundeen (richlundeen) merged 7 commits into
Conversation
Add the opt-in tools initializer and role-play converter. Share converter controls across chat and technique editing, preserve temporary converter settings, and expose typed system prompts with editable defaults. Require explicit technique selection in technique.execute. Co-authored-by: Copilot App <[email protected]>
Add incoming message context and security evaluation examples to the editable role-play prompt. Co-authored-by: Copilot App <[email protected]>
Keep scenario message inputs separate from scoring objectives, reuse parameter metadata for source prompts, and reject temporary technique file replacement. Remove the extra template parameter schema. Co-authored-by: Copilot App <[email protected]>
Richard Lundeen (richlundeen)
changed the title
Preserve upstream template-parameter identity hashing alongside per-call prompt rendering. Describe the final API in comments and tests without references to intermediate designs. Co-authored-by: Copilot App <[email protected]>
hannahwestra25
approved these changes
Oct 27, 2026
hannahwestra25
left a comment
hannahwestra25
left a comment
Contributor
There was a problem hiding this comment.
nits / minor fixes but lgtm
Reject incompatible prompt inputs before seed adaptation, expose shared capability metadata, preserve the SDK default, and require an explicit GUI selection. Restore responsive converter widths and update seeded picker coverage and typed mock helpers. Co-authored-by: Copilot App <[email protected]>
Reuse the scorer chooser for scenarios that accept a custom objective scorer, preserving defaults and source-pinned selections through estimates, launch, and resume. Separate converter groups and combine the edited indicator with reset. Co-authored-by: Copilot App <[email protected]>
Richard Lundeen (richlundeen)
enabled auto-merge
October 27, 2026 16:09
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 27, 2026
Contributor
Author
|
Copilot resolve the merge conflicts on this branch. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Prompt capability detection can silently discard supplied input, and late converter creation responses can overwrite newer chain edits.
2 open issues
What changed in this PR
Adds opt-in tool role-play testing to PyRIT’s converter and technique framework, with shared configuration controls across chat and scenario workflows.
Changes:
- Adds the tool role-play converter and initializer, editable prompt templates, and separate message input and scoring objectives.
- Shares converter pickers, cards, and temporary settings; requires explicit GUI technique selection while preserving SDK defaults.
- Adds scenario scorer selection, replay support, tests, and documentation.
| File | Description |
|---|---|
tests/unit/setup/test_tools_initializer.py |
Tests opt-in registration and converter composition. |
tests/unit/scenario/scenarios/technique/test_execute.py |
Tests separate prompts and scoring objectives. |
tests/unit/registry/test_resolution.py |
Tests prompt coercion and multiline metadata. |
tests/unit/registry/test_converter_registry.py |
Tests role-play parameter metadata. |
tests/unit/models/test_parameter.py |
Tests SeedPrompt external-input handling. |
tests/unit/converter/test_role_play_tool_call_converter.py |
Tests conversion, configuration, and reconstruction. |
tests/unit/backend/test_technique_edit_service.py |
Tests temporary converters and scorer replay. |
tests/unit/backend/test_scenario_run_service.py |
Updates constructor-forwarding expectations. |
tests/unit/backend/test_scenario_configuration_resolver.py |
Tests scorer override validation. |
tests/unit/backend/test_converter_service.py |
Tests source-specific editable settings. |
pyrit/setup/initializers/tools.py |
Registers the opt-in tool technique. |
pyrit/setup/initializers/__init__.py |
Exports ToolsInitializer lazily. |
pyrit/scenario/scenarios/technique/execute.py |
Adds separate message input and compatibility checks. |
pyrit/scenario/core/attack_technique_factory.py |
Exposes separate-prompt capability metadata. |
pyrit/registry/resolution.py |
Derives multiline controls and resolves prompt text. |
pyrit/registry/registry.py |
Preserves prompt metadata during reconstruction. |
pyrit/registry/components/scenario_registry.py |
Exposes scorer support and forwards constructor inputs. |
pyrit/registry/components/attack_technique_registry.py |
Resolves temporary converter settings. |
pyrit/models/technique_edit.py |
Accepts temporary converter specifications. |
pyrit/models/seeds/seed.py |
Extracts template parameter names. |
pyrit/models/parameter.py |
Supports text-based SeedPrompt editing. |
pyrit/models/catalog/scenario.py |
Adds scorer references to scenario requests. |
pyrit/datasets/converters/role_play_tool_calls.yaml |
Defines the role-play prompt template. |
pyrit/converter/translation_converter.py |
Exposes the loaded default prompt. |
pyrit/converter/role_play_tool_call_converter.py |
Implements the configurable role-play converter. |
pyrit/converter/llm_generic_text_converter.py |
Supplies current input when rendering templates. |
pyrit/converter/converter.py |
Declares multiline parameter metadata. |
pyrit/converter/__init__.py |
Exports the new converter lazily. |
pyrit/backend/services/technique_edit_service.py |
Retains temporary converter settings in responses. |
pyrit/backend/services/scenario_service.py |
Applies scorer overrides during scenario preparation. |
pyrit/backend/services/scenario_run_service.py |
Persists and restores scorer selections. |
pyrit/backend/services/scenario_configuration_resolver.py |
Resolves and validates registered scorers. |
pyrit/backend/services/converter_service.py |
Projects editable instance defaults. |
pyrit/backend/models/techniques.py |
Adds prompt-support metadata. |
pyrit/backend/models/converters.py |
Adds editable parameter metadata. |
pyrit/backend/mappers/technique_mappers.py |
Maps prompt-support capability. |
frontend/src/utils/targetIdentity.ts |
Centralizes default adversarial-target resolution. |
frontend/src/utils/groupConverters.ts |
Shares converter grouping logic. |
frontend/src/types/index.ts |
Extends converter and scenario API types. |
frontend/src/hooks/useTargetPreferences.ts |
Reuses shared target resolution. |
frontend/src/components/Techniques/TechniqueEditor.tsx |
Uses shared converter-chain controls. |
frontend/src/components/Techniques/TechniqueEditor.test.tsx |
Tests temporary settings and converter creation. |
frontend/src/components/Scenarios/ScenarioDetail.tsx |
Adds scorer selection and prompt-aware controls. |
frontend/src/components/Scenarios/ScenarioDetail.test.tsx |
Tests scorer, prompt, and technique selection flows. |
frontend/src/components/Scenarios/ScenarioDetail.styles.ts |
Styles the technique-reset control. |
frontend/src/components/Registry/TechniqueRegistry.test.tsx |
Updates shared-picker interactions. |
frontend/src/components/Registry/CreateConverterDialog.tsx |
Supports multiline defaults and preserved source settings. |
frontend/src/components/Registry/CreateConverterDialog.test.tsx |
Tests defaults, target preservation, and upload restrictions. |
frontend/src/components/Converters/SelectConverterInput.tsx |
Shares the picker with configurable labels and disabling. |
frontend/src/components/Converters/SelectConverterInput.styles.ts |
Extracts shared picker styles. |
frontend/src/components/Converters/ConverterChainField.tsx |
Adds ordered converter-chain editing. |
frontend/src/components/Converters/ConverterChainField.styles.ts |
Styles converter-chain fields. |
frontend/src/components/Converters/ConverterCard.tsx |
Shares stage settings and reordering controls. |
frontend/src/components/Converters/ConverterCard.styles.ts |
Defines shared converter-card styles. |
frontend/src/components/Chat/ConverterPanel/ConverterPanel.tsx |
Reuses shared cards, picker, and grouping. |
frontend/src/components/Chat/ConverterPanel/ConverterPanel.styles.ts |
Removes extracted shared styles. |
frontend/src/components/Chat/ChatScorerPicker.tsx |
Supports scenario-specific scorer selection. |
frontend/src/App.tsx |
Supplies scorer registry data to scenarios. |
frontend/e2e/registry.spec.ts |
Updates end-to-end picker interactions. |
doc/scanner/technique.py |
Documents selection, separate prompts, and scorers. |
doc/scanner/technique.ipynb |
Synchronizes guidance and notebook output. |
doc/code/converters/1_text_to_text_converters.py |
Documents role-play configuration and usage. |
doc/code/converters/1_text_to_text_converters.ipynb |
Synchronizes converter guidance. |
doc/code/converters/0_converters.ipynb |
Adds the converter to the modality table. |
Review effort: Balanced
Preserve preset technique selections during the merge. Share fixed attack parameter contracts between execution and capability metadata, and keep late converter creation responses from overwriting current chains or closing newer dialogs. Co-authored-by: Copilot App <[email protected]>
Richard Lundeen (richlundeen)
enabled auto-merge
October 27, 2026 19:37
Richard Lundeen (richlundeen)
deleted the
richlundeen-agentic-ctf-strategy
branch
October 27, 2026 20:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add an opt-in tool role-play converter and technique for authorized agent tests, with editable system prompts and separate message input and scoring objectives. The GUI requires an explicit technique selection; SDK callers retain the
red_teamingdefault.Parametermetadata for multilineSeedPromptcontrols, loaded defaults, and existing converter settings.Tests and Documentation
Updated converter documentation and the technique notebook. Temporary technique edits retain existing uploaded files; use a new registered converter to replace a file.
Validation commands and results
uv run --no-sync pytest tests\unit\scenario\scenarios\technique\test_execute.py tests\unit\converter\test_role_play_tool_call_converter.py tests\unit\backend\test_technique_edit_service.py tests\unit\scenario\test_default_run_size_estimates.py -q --tb=short— 320 passed.uv run --no-sync pytest tests\unit\registry\test_attack_technique_registry.py tests\unit\scenario\test_default_run_size_estimates.py -q --tb=short— 209 passed after restoring the SDK default contract.frontend:npm run type-checkandnpm run build— passed.frontend:npm test -- --runInBand src\components\Scenarios\ScenarioDetail.test.tsx src\components\Chat\ConverterPanel.test.tsx --silent— 161 passed.frontend, with isolated seeded servers:npx playwright test registry.spec.ts --project seeded --workers 1 --grep 'creates and selects a runtime technique'— 1 passed.uv run --no-sync ty check tests\unit\converter\test_role_play_tool_call_converter.py— passed.uv run --no-sync pre-commit run --all-files— all hooks passed.uv run --no-sync jupytext --to ipynb --execute --set-kernel python3 doc\scanner\technique.py— passed with local-only targets.The screenshot uses the real UI with an isolated local-only configuration. No model or CTF mission calls were made.