Repository navigation
BUG: red_team_social_bias multi-turn prompts have no role, so their seed groups fail validation #3555
Description
Describe the bug
_RedTeamSocialBiasDataset (pyrit/datasets/seed_datasets/remote/red_team_social_bias_dataset.py) turns each "Multi Turn" row into one SeedPrompt per user turn, sharing a prompt_group_id, with sequence=0, 1, 2, ..., but it does not set their role. SeedGroup requires a role on each sequence of a group with more than one sequence (_enforce_consistent_role), so every multi-turn group fails validation:
ValueError: No roles set for sequence 0 in a multi-sequence group. Please ensure at least one prompt within a sequence has an assigned role.
Fetching the dataset works, but anything that builds its seed groups raises: SeedDataset.seed_groups, and memory.get_seed_groups_async(dataset_name="red_team_social_bias") after the dataset is added to memory. Since one bad group makes the whole call fail, the single-prompt rows of the dataset cannot be loaded as seed groups either. The upstream dataset has 111 "Multi Turn" rows, each with two or more user turns (the turns are named system_1, user_2, assistant_3, user_4, ...).
The other remote loaders that build multi-sequence groups (for example cyberseceval_prompt_injection_dataset.py and mask_dataset.py) set role on each prompt.
Steps/Code to Reproduce
import asyncio
from unittest.mock import AsyncMock, patch
from pyrit.datasets.seed_datasets.remote.red_team_social_bias_dataset import _RedTeamSocialBiasDataset
from pyrit.memory import CentralMemory
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
# One "Multi Turn" row in the format of svannie678/red_team_repo_social_bias_prompts
rows = [
{
"prompt": "[{'role': 'system_1', 'body': 'You are a helpful assistant.'}, "
"{'role': 'user_2', 'body': 'First turn'}, {'role': 'assistant_3', 'body': 'Response'}, "
"{'role': 'user_4', 'body': 'Second turn'}]",
"prompt_type": "Multi Turn",
"categorization": "discrimination",
"organization": "org",
}
]
async def main():
await initialize_pyrit_async(memory_db_type=IN_MEMORY)
loader = _RedTeamSocialBiasDataset()
with patch.object(loader, "_fetch_from_huggingface_async", new=AsyncMock(return_value=rows)):
dataset = await loader.fetch_dataset_async()
print("roles:", [seed.role for seed in dataset.seeds])
memory = CentralMemory.get_memory_instance()
await memory.add_seed_datasets_to_memory_async(datasets=[dataset], added_by="repro")
groups = await memory.get_seed_groups_async(dataset_name="red_team_social_bias")
print("groups:", [[(p.role, p.value) for p in g.prompts] for g in groups])
asyncio.run(main())Expected Results
roles: ['user', 'user']
groups: [[('user', 'First turn'), ('user', 'Second turn')]]
Actual Results
roles: [None, None]
Traceback (most recent call last):
...
pydantic_core._pydantic_core.ValidationError: 1 validation error for SeedGroup
Value error, No roles set for sequence 0 in a multi-sequence group. Please ensure at least one prompt within a sequence has an assigned role. [type=value_error, ...]
Screenshots
N/A
Versions
- OS: Windows 11
- Python version: 3.11.9
- PyRIT version: installed from main (d2c6fff) in editable mode
Activity
romanlutz commentedon Oct 27, 2026
Joel (@aspire488) please try to claim it first?
Shivam Sharma (@ShivamSharma43) can you please indicate that you're planning to solve it yourself? This is quite frustrating for people who think they're helping to then see that somebody posted a PR. Of course, they should also really say they want to do it first.
aspire488 commentedon Oct 27, 2026
Hi Roman, thanks for pointing this out. You're right — I should have explicitly claimed the issue before opening a PR.
I see that Shivam's PR #3557 has now been merged, so I won't pursue a duplicate fix. Apologies for the confusion, and thanks for the feedback. I'll make sure to communicate my intent on issues before starting work in the future.