Repository navigation
FEAT: Use SeedDataset-backed inline data for GCG - #3607
Open
Roman Lutz (romanlutz) wants to merge 17 commits into
Open
Roman Lutz (romanlutz) wants to merge 17 commits into
Roman Lutz (romanlutz) wants to merge 17 commits into
Conversation
…ft#1902) Co-authored-by: Copilot <[email protected]>
Add optional protocol fields on GCGAlgorithmConfig and wire GCGMultiPromptAttack.step to dispatch through protocol objects with default fallbacks that preserve legacy behavior when unset. Also wire suffix initialization through config and add unit coverage for config validation, manager wiring, default parity, and custom dispatch. Co-authored-by: Copilot <[email protected]>
Avoid failing the GCG step when logging tokenized control length for custom candidate-filter outputs that are not directly re-tokenizable by the active tokenizer. Co-authored-by: Copilot <[email protected]>
…handling - Add tests for __init__ with custom sampling/loss/filter protocols - Add tests for _resolve_* methods returning defaults - Add test for _get_control_length success and error paths - Add test for _resolve_control_init raising ValueError when suffix_init configured but no workers These tests cover previously uncovered lines in gcg_attack.py (lines 151, 163-165) and generator.py (line 463) to meet the 90% diff coverage requirement. Co-authored-by: Copilot <[email protected]>
Align test_get_control_length_success with _get_control_length behavior, which intentionally drops the first token before measuring length. Co-authored-by: Copilot <[email protected]>
Cover GCGMultiPromptAttack constructor wiring via real __init__, exercise gradient-shape mismatch sampling path in step(), and cover _resolve_control_init fallback when suffix_init is not configured. Co-authored-by: Copilot <[email protected]>
Use verbose=True in the gradient-shape mismatch step test so the prompt iteration yields integer indices (matching attack.step expectations) and avoids tuple indexing errors. Co-authored-by: Copilot <[email protected]>
Replace the URL-string-centric GCGDataConfig with an inline goal/target carrier so the JSON shipped to Azure ML embeds the resolved data. The compute container no longer fetches data from the public internet, which makes GCG runnable in air-gapped subscriptions. - GCGDataConfig now holds train/test goals+targets inline (paired-length validation) plus a from_seed_dataset adapter and a gcg_target metadata key. - Add fetch_advbench_for_gcg_async + private _AdvBenchGCGLoader to materialize the public AdvBench dataset as a SeedDataset (target under metadata). - load_goals_and_targets becomes a torch-free inline accessor; run.py is now URL-free and drops the random_seed arg. - Lazily export fetch_advbench_for_gcg_async; update the AML notebook to fetch + from_seed_dataset. - Update/add unit tests for serialization, validation, adapter, and fetcher. Co-authored-by: Copilot <[email protected]>
…set-adapter # Conflicts: # pyrit/auxiliary_attacks/gcg/config.py # tests/unit/auxiliary_attacks/gcg/test_config.py
…set-adapter # Conflicts: # doc/code/executor/gcg/1_gcg_azure_ml.ipynb # doc/code/executor/gcg/1_gcg_azure_ml.py # pyrit/executor/promptgen/gcg/__init__.py # pyrit/executor/promptgen/gcg/config.py # pyrit/executor/promptgen/gcg/data.py # tests/unit/executor/promptgen/gcg/test_advbench_fetcher.py # tests/unit/executor/promptgen/gcg/test_data_and_config.py
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
Reject malformed inline data and completion metadata, use final error-free assistant text, and require configurable scorer acceptance for generated prefixes. Add regression coverage and document migration from CSV-based configuration. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
Use PyRIT configuration for environment loading without unrelated target initialization, and use the default Azure credential chain for AML. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
Distinguish five optimization steps per goal from thirty logged checkpoints. Reject empty final suffixes and non-finite final losses. Verified the completed AML run's submitted pairs against bundled AdvBench and runtime target augmentation; retained evidence in session artifacts. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
Roman Lutz (romanlutz)
changed the title
Preserve SeedDataset-backed inline data while adopting main's shared lazy export contract and updated GCG implementation. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 46bcee92-47ce-448a-b0ac-679de6b7f6ae
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GCG needs paired objectives and target completion prefixes, but its CSV/URL-based input path bypassed PyRIT's dataset loaders and required dataset retrieval on Azure ML compute. This B4 change resolves supplied SeedDatasets locally and embeds the resulting goals and completions in the submitted JSON, making the compute-side data ingestion URL-free.
metadata["gcg_target"], preserving existing harm categories.Experimental API update: GCG is experimental and does not promise API compatibility, so this is not classified as a breaking release change.
GCGDataConfig(train_data=..., test_data=..., n_train_data=..., n_test_data=...)andload_goals_and_targets(..., random_seed=...)are replaced by caller-side loading/selection/splitting and inline config construction. The notebook includes migration examples. This intentionally avoids retaining compute-side URL fetching in the experimental GCG API. Follows the wiring in #2070 and accommodates the GCG migration in #2197.Tests and Documentation
Added regression coverage for inline serialization, dataset adaptation, invalid metadata/JSON, final-response handling, scorer acceptance, custom prompts, and bundled AdvBench completions. Updated both paired AML notebook formats.
Validation performed in this worktree:
uv run --link-mode=copy python -m pytest tests\unit\datasets\test_adv_bench_dataset.py tests\unit\executor\promptgen\gcg\ -q: 136 passed, 9 skipped at the adapter-fix commit. Skips were Torch-dependent tests before installing the GCG extra.uv run --extra gcg --link-mode=copy python -m pytest tests\unit\datasets\test_adv_bench_dataset.py tests\unit\executor\promptgen\gcg\test_config.py tests\unit\executor\promptgen\gcg\test_data.py -q: 121 passed after the final notebook changes.uv run --link-mode=copy ty check pyrit\executor\promptgen\gcg\config.py pyrit\executor\promptgen\gcg\data.py tests\unit\executor\promptgen\gcg\test_config.py tests\unit\executor\promptgen\gcg\test_data.py: passed.uv run --extra gcg --link-mode=copy pre-commit run --files doc\code\executor\gcg\1_gcg_azure_ml.py doc\code\executor\gcg\1_gcg_azure_ml.ipynb: passed. Source/test hooks also passed at their implementation commit.git diff --check: passed. Jupytext-parsed cell sources match between the.pyand.ipynbfiles.Live AML execution: Executed all eight code cells using
uv run --extra gcg --link-mode=copy python -with NBClient and this worktree's kernel. Run dynamic_nerve_96b78cyldr completed successfully onStandard_NC24ads_A100_v4. The submitted JSON exactly matches the first five bundled AdvBench goal/completion pairs; runtime goals match, and runtime targets reflect the existing GCG phrasing augmentation. Five optimization iterations per goal produced 30 logged checkpoints, including five initial checkpoints. The final suffix was nonempty and final loss was 1.451171875.The final notebook result checks were then executed against that downloaded result and rejected six invalid-output cases; no additional GPU job was submitted for that reporting change. The live run covers the metadata-backed dataset path, not missing-completion generation, which has unit coverage only. Optimization completion does not establish suffix effectiveness. The full repository test suite was not run.