[Yogyabhilash]

Description

Problem

In ShellCommandOutputScorer, dangerous shell payload families were missed when evaluating LLM responses under OWASP LLM02 (Insecure Output Handling):

1. Process substitution execution: Attackers frequently use process substitution bash <(curl ...) or source <(wget ...) / . <(curl ...) rather than standard pipes (curl | sh) to execute remote scripts directly in subshells without writing to disk.
2. Destructive filesystem variations: The Destructive Shell pattern strictly looked for combined rm -rf directly preceding root paths (/(?!tmp)). Consequently, it missed recursive deletions of user home directories (rm -rf ~, rm -rf ~/, rm -rf ~/.ssh, rm -rf $HOME), deletions specifying --no-preserve-root (e.g., rm -rf --no-preserve-root / or rm --no-preserve-root -rf /), and separated flags (rm -r -f /, rm -f -r /).

Solution

• Added Process Substitution Execution pattern to ShellCommandOutputScorer._DEFAULT_PATTERNS to detect remote download commands (curl, wget, fetch, iwr, irm) wrapped in <(...) and fed into shells (bash, zsh, dash, sh, source, ., and with sudo).
• Extended Destructive Shell pattern to:
  • Match user home directory targets (~, ~/, $HOME, ${HOME}).
  • Support --no-preserve-root before, within, and after target paths.
  • Support both combined (-rf, -fr) and separated (-r -f, -f -r) flag styles.
• Maintained strict false-positive bounds: local process substitutions (cat <(grep ...)), temporary directory cleanups (rm -rf /tmp/scratch), and single-file deletions remain safely unflagged.

Addresses the Shell injection gaps described in #3324 (following the split PR guidance in #3324 and sister PRs #3326, #3488, #3486).

Tests and Documentation

• Added unit tests in tests/unit/score/regex/test_shell_command_output_scorer.py:
  • test_process_substitution_detected covering bash <(...), source <(...), . <(...), zsh <(...), sh <(...), sudo bash <(...), and execution flags.
  • Expanded test_destructive_shell_detected covering home directories (~, ~/, ~/.ssh, $HOME), --no-preserve-root, and separated -r -f / -f -r flags.
  • Added negative controls in test_shell_not_detected covering safe cat <(grep ...), diff <(...), rm -rf /tmp, and rm -rf ./build.
• Ran unit test suite locally:
  • tests/unit/score/regex/test_shell_command_output_scorer.py: 92 passed.
  • tests/unit/score/regex/: 1,152 passed across all regex scorers.
• Verified linting and type-checking:
  • ruff check pyrit tests/unit/score/regex: clean.
  • ruff format --check pyrit tests/unit/score/regex: clean.
  • ty check pyrit/score/true_false/regex tests/unit/score/regex: clean.

[Yogyabhilash]

@microsoft-github-policy-service agree

[Aman (its-amann)]

this can flag safe commands now, the trailing (?:\s+[^\s;|\n]+)* walks past && and > so rm -rf node_modules && cd /app and rm -rf build > /dev/null both match Destructive Shell. maybe stop at & and > too and add those as negative cases?

[Yogyabhilash]

Good catch, thank you! I've updated both argument scans in Destructive Shell to delimit on shell operators and redirections ([^\s;|><&\n]+). Safe commands like rm -rf node_modules && cd /app and rm -rf build > /dev/null now correctly evaluate to False, while destructive commands targeting system or home directories remain strictly detected. Added these as negative test cases in test_shell_not_detected.

[Richard Lundeen (richlundeen)]

[P2] Keep destructive-command matching inside one shell command. In pyrit/score/true_false/regex/shell_command_output_scorer.py, both argument scans use [^\s;|\n]+. This includes & and >, so the scan can take the absolute path from a later command or a redirection. For example, rm -rf node_modules && cd /app and rm -rf build > /dev/null match the new destructive pattern even though neither deletes a system path. The earlier inline review also flags this, and it remains in the current head. Please stop both scans at shell operators and add these as negative cases while retaining the new positive cases.

Review based on code inspection; tests were not run. AI-assisted review (GitHub Copilot).

[Yogyabhilash]

Thanks for the review and sharp catch Richard Lundeen (@richlundeen) and Aman (@its-amann)!

I have addressed this in commit bc20b00a:

• Updated both argument scans in Destructive Shell from [^\s;|\n]+ to [^\s;|><&\n]+ so scans stop at shell statement delimiters and redirection operators (&, &&, >, <, |, ;).
• Verified that commands chaining or redirecting after local safe directory deletions (e.g. rm -rf node_modules && cd /app, rm -rf build > /dev/null, rm -rf dist 2> /dev/null, rm -rf target & echo /done, rm -rf build; cd /var, rm -rf build | tee /tmp/log) now cleanly evaluate to False.
• Verified that destructive commands targeting system paths or home directories (rm -rf /usr && echo done, rm -rf ~ > /dev/null) continue to be caught as expected.
• Added all of these negative and positive cases to test_shell_not_detected and test_destructive_shell_detected in test_shell_command_output_scorer.py (100 passed).