[abrichr]

What kind of change does this PR introduce?

New feature (engine, packages/flow). Part of #1891.

Summary

The IR can substitute parameters, check guards and loop over worklists, but it can't compute a value mid-run. This adds a small expression language for that, in a new module, openadapt_flow.transforms.

• Its own parser. A tokenizer, parser and type checker written for this language. Nothing goes to eval, exec, compile or ast.
• No escape routes. There is no attribute access, indexing syntax, import, assignment, lambda or comprehension. A bare name can only be a run parameter. A call can only name a function in the closed standard library. Dunder names are rejected while tokenizing.
• Typed, with no truthiness. The whole expression is type-checked before anything runs, and the evaluator re-checks types so a checker bug still fails closed.
• Total, bounded and deterministic. Source length, node count, nesting depth, string length, list length and integer size are all capped. There is no clock (today() is deliberately missing), no randomness, no I/O and no locale.
• Standard library:
  • strings: lower, upper, trim, collapse_spaces, length, replace, substring, starts_with, ends_with, contains, pad_left, text, to_int
  • lists: split, join, at, count
  • dates: date, parse_date and format_date (a locale-free YYYY/MM/DD/M/D format), add_days, add_months (clamps to the end of the month), days_between, year, month, day
  • regex: matches and extract. Both use full match only, with a literal pattern from a conservative subset. The subset refuses back-references, look-around, inline flags, a repeated group that repeats or alternates inside, more than 2 unbounded or 4 variable quantifiers, and input over 512 characters.
  • fuzzy_match(value, options, threshold): the threshold is required, must be a literal, and must be in (0, 1]. An exact match after case and space folding wins. Otherwise exactly one option must clear the threshold. If none does, it refuses with below_threshold. If two or more do, it refuses with ambiguous.
  • if_else
• Refusals. Anything outside the language raises ExpressionRefused with a closed RefusalCode. Refusal messages never repeat parameter values.
• IR declaration. ParamTransform(output_param, expression) is checked when it's declared, so a bad expression fails to load.
• Receipt records. apply_transforms(params, transforms) runs in order on a copy of the params. It's all or nothing. It won't overwrite an existing parameter. It returns one TransformRecord per computed value with:
  • the expression and its SHA-256
  • the names of the inputs it read and the SHA-256 of each input value
  • the computed value and its type
  • for each fuzzy match: the score, threshold, runner-up score and the index of the chosen option

One language, shared with effect models (second commit)

• LANGUAGE_VERSION (openadapt.expr/v1) and operator_set_sha256() name the exact functions, signatures, operators, bounds and refusal codes.
• Every TransformRecord binds both values. A test pins the digest, so any change to the library has to come as a reviewed new version.
• evaluate_predicate(source, params) gives a three-valued outcome: true, false or unknown (with the refusal code). It never raises, and a refusal is never a pass. A future effect-model evaluator can use the same language for its before/after predicates.
• docs/EXPRESSIONS.md documents the language.

How this was tested

• pytest -q tests/test_transforms.py: 114 passed. The original 111 are unchanged, plus 3 contract tests. On base it fails at collection, because the module doesn't exist yet.
  • Behavior: string functions, split and join on codes, date offsets and formats, regex extract, a fuzzy provider-to-option match, the receipt record contents, determinism, and the all-or-nothing rule.
  • Safety, 90+ cases:
    • Python escape attempts all refuse: __import__, x.__class__.__mro__, ().__class__.__base__.__subclasses__(), eval, exec, open, getattr, globals, lambdas, comprehensions, indexing, **, f-strings and assignment.
    • Names like os and builtins refuse as unknown_name.
    • Type errors and truthiness refuse before anything runs.
    • ReDoS patterns refuse: (a+)+, (a|aa)*, back-references and look-around.
    • Size and depth limits refuse instead of growing.
    • Bad dates and integers refuse.
    • A seeded fuzz of 3,000 random token strings only ever returns a value or refuses.
• Also passes tests/test_program_ir_phase1.py (134 passed together with the new file, rechecked on head 46544b5).
• Also passes ruff check, ruff format --check, mypy, the strict safety-path mypy flags on the new module, a Python 3.10 compile, check_engine_generality.py, check_release_consistency.py and the root check_source_boundary.py.

Not in this slice (follow-ups on #1891)

• No v2 operators. This PR is the openadapt.expr/v1 slice only. Any new operators come later as a separately reviewed version with a new digest.
• Not wired into the runtime yet. Step and Workflow don't carry ParamTransform yet, and the replayer doesn't call apply_transforms or write TransformRecords to RunReport or the run receipt. I left that out on purpose: those models feed contract digests, and that change deserves its own review.
• Fuzzy matching isn't read from the screen. The options come from a parameter or a list literal. Reading dropdown options from the live screen is a separate slice.
• Names are limited. Parameters can be referenced only by [A-Za-z_][A-Za-z0-9_]* names.
• The regex subset is a conservative heuristic, not a proof. It's backed by small input caps.
• No lab or GUI runs. This slice is software only.

Checklist

• I have tested this change locally
• Documentation is updated if needed (module docstring describes the language; user docs come with the runtime wiring)
• CI passes (CI runs now that the PR is marked ready)

[abrichr]

Independent adversarial review: BLOCK

Head: 46544b5837b6d03dd9c783a4d396116a0a6c9553

What I ran

• pytest tests/test_transforms.py: 114 passed. ruff check and ruff format --check on both changed files: clean. mypy openadapt_flow/transforms.py: no issues. gh pr checks 1983: all green.
• Scratch probes outside the repo for parser escapes (dunders, attribute access, indexing, fullwidth and zero-width identifiers, chained comparisons, adjacent strings), type confusion (bool vs int, number in text/==/if_else), threshold bypasses, int and date overflow, regex backtracking, fuzzy cost, and surrogates. I also ran a Hypothesis fuzz (40,000 token-sequence examples, passed) that checks only ExpressionRefused escapes and that predicates are three-valued.

The parser, type checker, int/date bounds, fuzzy ambiguity rule and all-or-nothing apply_transforms held up. Every escape attempt above was refused with the expected code. Two resource bounds can be bypassed, and the evaluator hangs on input that passes declaration-time validation.

Findings

P1. Catastrophic regex backtracking through 4-digit repeat counts. packages/flow/openadapt_flow/transforms.py:591 (and the same regex at :709) only recognizes {n} / {n,m} with at most 3 digits. Python accepts {1000}, {0100} and {0,9999}, so _quantifier_at returns "no quantifier". That skips both the nested-repeat rule and the hi > 100 bound. ParamTransform(expression="if_else(matches(x, '(a?){1000}b'), 'y', 'n')") validates at load. evaluate_predicate("matches(x, '(a?){1000}b')", {"x": "a"*16}) did not return within 30 s, and (a?){0100}b behaves the same way. This is a denial of service, and evaluation never fails closed. Suggested fix: treat any { that re would read as a repeat as a quantifier, using \{\d*(,\d*)?\} with no digit cap, and refuse counts above the bound instead of ignoring them. Add (a?){1000}b and (a?){0100}b to test_backtracking_prone_patterns_are_refused.

P1. fuzzy_match length bound is checked before NFKC expansion. transforms.py:1132 caps raw lengths at MAX_FUZZY_CHARS, but _normalize (:733) runs NFKC afterwards, which can expand one code point to 18 (for example U+FDFA). With x = "ﷺ"*256 and 38 similar options (inside MAX_STRING_CHARS), one evaluate_predicate call took 36.8 s in difflib. Suggested fix: apply the length and total-work bounds to the normalized strings, or bound the sum of len(q)*len(o) over the options.

P2. Non-taxonomy exceptions from apply_transforms. _sha256 (:1399, used at :1436-1438) raises UnicodeEncodeError for a parameter value that contains a lone surrogate (e.g. "\udc00"). The function still fails closed because nothing is returned, but a caller that switches on RefusalCode gets an uncaught crash instead. Suggested fix: refuse non-encodable text as INVALID_VALUE when parameters are checked (:1416).

P3

• lower() (:834) is not length-checked. "İ"*10000 lowers to 20000 chars, which is over MAX_STRING_CHARS. upper() is checked; lower should be too.
• PredicateResult.outcome (:1300) is a non-empty string, so a caller writing if r.outcome: passes on "false" and "unknown". Consider a passed: bool property that is true only for "true", or an Enum, since this is the shared predicate API for the effect model.
• Determinism across interpreters: casefold, lower/upper, NFKC, and Unicode \d/\w in patterns depend on unicodedata.unidata_version. extract(x, '(\d+)', 1) returns Arabic-Indic digits. operator_set() does not bind the Unicode version, so the same receipt can differ across Python versions. Consider compiling with re.ASCII and adding unidata_version to the operator set.
• if_else evaluation (:1120) uses Python truthiness instead of _as_bool, unlike the other operators' fail-closed re-checks.
• An "unknown" predicate result drops inputs (:1323), which loses evidence the receipt could carry.

BLOCK is for the two P1 resource-bound bypasses. Both are small, local fixes. With them and regression tests in place, I would expect to approve.