Skip to content

Evaluate generated effect models through the shared expression language #1986

Open

Description

@abrichr

Evaluate generated effect models with the shared expression language

Generated save contracts need an executable representation for record identity, intended field changes and permitted related effects. Add a typed effect model that evaluates captured observations through the expression language from #1891 / #1983 and exposes a separate shadow result.

The model binds task, before and after values to the shared typed expression API. Equality, unchanged values, keyed collections, cardinality and allowed deltas use that language's operators and limits. Extend the shared language for missing generic operations. Keep one parser, type checker and evaluator.

Scope:

  • Add packages/flow/openadapt_flow/runtime/effects/effect_model.py for strict model/input/result types, caller-owned observation bindings and evaluation through openadapt_flow.transforms. Reuse existing effect/value contracts and normalization where semantics match. Pin the language version and supported operator profile.
  • Add packages/flow/openadapt_flow/generate/effect_model_admission.py for a pure admission summary over caller-supplied labelled replay results and model/language/schema/reader identities. Reuse zero-touch clean/fault separation.
  • Add an opt-in hook in packages/flow/openadapt_flow/shadow.py over captured evidence. Return the candidate result separately from the canonical verdict, ledger and signed receipt.
  • Add packages/flow/tests/test_effect_model.py, packages/flow/tests/test_effect_model_admission.py and focused shadow integration cases. Shared operator semantics are tested with the language, using generic synthetic records.

Acceptance:

  • Reject unknown fields, unsupported versions/operators, invalid bindings and exhausted shared budgets. Model data supplies neither executable code nor operator callbacks.
  • Bind required observation coverage, source/schema identity and freshness independently of model expressions. A model cannot waive reads or clear a prior refutation. Evaluation refusal yields an uncertain candidate result; a validated contradiction yields a refuted candidate result.
  • Preserve null versus absent values, exact scalar types and duplicate-key failures. Cover wrong values, unexpected records, cardinality changes and unintended deltas. Fuzzy matching cannot establish record identity or satisfy effect acceptance predicates.
  • Admit only nonempty clean and independently labelled fault replay sets with all clean cases confirmed and all faults refused. Missing/conflicting labels, missing required kinds or mismatched identities prevent admission. Replay evaluates observations without issuing writes.
  • Prove that candidate success, refusal and failure leave canonical verdicts, ledger entries and receipt bytes unchanged. Candidate inputs exclude agent outcome claims and narrative.

Application models, recipes and case data are supplied by callers. The first increment is this shared-language consumer, admission function and shadow integration.

Activity

  1. added
    agent-readyReady for an automated agent to pick up
    claimedAn agent is working on this
    on Nov 5, 2026
  2. abrichr commented on Nov 5, 2026

    @abrichr
    MemberAuthor

    Claimed by Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyReady for an automated agent to pick upclaimedAn agent is working on this

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions