From df99bbfaa0dc99002f9459eb6fe50ee8661525e1 Mon Sep 17 00:00:00 2001 From: abrichr Date: Thu, 5 Nov 2026 05:19:39 -0500 Subject: [PATCH 1/2] docs(audit): mark read-back ranks 4 and 9 fixed by #1747 Closes the realism-audit bookkeeping for OpenAdapt#1472. Neighbouring and fuzzy read-back confirms were removed in #1747; the Hypothesis controls in test_realism_readback_values.py pass on main. --- docs/audits/REALISM_AUDIT.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/audits/REALISM_AUDIT.md b/docs/audits/REALISM_AUDIT.md index 6021fde5ae..42ffb87b0e 100644 --- a/docs/audits/REALISM_AUDIT.md +++ b/docs/audits/REALISM_AUDIT.md @@ -38,12 +38,12 @@ Paths are relative to `packages/flow/openadapt_flow/` unless they start with | 1 | Replay: record identity | A row whose visible text matches the recording is the recorded row. Identity is verified per row and never checked for uniqueness on screen (`runtime/identity.py`). | Two worklist rows read "John Smith Follow-up Dr Lee" and swap order between recording and replay. Replay opens and signs the other patient's chart. | Homonyms in `validation/dense_surface.py` always differ in date of birth or record number. | OPEN, reproduced → **fixed** ([#1504](https://github.com/OpenAdaptAI/OpenAdapt/pull/1504)) | M × H | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_identical_rows_that_swap_order_never_sign_the_other_chart` (real Chromium, passes). Issue [#1469](https://github.com/OpenAdaptAI/OpenAdapt/issues/1469) | | 2 | Durable resume | A save that was delivered cleanly but then failed its record check wrote nothing. The pause carries `delivery_uncertainty` only when the delivery itself raised (`runtime/durable/controller.py:621-641`). The resume guard checks only that field (`runtime/durable/resume.py:1050-1067`). | The system of record is slow and the read-back times out after Save was clicked. The operator approves the default "resume", and Save is clicked again, creating a duplicate record. | `tests/test_durable_runtime.py::test_resume_continues_from_last_checkpoint` asserts the re-delivery as correct. `FakeBackend` writes nothing. | OPEN, reproduced → **fixed** ([#1485](https://github.com/OpenAdaptAI/OpenAdapt/pull/1485)) | H × H | `test_realism_durable_resume.py` (the backend writes a row per press), fix PR [#1485](https://github.com/OpenAdaptAI/OpenAdapt/pull/1485). Issue [#1470](https://github.com/OpenAdaptAI/OpenAdapt/issues/1470) | | 3 | Capture: input attribution | Every key and click during a window-scoped recording belongs to the target window. `_geometry_for_action` checks identity, visibility and bounds, never focus (`packages/capture/openadapt_capture/window_capture.py:597-628`). The active-window poller is off in window mode (`recorder.py:3149`). | Mid-recording, the user switches to a chat app or a password prompt and types. Those keys are stored as target actions against the target's frame, including the other app's text. | `FakePlatform` (`packages/capture/tests/test_window_capture.py`) has no notion of focus. | OPEN → **fixed** ([#1483](https://github.com/OpenAdaptAI/OpenAdapt/pull/1483)) | H × H | A focus probe on the capture scope; a fake platform that reports another focused process must refuse the key. Owned by the window-identity session (see [Coordination](#coordination)). Issue [#1471](https://github.com/OpenAdaptAI/OpenAdapt/issues/1471) | -| 4 | Read-back values | A non-exact read-back match means the expected value is present. `compare_value(..., exact=False)` is a substring test (`runtime/effects/readback_values.py:214`), used for every on-screen check (`runtime/effects/onscreen.py:603-610`). | The expected amount is 250 and the screen shows "$1,250.00". The expected quantity is 5 and the screen shows "Qty 15". A credit "$100.00 CR" reads as 100.00. Each confirms the write. | `test_onscreen_verifier.py` confirms embedded labels and never tests a neighbouring number. | **OPEN, reproduced** | H × H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (Hypothesis over integers and formats; xfail). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | +| 4 | Read-back values | A non-exact read-back match means the expected value is present. `compare_value(..., exact=False)` is a substring test (`runtime/effects/readback_values.py:214`), used for every on-screen check (`runtime/effects/onscreen.py:603-610`). | The expected amount is 250 and the screen shows "$1,250.00". The expected quantity is 5 and the screen shows "Qty 15". A credit "$100.00 CR" reads as 100.00. Each confirms the write. | `test_onscreen_verifier.py` confirms embedded labels and never tests a neighbouring number. | OPEN, reproduced → **fixed** ([#1747](https://github.com/OpenAdaptAI/OpenAdapt/pull/1747)) | H × H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (Hypothesis over integers and formats; passes). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | | 5 | Teach: parameter inference | A value typed whole in one step is the parameter, even if the same value also appears inside longer text in another step (`jit/lift.py`). | The agent types the patient's name into its field and into a note. The note keeps the demonstrated name on every later run. The record check passes because the literal is constant. | `tests/test_jit.py` asserted the old behaviour. | OPEN → **fixed** | H × H | Hypothesis property in [`test_jit.py`](../../packages/flow/tests/test_jit.py) (fix PR [#1465](https://github.com/OpenAdaptAI/OpenAdapt/pull/1465)) | | 6 | Effect checks: baselines | A record's identity is its raw `id`, or `str()` of each field (`runtime/effects/effect.py` `stable_id`). The baseline is stored with `model_dump(mode="json")` (`verify_run.py:709`). | Postgres, MSSQL and Oracle return UUID, Decimal and datetime values. After the JSON round trip they no longer compare equal, so every existing row counts as new. `count_new_only` then confirms a write that never happened. | `test_verify_run*.py` and `test_effect_sql.py` use SQLite ints and strings. | OPEN → **fixed** ([#1482](https://github.com/OpenAdaptAI/OpenAdapt/pull/1482)) | M × H | Hypothesis round trip in `tests/test_stable_id_json_baseline.py` | | 7 | Teach: canaries | Canary values are different from run 1's. Nothing checks it (`compile_agent/pipeline.py:1010-1041`). | A canary run with run 1's values passes even when the program types a constant, which makes canaries blind to ranks 5 and 8. | Every test happens to use different values. | OPEN | H × H | Property test: admitting with canary values equal to run 1's must refuse. Issue [#1473](https://github.com/OpenAdaptAI/OpenAdapt/issues/1473). | | 8 | Teach: parameter inference | A value typed whole exactly once belongs to the field it was typed in, with no record-field check. Record binding matches any field whose value equals the input (`jit/lift.py:164-171`, `record_sor.py:272-291`). | `quantity="1"` is set with a stepper, and "1" is typed into refills. Refills becomes the quantity parameter. | Synthetic workflows with long, unique values. | OPEN after the write | M × H | Hypothesis workflows with short values that collide. Issue [#1474](https://github.com/OpenAdaptAI/OpenAdapt/issues/1474). | -| 9 | Read-back values | A fuzzy text match whose longest common run covers 90% of the value confirms it (`runtime/effects/onscreen.py:262-277`). | "follow-up scheduled for 2026-10-29" confirms "…2026-10-21". "john smithson" confirms "John Smith". | No test with a difference at the end. | **OPEN, reproduced** | M × M-H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (xfail). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | +| 9 | Read-back values | A fuzzy text match whose longest common run covers 90% of the value confirms it (`runtime/effects/onscreen.py:262-277`). | "follow-up scheduled for 2026-10-29" confirms "…2026-10-21". "john smithson" confirms "John Smith". | No test with a difference at the end. | OPEN, reproduced → **fixed** ([#1747](https://github.com/OpenAdaptAI/OpenAdapt/pull/1747)) | M × M-H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (passes; `value_matches` is whole-field). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | | 10 | UI read-back | A "not found" page proves a record is absent. The browser read-back returns before its identity check (`runtime/effects/ui_readback.py:1089-1104`). The desktop read-back already guards this (`desktop_readback.py:1310`). | A void or delete workflow is verified by absence, but a wrong URL, an access-denied page or a filtered list also shows "not found". | `test_ui_readback_verifier.py` has no such case. | OPEN | M × H | Port the desktop test. Issue [#1475](https://github.com/OpenAdaptAI/OpenAdapt/issues/1475). | | 11 | Effect checks: paging | The first page of a FHIR or REST search holds every match (`runtime/effects/fhir.py:245`, `rest.py:100-150`). | Servers page at 20 to 50 results. A duplicate on page 2 still confirms `expected_count=1`. | `test_effect_fhir.py` has no paging. | OPEN | M × H | Fake FHIR server with `_count=2`. Issue [#1476](https://github.com/OpenAdaptAI/OpenAdapt/issues/1476). | | 12 | Capture: several windows | When several windows match, the largest is the target (`packages/capture/openadapt_capture/window_capture.py:2050`, `window_capture_linux.py:553`). macOS raises `WindowCaptureAmbiguousError` instead. | Two browser or EMR windows are open on different patients. The bigger one is recorded. | `test_window_capture_linux.py::test_resolver_binds_process_and_root_pixel_coordinates` asserted that the largest wins. | OPEN → **fixed** | M × H | [`test_realism_capture.py`](../../packages/capture/tests/test_realism_capture.py) (fix PR [#1468](https://github.com/OpenAdaptAI/OpenAdapt/pull/1468)). Issue [#1477](https://github.com/OpenAdaptAI/OpenAdapt/issues/1477) | From bbc6197a8d92a601506dd4022b68812523b2b39c Mon Sep 17 00:00:00 2001 From: abrichr Date: Thu, 5 Nov 2026 05:20:51 -0500 Subject: [PATCH 2/2] docs(audit): mark FHIR/REST paging rank 11 fixed by #1538 Closes the realism-audit bookkeeping for OpenAdapt#1476. Paged search-sets are followed whole (or read as unreadable) since #1538; test_effect_paging.py pins the controls. --- docs/audits/REALISM_AUDIT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/audits/REALISM_AUDIT.md b/docs/audits/REALISM_AUDIT.md index 42ffb87b0e..ca32dd6d79 100644 --- a/docs/audits/REALISM_AUDIT.md +++ b/docs/audits/REALISM_AUDIT.md @@ -45,7 +45,7 @@ Paths are relative to `packages/flow/openadapt_flow/` unless they start with | 8 | Teach: parameter inference | A value typed whole exactly once belongs to the field it was typed in, with no record-field check. Record binding matches any field whose value equals the input (`jit/lift.py:164-171`, `record_sor.py:272-291`). | `quantity="1"` is set with a stepper, and "1" is typed into refills. Refills becomes the quantity parameter. | Synthetic workflows with long, unique values. | OPEN after the write | M × H | Hypothesis workflows with short values that collide. Issue [#1474](https://github.com/OpenAdaptAI/OpenAdapt/issues/1474). | | 9 | Read-back values | A fuzzy text match whose longest common run covers 90% of the value confirms it (`runtime/effects/onscreen.py:262-277`). | "follow-up scheduled for 2026-10-29" confirms "…2026-10-21". "john smithson" confirms "John Smith". | No test with a difference at the end. | OPEN, reproduced → **fixed** ([#1747](https://github.com/OpenAdaptAI/OpenAdapt/pull/1747)) | M × M-H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (passes; `value_matches` is whole-field). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | | 10 | UI read-back | A "not found" page proves a record is absent. The browser read-back returns before its identity check (`runtime/effects/ui_readback.py:1089-1104`). The desktop read-back already guards this (`desktop_readback.py:1310`). | A void or delete workflow is verified by absence, but a wrong URL, an access-denied page or a filtered list also shows "not found". | `test_ui_readback_verifier.py` has no such case. | OPEN | M × H | Port the desktop test. Issue [#1475](https://github.com/OpenAdaptAI/OpenAdapt/issues/1475). | -| 11 | Effect checks: paging | The first page of a FHIR or REST search holds every match (`runtime/effects/fhir.py:245`, `rest.py:100-150`). | Servers page at 20 to 50 results. A duplicate on page 2 still confirms `expected_count=1`. | `test_effect_fhir.py` has no paging. | OPEN | M × H | Fake FHIR server with `_count=2`. Issue [#1476](https://github.com/OpenAdaptAI/OpenAdapt/issues/1476). | +| 11 | Effect checks: paging | The first page of a FHIR or REST search holds every match (`runtime/effects/fhir.py:245`, `rest.py:100-150`). | Servers page at 20 to 50 results. A duplicate on page 2 still confirms `expected_count=1`. | `test_effect_fhir.py` has no paging. | OPEN → **fixed** ([#1538](https://github.com/OpenAdaptAI/OpenAdapt/pull/1538)) | M × H | [`test_effect_paging.py`](../../packages/flow/tests/test_effect_paging.py) (fake FHIR/REST pages; passes). Issue [#1476](https://github.com/OpenAdaptAI/OpenAdapt/issues/1476). | | 12 | Capture: several windows | When several windows match, the largest is the target (`packages/capture/openadapt_capture/window_capture.py:2050`, `window_capture_linux.py:553`). macOS raises `WindowCaptureAmbiguousError` instead. | Two browser or EMR windows are open on different patients. The bigger one is recorded. | `test_window_capture_linux.py::test_resolver_binds_process_and_root_pixel_coordinates` asserted that the largest wins. | OPEN → **fixed** | M × H | [`test_realism_capture.py`](../../packages/capture/tests/test_realism_capture.py) (fix PR [#1468](https://github.com/OpenAdaptAI/OpenAdapt/pull/1468)). Issue [#1477](https://github.com/OpenAdaptAI/OpenAdapt/issues/1477) | | 13 | Replay: typed-input check | Typing into a field produces the field's old value plus the typed text, and that counts as landed (`runtime/replayer.py:16545-16555`). Nothing clears the field first. | A search box keeps its last query, or a restored draft fills the field. The step saves "SmithJones". | No test with a prefilled field. In a real browser, the click's region check halts this case today. | CLOSED in the tested shape; OPEN when the fill arrives after the click check | M × H | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_a_field_autofilled_on_focus_never_saves_a_concatenation` (passes; pins the halt) | | 14 | Replay: browser dialogs | A dialog the recording never saw is auto-dismissed by Playwright and not reported (`backends/playwright_backend.py:1137-1140, 3200-3232`). | A `confirm()` before Save returns false, so the save silently doesn't happen. | `test_multi_surface_dialog.py` covers recorded dialogs only. | CLOSED when the step has a visible postcondition (tested); OPEN when it has none | H × M | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_an_unrecorded_confirm_dialog_never_ends_in_success` (passes; pins the halt) |