Skip to content

fix(capture): seal policy-dropped source ordinals and hit-test macOS system chrome - #1984

Open
abrichr wants to merge 1 commit into
mainfrom
agent/1976
Open

abrichr wants to merge 1 commit into
mainfrom
agent/1976

Conversation

@abrichr

@abrichr abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member

Closes #1976. Part of #1867.

What changed

  • Dropped input no longer leaves an unexplained hole. Each native input reserves a source ordinal when it arrives. When window-scoped recording drops it (a click on another window, typing in another app, the release of a dropped press), the journal now records that ordinal with its reason code. The recorder writes these to source-ordinal-skips.json before sealing, so the artifact manifest covers the file. It holds ordinals and reason codes only.
  • Verification is not weakened. The no-missing-ordinal check stays. It accepts an empty ordinal only when the sealed record names it, with a known drop reason, in a window-scoped (v2) capture. A hole with no record (a crash, a lost write, a bug) still fails. So does a record that names an ordinal holding an event, an unknown reason, a non-canonical or empty record, a record in a non-window capture, and a record added after sealing. A discard without a known policy reason is refused when it happens. A failed producer is still an error, never a skip.
  • macOS system chrome is hit-tested. MacOSInputFocusProbe.window_at treated full-screen Dock (layer 20) and Notification Center (layer 23) windows as the window under every point, so clicks on the target were dropped. A window at the Dock level or above (below the status level) now counts only when the system Accessibility hit test (AXUIElementCopyElementAtPosition) says its process takes the click there, for example a Dock tile or a notification banner. When the hit test can't answer, the owner is unknown and the click is dropped (fail closed). Other apps' windows still block as before.

How I tested it

  • New tests: tests/test_source_skips.py (record format, journal), verifier cases in tests/test_capture_terminal.py, and in tests/test_window_input_focus.py an end-to-end record() run. That run clicks and types into another window, then goes through Recorder._verify_completed_capture, sealing and CaptureSession.validate_sealed. It also adds macOS probe tests with full-screen Dock and Notification Center windows.
  • Control: with the three production files reverted to origin/main and the new tests kept, 12 tests fail. These include the end-to-end test and the Dock/Notification Center test. With the fix, all 79 tests in these three files pass.
  • Full capture suite on macOS, Python 3.12: pytest tests/ -m "not slow" gave 1125 passed, 11 skipped. ruff check openadapt_capture/, the py310 E9 check and check_changelog.py pass.

What I could not test

…system chrome

Window-scoped recording dropped input that went to another window but left
its reserved source ordinal empty, so the sealed journal had a hole and
verification refused the whole recording ('missing ordinal').

The journal now records each policy drop with its reason, and the recorder
writes them to a sealed source-ordinal-skips.json. The verifier accepts an
empty ordinal only when that record names it with a known reason in a
window-scoped capture. Unrecorded holes still fail.

On macOS, full-screen Dock and Notification Center windows no longer count
as the window under the pointer unless the Accessibility hit test says their
process takes the click there; unknown fails closed.
@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 5, 2026
@abrichr
abrichr marked this pull request as ready for review November 5, 2026 18:06
@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Lab verification of 632a7b4:

macOS 15.7.7 guest, TextEdit, window-scoped record with this branch's Capture installed editable. A real click (hardware-level input, not injected) on the TextEdit window, 28 typed characters and Command-S. The capture sealed COMPLETE with 202 frames and 62 actions (2 click rows, 30 press, 30 release). Ordinals are contiguous (1..264), no input was dropped, and there is no skip record. CaptureSession.validate_sealed and load_verified pass. On main, the same click was dropped as pointer_over_other_window because of the full-screen Dock and Notification Center windows, and verification failed. The Flow step after it still fails, on unmapped key 'space'. That is #1963 and not related to this PR.

Linux X11 (Ubuntu 24.04 under Xvfb + openbox), #1966's qualify_linux_window_record.py without --skip-other-clicks, on a local merge of this branch with agent/1867 (not pushed). Two clicks in the target, two in the "Other" window, then one in the target: passed: true, record exit 0. It recorded exactly the 3 target clicks at [30,20], [60,30], [90,25], with every frame 560x360. With --skip-other-clicks the control also passes. One earlier two-window attempt failed for an unrelated reason. Capture's Linux resolver said "the process that owns pinned window … exited or restarted". It derives process identity from psutil create_time(), which can shift in WSL when the boot-time clock is adjusted. The next run passed. That looks like a separate Linux-identity flake worth its own issue.

CI on this PR is green (22 pass). The macOS and Windows capture unit jobs are skipped on PRs.

@abrichr

abrichr commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review: BLOCK

From: Codex

Reviewed head 632a7b4756964a971a5698f9b91fd89db657b849 against literal parent/base 9f503bd8115710bd181b5d0cfaf6ece144a08fad.

[P1] window_input_focus.py:341–343 can attribute another process's click to the target. After Accessibility names a PID different from the full-screen chrome, the loop skips that chrome but returns the first normal window without reconciling its PID with the hit test. When AX names a process absent from the Quartz candidates, or one present only at the filtered status-window level, the target behind it is returned. The actual WindowCaptureScope.assert_input_on_target then admits the click despite the contradictory owner.

Refuse an unresolved or contradictory owner before accepting a normal candidate. Keep the legitimate transparent-chrome positive and actual-chrome refusal cases.

Gated Linux validation: the three unchanged author files passed 79 tests; their opt-in live macOS test skipped. Ten independent head/parent probe controls produced seven passes and three failures: both ownership-disagreement cases failed at head and passed with the exact parent probe. The remaining failure is the parent's old refusal of legitimate target clicks through chrome, which the head fixes. The shared input-admission implementation is byte-identical at head and parent.

These are component controls using fake native bindings and the real admission check. No independent native macOS run or full-suite qualification is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-needed Needs an independent review from another vendor's agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Capture: window-scoped recording fails with 'missing ordinal' after input to another window is dropped

1 participant